2024-04-21T08:27:00.203 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-21T08:27:00.204 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi --description fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} --archive /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520 --verbose --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/orig.config.yaml 2024-04-21T08:27:00.265 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T08:27:00.482 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520 branch: wip-lusov-quiescer description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} email: null first_in_suite: false job_id: '7666520' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 4f6bc5273643aaf5660909e85400647a8951c10b owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7599 sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e sleep_before_teardown: 0 subset: 111/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa suite_relpath: qa suite_repo: https://github.com/leonid-s-usov/ceph-ceph suite_sha1: 4f6bc5273643aaf5660909e85400647a8951c10b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDjX0ijBTgLbBPUyjZ3nVXCfPy93da4luDPppeYTkoeK3VUkNvP74ytGMnARsTwcQXdfHAXq2vUJFV1poy7h3k= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4TrixGAKYMBUBaznn3bph4e44sEH3clkY2q4tMkTVDky9gv6scAOb9g0W4aWl+MTC92IGXxocVOBbocn8CsOk= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHQbN6MN5RiTZ5v4DsWWIfrctTqM3x8cZ2VWQSYOaxwxFN5lfvQalZiR50T+HhQMwG4KqaINUkDqHYE40tslcms= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - kernel_untar_build.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-21_08:12:55 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-21T08:27:00.483 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa; will attempt to use it 2024-04-21T08:27:00.483 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks 2024-04-21T08:27:00.484 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-21T08:27:00.485 INFO:teuthology.task.internal:Checking packages... 2024-04-21T08:27:00.509 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a630765c1f93c6e61d9b40c4b76437611d5ef29e' 2024-04-21T08:27:00.509 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-21T08:27:00.509 INFO:teuthology.packaging:ref: None 2024-04-21T08:27:00.509 INFO:teuthology.packaging:tag: None 2024-04-21T08:27:00.509 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-21T08:27:00.509 INFO:teuthology.packaging:sha1: a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:27:00.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-21T08:27:00.750 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154.ga630765c 2024-04-21T08:27:00.751 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-21T08:27:00.758 INFO:teuthology.task.internal:no buildpackages task found 2024-04-21T08:27:00.758 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-21T08:27:00.822 INFO:teuthology.task.internal:Saving configuration 2024-04-21T08:27:00.845 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-21T08:27:00.855 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T08:27:00.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:57.320004', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDjX0ijBTgLbBPUyjZ3nVXCfPy93da4luDPppeYTkoeK3VUkNvP74ytGMnARsTwcQXdfHAXq2vUJFV1poy7h3k='} 2024-04-21T08:27:00.905 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:57.318679', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4TrixGAKYMBUBaznn3bph4e44sEH3clkY2q4tMkTVDky9gv6scAOb9g0W4aWl+MTC92IGXxocVOBbocn8CsOk='} 2024-04-21T08:27:00.924 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:57.317285', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHQbN6MN5RiTZ5v4DsWWIfrctTqM3x8cZ2VWQSYOaxwxFN5lfvQalZiR50T+HhQMwG4KqaINUkDqHYE40tslcms='} 2024-04-21T08:27:00.924 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-21T08:27:00.930 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-21T08:27:00.930 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-21T08:27:00.930 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-21T08:27:00.930 INFO:teuthology.run_tasks:Running task console_log... 2024-04-21T08:27:01.040 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f39a60680d0>, signals=[15]) 2024-04-21T08:27:01.041 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-21T08:27:01.048 INFO:teuthology.task.internal:Opening connections... 2024-04-21T08:27:01.049 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:27:01.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:27:01.148 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-04-21T08:27:01.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:27:01.223 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-04-21T08:27:01.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:27:01.303 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-21T08:27:01.310 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2024-04-21T08:27:01.328 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2024-04-21T08:27:01.329 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="9" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="9" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:27:01.387 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2024-04-21T08:27:01.388 INFO:teuthology.orchestra.run.smithi057.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:27:01.388 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:27:01.388 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:27:01.388 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:27:01.388 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:27:01.388 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:27:01.389 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2024-04-21T08:27:01.413 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-04-21T08:27:01.431 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-04-21T08:27:01.431 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-04-21T08:27:01.488 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-04-21T08:27:01.488 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-04-21T08:27:01.488 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-04-21T08:27:01.488 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:27:01.488 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:27:01.489 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:27:01.489 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-04-21T08:27:01.512 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-04-21T08:27:01.530 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-04-21T08:27:01.531 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-04-21T08:27:01.587 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-04-21T08:27:01.587 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-04-21T08:27:01.587 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-21T08:27:01.588 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-21T08:27:01.588 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-04-21T08:27:01.612 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-21T08:27:01.624 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-21T08:27:01.630 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-21T08:27:01.630 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:27:01.633 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:27:01.635 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-04-21T08:27:01.651 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-21T08:27:01.658 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-21T08:27:01.658 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:27:01.690 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:27:01.692 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-04-21T08:27:01.793 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-21T08:27:01.872 INFO:teuthology.run_tasks:Running task kernel... 2024-04-21T08:27:01.886 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-21T08:27:01.886 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-21T08:27:01.886 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-21T08:27:01.886 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:27:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:27:01.904 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-04-21T08:27:01.963 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-437.el9.x86_64 2024-04-21T08:27:01.963 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.14.0-437.el9.x86_64 2024-04-21T08:27:01.963 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2024-04-21T08:27:04.317 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-04-21T08:27:06.110 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-04-21T08:27:18.018 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 768 kB/s | 6.2 MB 00:08 2024-04-21T08:27:19.569 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-21T08:27:21.397 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-21T08:27:26.569 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-21T08:27:28.675 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-21T08:27:28.676 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-21T08:27:28.779 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-04-21T08:27:28.780 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-04-21T08:27:28.780 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-04-21T08:27:28.902 DEBUG:teuthology.orchestra.run.smithi057:> echo no | sudo yum reinstall kernel || true 2024-04-21T08:27:29.419 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:03 ago on Sun 21 Apr 2024 08:27:26 AM UTC. 2024-04-21T08:27:29.543 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-04-21T08:27:29.556 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-21T08:27:29.556 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-04-21T08:27:29.556 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout:Reinstalling: 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 255 k 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 0 2024-04-21T08:27:29.557 INFO:teuthology.orchestra.run.smithi057.stderr:Operation aborted. 2024-04-21T08:27:29.609 INFO:teuthology.orchestra.run.smithi057.stdout:Is this ok [y/N]: 2024-04-21T08:27:29.609 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum reinstall -y kernel || true 2024-04-21T08:27:30.123 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:04 ago on Sun 21 Apr 2024 08:27:26 AM UTC. 2024-04-21T08:27:30.244 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:Reinstalling: 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 255 k 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 0 2024-04-21T08:27:30.245 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-04-21T08:27:30.633 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-437.el9.x86_64.rpm 978 kB/s | 255 kB 00:00 2024-04-21T08:27:30.633 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:27:30.633 INFO:teuthology.orchestra.run.smithi057.stdout:Total 656 kB/s | 255 kB 00:00 2024-04-21T08:27:30.654 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-04-21T08:27:30.857 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-04-21T08:27:30.857 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-04-21T08:27:30.862 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-04-21T08:27:30.862 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-04-21T08:27:31.016 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-04-21T08:27:31.097 INFO:teuthology.orchestra.run.smithi057.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:27:31.097 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:27:31.815 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:27:31.815 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-21T08:27:32.482 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-21T08:27:32.482 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:27:32.482 INFO:teuthology.orchestra.run.smithi057.stdout:Reinstalled: 2024-04-21T08:27:32.483 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-21T08:27:32.483 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:27:32.483 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-04-21T08:27:32.660 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-04-21T08:27:32.748 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-21T08:27:32.748 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-21T08:27:32.749 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-21T08:27:32.749 DEBUG:teuthology.parallel:result is None 2024-04-21T08:27:32.749 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-21T08:27:32.756 INFO:teuthology.task.internal:Creating test directory... 2024-04-21T08:27:32.756 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:27:32.760 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:27:32.762 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-21T08:27:32.780 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-21T08:27:32.786 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-21T08:27:32.791 INFO:teuthology.task.internal:Creating archive directory... 2024-04-21T08:27:32.792 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:27:32.815 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:27:32.820 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-21T08:27:33.156 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-21T08:27:33.163 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-21T08:27:33.163 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:27:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:27:33.180 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-21T08:27:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:27:33.197 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:27:33.199 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:27:33.222 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-21T08:27:33.244 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:27:33.262 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:27:33.264 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:27:33.268 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:27:33.296 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:27:33.300 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-21T08:27:33.301 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-21T08:27:33.308 INFO:teuthology.task.internal:Configuring sudo... 2024-04-21T08:27:33.308 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:27:33.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:27:33.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-21T08:27:33.368 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-21T08:27:33.376 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-21T08:27:33.376 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:27:33.379 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:27:33.406 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-21T08:27:33.424 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:27:33.472 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:27:33.528 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:27:33.610 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:27:33.652 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:27:33.652 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:27:33.717 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:27:33.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:27:33.806 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:27:33.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:27:33.931 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:27:33.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:27:33.997 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:27:34.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-21T08:27:34.099 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:27:34.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-21T08:27:34.225 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:27:34.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-21T08:27:34.290 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2024-04-21T08:27:34.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-04-21T08:27:34.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-04-21T08:27:34.326 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:27:34.332 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:27:34.373 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T08:27:34.670 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-21T08:27:34.678 INFO:teuthology.task.internal:Starting timer... 2024-04-21T08:27:34.678 INFO:teuthology.run_tasks:Running task pcp... 2024-04-21T08:27:34.703 INFO:teuthology.run_tasks:Running task selinux... 2024-04-21T08:27:34.711 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-21T08:27:34.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2024-04-21T08:27:34.799 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2024-04-21T08:27:34.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-04-21T08:27:34.852 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-04-21T08:27:34.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-04-21T08:27:34.921 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-04-21T08:27:34.923 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-21T08:27:34.923 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2024-04-21T08:27:34.945 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2024-04-21T08:27:34.945 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-04-21T08:27:34.968 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-04-21T08:27:34.969 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-04-21T08:27:34.995 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-04-21T08:27:34.995 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-04-21T08:27:34.996 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:27:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:27:35.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:27:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:27:35.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T08:27:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:27:35.075 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-21T08:27:35.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:27:35.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:27:35.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-04-21T08:27:35.163 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-21T08:27:35.175 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-21T08:27:35.193 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-21T08:27:35.194 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi129.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-21T08:31:24.833 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-04-21T08:31:24.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-21T08:31:24.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:31:24.918 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-04-21T08:31:24.998 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-04-21T08:31:24.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-21T08:31:24.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:31:25.071 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-04-21T08:31:25.154 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-04-21T08:31:25.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-21T08:31:25.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-21T08:31:25.230 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-04-21T08:31:25.309 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-21T08:31:25.309 INFO:teuthology.run_tasks:Running task clock... 2024-04-21T08:31:25.320 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-21T08:31:25.320 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:31:25.321 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:31:25.323 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:31:25.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:31:25.325 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-21T08:31:25.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T08:31:25.352 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:31:25.355 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:31:25.366 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:31:25.370 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:31:25.377 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-21T08:31:25.389 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-04-21T08:31:25.390 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-21T08:31:25.395 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2024-04-21T08:31:25.398 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-04-21T08:31:25.407 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2024-04-21T08:31:25.412 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:31:25.413 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-04-21T08:31:25.421 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:31:25.423 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-04-21T08:31:25.424 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:31:25.435 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:31:25.436 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-21T08:31:25.449 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-21T08:31:25.482 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:31:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:31:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-04-21T08:31:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.485 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.495 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:31:25.499 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:31:25.499 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-04-21T08:31:25.499 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.499 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.499 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.499 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.512 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-04-21T08:31:25.516 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T08:31:25.516 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-04-21T08:31:25.516 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.516 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.516 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.516 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T08:31:25.517 INFO:teuthology.run_tasks:Running task install... 2024-04-21T08:31:25.528 DEBUG:teuthology.task.install:project ceph 2024-04-21T08:31:25.528 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'}} 2024-04-21T08:31:25.528 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e'} 2024-04-21T08:31:25.528 INFO:teuthology.task.install:Using flavor: default 2024-04-21T08:31:25.534 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-21T08:31:25.534 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-21T08:31:25.534 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:31:25.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:31:25.537 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:31:25.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:31:25.538 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'tag': None, 'wait_for_package': False} 2024-04-21T08:31:25.538 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:31:25.764 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:31:25.764 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:31:25.767 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:31:25.767 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:31:25.777 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/ 2024-04-21T08:31:25.778 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3154.ga630765c 2024-04-21T08:31:25.883 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:31:25.883 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:31:25.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:31:25.885 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:31:25.885 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:31:25.885 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:31:25.910 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:31:25.910 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:31:25.912 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:31:25.912 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:31:25.947 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-21T08:31:25.947 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:31:25.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-21T08:31:25.972 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-21T08:31:25.973 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a630765c1f93c6e61d9b40c4b76437611d5ef29e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-21T08:31:25.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:31:25.979 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:31:26.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-21T08:31:26.048 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:31:26.055 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:31:26.111 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-21T08:31:26.112 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-04-21T08:31:26.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-04-21T08:31:26.120 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2024-04-21T08:31:26.121 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2024-04-21T08:31:26.174 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-04-21T08:31:26.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-04-21T08:31:26.337 INFO:teuthology.orchestra.run.smithi136.stdout:48 files removed 2024-04-21T08:31:26.356 INFO:teuthology.orchestra.run.smithi057.stdout:48 files removed 2024-04-21T08:31:26.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:31:26.384 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:31:26.399 INFO:teuthology.orchestra.run.smithi129.stdout:48 files removed 2024-04-21T08:31:26.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-21T08:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-04-21T08:31:27.098 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 189 kB/s | 83 kB 00:00 2024-04-21T08:31:27.180 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 169 kB/s | 83 kB 00:00 2024-04-21T08:31:27.508 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 27 kB/s | 12 kB 00:00 2024-04-21T08:31:27.739 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 23 kB/s | 12 kB 00:00 2024-04-21T08:31:27.815 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 17 kB/s | 12 kB 00:00 2024-04-21T08:31:28.051 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-04-21T08:31:28.348 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 3.3 kB/s | 1.9 kB 00:00 2024-04-21T08:31:28.819 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-21T08:31:29.092 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-21T08:31:30.624 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 701 B/s | 1.9 kB 00:02 2024-04-21T08:31:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-21T08:31:32.015 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-21T08:31:32.490 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-04-21T08:31:34.375 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-21T08:31:35.536 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.2 MB 00:00 2024-04-21T08:31:37.245 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-21T08:31:38.263 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-21T08:31:40.233 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.2 MB 00:02 2024-04-21T08:31:41.751 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-04-21T08:31:42.704 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-21T08:31:43.388 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-21T08:31:43.415 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 807 kB/s | 6.2 MB 00:07 2024-04-21T08:31:44.800 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-04-21T08:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:45.551 INFO:teuthology.orchestra.run.smithi136.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:31:45.551 INFO:teuthology.orchestra.run.smithi136.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:31:45.552 INFO:teuthology.orchestra.run.smithi136.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:31:45.553 INFO:teuthology.orchestra.run.smithi136.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:31:45.555 INFO:teuthology.orchestra.run.smithi136.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:31:45.557 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:31:45.558 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:31:45.618 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:31:45.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:31:45.644 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:31:45.645 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:31:45.646 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:31:45.647 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:31:45.648 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:31:45.649 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:31:45.650 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:31:45.651 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:31:45.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:31:45.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:31:45.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:31:45.655 INFO:teuthology.orchestra.run.smithi136.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout:Install 148 Packages 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-04-21T08:31:45.656 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:31:45.657 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 218 M 2024-04-21T08:31:45.657 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-04-21T08:31:47.902 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-21T08:31:47.964 INFO:teuthology.orchestra.run.smithi136.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-21T08:31:48.231 INFO:teuthology.orchestra.run.smithi136.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 3.3 MB/s | 890 kB 00:00 2024-04-21T08:31:48.281 INFO:teuthology.orchestra.run.smithi136.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 2.9 MB/s | 145 kB 00:00 2024-04-21T08:31:48.573 INFO:teuthology.orchestra.run.smithi136.stdout:(4/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 8.1 MB/s | 2.4 MB 00:00 2024-04-21T08:31:48.723 INFO:teuthology.orchestra.run.smithi136.stdout:(5/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 10 MB/s | 1.5 MB 00:00 2024-04-21T08:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout:(6/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 8.2 MB/s | 4.7 MB 00:00 2024-04-21T08:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout:(7/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 3.4 MB/s | 5.4 MB 00:01 2024-04-21T08:31:50.146 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:50.147 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:31:50.158 INFO:teuthology.orchestra.run.smithi057.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:31:50.159 INFO:teuthology.orchestra.run.smithi057.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:31:50.160 INFO:teuthology.orchestra.run.smithi057.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:31:50.160 INFO:teuthology.orchestra.run.smithi057.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:31:50.162 INFO:teuthology.orchestra.run.smithi057.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:31:50.164 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:31:50.165 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:31:50.228 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================= 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================= 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:31:50.238 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:31:50.239 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:31:50.240 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:31:50.240 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:31:50.240 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:31:50.242 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:31:50.243 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:31:50.244 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:31:50.245 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:31:50.246 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:31:50.247 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:31:50.248 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:31:50.249 INFO:teuthology.orchestra.run.smithi057.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:31:50.250 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:31:50.251 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout:Installing weak dependencies: 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================= 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout:Install 148 Packages 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 2 Packages 2024-04-21T08:31:50.252 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:31:50.253 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 218 M 2024-04-21T08:31:50.253 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-04-21T08:31:50.823 INFO:teuthology.orchestra.run.smithi136.stdout:(8/150): ceph-osd-19.0.0-3154.ga630765c.el9.x86 11 MB/s | 17 MB 00:01 2024-04-21T08:31:50.874 INFO:teuthology.orchestra.run.smithi136.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 505 kB/s | 25 kB 00:00 2024-04-21T08:31:51.224 INFO:teuthology.orchestra.run.smithi136.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 5.7 MB/s | 11 MB 00:01 2024-04-21T08:31:51.333 INFO:teuthology.orchestra.run.smithi136.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 2.0 MB/s | 225 kB 00:00 2024-04-21T08:31:51.483 INFO:teuthology.orchestra.run.smithi136.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 180 kB/s | 27 kB 00:00 2024-04-21T08:31:51.742 INFO:teuthology.orchestra.run.smithi136.stdout:(13/150): ceph-common-19.0.0-3154.ga630765c.el9 5.3 MB/s | 21 MB 00:03 2024-04-21T08:31:51.767 INFO:teuthology.orchestra.run.smithi136.stdout:(14/150): libcephfs-devel-19.0.0-3154.ga630765c 112 kB/s | 32 kB 00:00 2024-04-21T08:31:51.826 INFO:teuthology.orchestra.run.smithi136.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 2.7 MB/s | 163 kB 00:00 2024-04-21T08:31:51.885 INFO:teuthology.orchestra.run.smithi136.stdout:(16/150): libcephfs2-19.0.0-3154.ga630765c.el9. 5.1 MB/s | 746 kB 00:00 2024-04-21T08:31:51.910 INFO:teuthology.orchestra.run.smithi136.stdout:(17/150): librados-devel-19.0.0-3154.ga630765c. 1.5 MB/s | 127 kB 00:00 2024-04-21T08:31:51.977 INFO:teuthology.orchestra.run.smithi136.stdout:(18/150): libradosstriper1-19.0.0-3154.ga630765 5.3 MB/s | 501 kB 00:00 2024-04-21T08:31:52.069 INFO:teuthology.orchestra.run.smithi136.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 492 kB/s | 45 kB 00:00 2024-04-21T08:31:52.178 INFO:teuthology.orchestra.run.smithi136.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 1.2 MB/s | 135 kB 00:00 2024-04-21T08:31:52.278 INFO:teuthology.orchestra.run.smithi136.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 1.6 MB/s | 162 kB 00:00 2024-04-21T08:31:52.386 INFO:teuthology.orchestra.run.smithi136.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 2.9 MB/s | 321 kB 00:00 2024-04-21T08:31:52.487 INFO:teuthology.orchestra.run.smithi136.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 2.9 MB/s | 298 kB 00:00 2024-04-21T08:31:52.570 INFO:teuthology.orchestra.run.smithi136.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 1.2 MB/s | 100 kB 00:00 2024-04-21T08:31:52.662 INFO:teuthology.orchestra.run.smithi136.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 932 kB/s | 85 kB 00:00 2024-04-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi136.stdout:(26/150): librgw2-19.0.0-3154.ga630765c.el9.x86 6.1 MB/s | 5.4 MB 00:00 2024-04-21T08:31:52.888 INFO:teuthology.orchestra.run.smithi136.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-21T08:31:52.972 INFO:teuthology.orchestra.run.smithi136.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 411 kB/s | 34 kB 00:00 2024-04-21T08:31:53.072 INFO:teuthology.orchestra.run.smithi136.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.4 MB/s | 142 kB 00:00 2024-04-21T08:31:53.139 INFO:teuthology.orchestra.run.smithi136.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 6.5 MB/s | 3.1 MB 00:00 2024-04-21T08:31:53.898 INFO:teuthology.orchestra.run.smithi136.stdout:(31/150): ceph-test-19.0.0-3154.ga630765c.el9.x 16 MB/s | 49 MB 00:03 2024-04-21T08:31:53.956 INFO:teuthology.orchestra.run.smithi136.stdout:(32/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 4.0 MB/s | 3.5 MB 00:00 2024-04-21T08:31:53.981 INFO:teuthology.orchestra.run.smithi136.stdout:(33/150): ceph-mgr-modules-core-19.0.0-3154.ga6 2.9 MB/s | 249 kB 00:00 2024-04-21T08:31:54.015 INFO:teuthology.orchestra.run.smithi136.stdout:(34/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 845 kB/s | 49 kB 00:00 2024-04-21T08:31:54.040 INFO:teuthology.orchestra.run.smithi136.stdout:(35/150): ceph-prometheus-alerts-19.0.0-3154.ga 281 kB/s | 16 kB 00:00 2024-04-21T08:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout:(36/150): ceph-mgr-diskprediction-local-19.0.0- 7.5 MB/s | 7.4 MB 00:00 2024-04-21T08:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout:(37/150): ceph-volume-19.0.0-3154.ga630765c.el9 1.9 MB/s | 273 kB 00:00 2024-04-21T08:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout:(38/150): cephadm-19.0.0-3154.ga630765c.el9.noa 4.7 MB/s | 762 kB 00:00 2024-04-21T08:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout:(39/150): cephfs-top-19.0.0-3154.ga630765c.el9. 197 kB/s | 18 kB 00:00 2024-04-21T08:31:54.350 INFO:teuthology.orchestra.run.smithi136.stdout:(40/150): fuse-2.9.9-15.el9.x86_64.rpm 417 kB/s | 80 kB 00:00 2024-04-21T08:31:54.450 INFO:teuthology.orchestra.run.smithi136.stdout:(41/150): libquadmath-11.4.1-3.el9.x86_64.rpm 867 kB/s | 195 kB 00:00 2024-04-21T08:31:54.475 INFO:teuthology.orchestra.run.smithi136.stdout:(42/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-04-21T08:31:54.534 INFO:teuthology.orchestra.run.smithi136.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-21T08:31:54.593 INFO:teuthology.orchestra.run.smithi136.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-21T08:31:54.660 INFO:teuthology.orchestra.run.smithi136.stdout:(45/150): python3-cryptography-36.0.1-4.el9.x86 5.9 MB/s | 1.2 MB 00:00 2024-04-21T08:31:54.718 INFO:teuthology.orchestra.run.smithi136.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-21T08:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 598 kB/s | 125 kB 00:00 2024-04-21T08:31:54.861 INFO:teuthology.orchestra.run.smithi136.stdout:(48/150): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-04-21T08:31:55.319 INFO:teuthology.orchestra.run.smithi136.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 233 kB/s | 107 kB 00:00 2024-04-21T08:31:55.428 INFO:teuthology.orchestra.run.smithi136.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 198 kB/s | 21 kB 00:00 2024-04-21T08:31:55.595 INFO:teuthology.orchestra.run.smithi136.stdout:(51/150): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-04-21T08:31:55.695 INFO:teuthology.orchestra.run.smithi136.stdout:(52/150): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-21T08:31:55.720 INFO:teuthology.orchestra.run.smithi136.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-04-21T08:31:55.746 INFO:teuthology.orchestra.run.smithi136.stdout:(54/150): automake-1.16.2-8.el9.noarch.rpm 671 kB/s | 689 kB 00:01 2024-04-21T08:31:55.770 INFO:teuthology.orchestra.run.smithi057.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-21T08:31:55.821 INFO:teuthology.orchestra.run.smithi136.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 181 kB/s | 18 kB 00:00 2024-04-21T08:31:55.838 INFO:teuthology.orchestra.run.smithi136.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-04-21T08:31:55.930 INFO:teuthology.orchestra.run.smithi136.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-04-21T08:31:56.003 INFO:teuthology.orchestra.run.smithi057.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 3.7 MB/s | 890 kB 00:00 2024-04-21T08:31:56.005 INFO:teuthology.orchestra.run.smithi136.stdout:(58/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-04-21T08:31:56.087 INFO:teuthology.orchestra.run.smithi057.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.7 MB/s | 145 kB 00:00 2024-04-21T08:31:56.130 INFO:teuthology.orchestra.run.smithi136.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-04-21T08:31:56.239 INFO:teuthology.orchestra.run.smithi136.stdout:(60/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 235 kB/s | 25 kB 00:00 2024-04-21T08:31:56.289 INFO:teuthology.orchestra.run.smithi136.stdout:(61/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.0 MB/s | 3.0 MB 00:00 2024-04-21T08:31:56.339 INFO:teuthology.orchestra.run.smithi136.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-21T08:31:56.398 INFO:teuthology.orchestra.run.smithi136.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-21T08:31:56.457 INFO:teuthology.orchestra.run.smithi136.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-04-21T08:31:56.499 INFO:teuthology.orchestra.run.smithi136.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 351 kB/s | 35 kB 00:00 2024-04-21T08:31:56.562 INFO:teuthology.orchestra.run.smithi057.stdout:(4/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 5.0 MB/s | 2.4 MB 00:00 2024-04-21T08:31:56.624 INFO:teuthology.orchestra.run.smithi136.stdout:(66/150): libtool-2.4.6-45.el9.x86_64.rpm 941 kB/s | 582 kB 00:00 2024-04-21T08:31:56.629 INFO:teuthology.orchestra.run.smithi057.stdout:(5/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 5.2 MB/s | 5.4 MB 00:01 2024-04-21T08:31:56.796 INFO:teuthology.orchestra.run.smithi057.stdout:(6/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-21T08:31:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 1.7 MB/s | 21 MB 00:12 2024-04-21T08:31:57.213 INFO:teuthology.orchestra.run.smithi057.stdout:(7/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 8.1 MB/s | 4.7 MB 00:00 2024-04-21T08:31:57.974 INFO:teuthology.orchestra.run.smithi136.stdout:(67/150): openblas-openmp-0.3.21-2.el9.x86_64.r 3.1 MB/s | 4.7 MB 00:01 2024-04-21T08:31:58.046 INFO:teuthology.orchestra.run.smithi136.stdout:(68/150): postgresql-13.14-1.el9.x86_64.rpm 1.1 MB/s | 1.6 MB 00:01 2024-04-21T08:31:58.096 INFO:teuthology.orchestra.run.smithi057.stdout:(8/150): ceph-common-19.0.0-3154.ga630765c.el9. 8.5 MB/s | 21 MB 00:02 2024-04-21T08:31:58.158 INFO:teuthology.orchestra.run.smithi136.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-21T08:31:58.163 INFO:teuthology.orchestra.run.smithi057.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 378 kB/s | 25 kB 00:00 2024-04-21T08:31:58.422 INFO:teuthology.orchestra.run.smithi057.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 8.8 MB/s | 11 MB 00:01 2024-04-21T08:31:58.505 INFO:teuthology.orchestra.run.smithi057.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 2.6 MB/s | 225 kB 00:00 2024-04-21T08:31:58.580 INFO:teuthology.orchestra.run.smithi057.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 361 kB/s | 27 kB 00:00 2024-04-21T08:31:58.774 INFO:teuthology.orchestra.run.smithi057.stdout:(13/150): ceph-osd-19.0.0-3154.ga630765c.el9.x8 8.5 MB/s | 17 MB 00:01 2024-04-21T08:31:58.798 INFO:teuthology.orchestra.run.smithi057.stdout:(14/150): libcephfs-devel-19.0.0-3154.ga630765c 146 kB/s | 32 kB 00:00 2024-04-21T08:31:58.873 INFO:teuthology.orchestra.run.smithi057.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 2.1 MB/s | 163 kB 00:00 2024-04-21T08:31:58.906 INFO:teuthology.orchestra.run.smithi057.stdout:(16/150): libcephfs2-19.0.0-3154.ga630765c.el9. 5.5 MB/s | 746 kB 00:00 2024-04-21T08:31:58.932 INFO:teuthology.orchestra.run.smithi057.stdout:(17/150): librados-devel-19.0.0-3154.ga630765c. 2.1 MB/s | 127 kB 00:00 2024-04-21T08:31:58.990 INFO:teuthology.orchestra.run.smithi057.stdout:(18/150): libradosstriper1-19.0.0-3154.ga630765 5.9 MB/s | 501 kB 00:00 2024-04-21T08:31:59.057 INFO:teuthology.orchestra.run.smithi057.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 678 kB/s | 45 kB 00:00 2024-04-21T08:31:59.141 INFO:teuthology.orchestra.run.smithi057.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 1.6 MB/s | 135 kB 00:00 2024-04-21T08:31:59.216 INFO:teuthology.orchestra.run.smithi057.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 2.1 MB/s | 162 kB 00:00 2024-04-21T08:31:59.241 INFO:teuthology.orchestra.run.smithi136.stdout:(70/150): postgresql-contrib-13.14-1.el9.x86_64 719 kB/s | 911 kB 00:01 2024-04-21T08:31:59.300 INFO:teuthology.orchestra.run.smithi057.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 3.8 MB/s | 321 kB 00:00 2024-04-21T08:31:59.414 INFO:teuthology.orchestra.run.smithi057.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 2.6 MB/s | 298 kB 00:00 2024-04-21T08:31:59.509 INFO:teuthology.orchestra.run.smithi057.stdout:(24/150): librgw2-19.0.0-3154.ga630765c.el9.x86 9.3 MB/s | 5.4 MB 00:00 2024-04-21T08:31:59.542 INFO:teuthology.orchestra.run.smithi057.stdout:(25/150): python3-rgw-19.0.0-3154.ga630765c.el9 778 kB/s | 100 kB 00:00 2024-04-21T08:31:59.568 INFO:teuthology.orchestra.run.smithi057.stdout:(26/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-21T08:31:59.652 INFO:teuthology.orchestra.run.smithi057.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-21T08:31:59.727 INFO:teuthology.orchestra.run.smithi057.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 457 kB/s | 34 kB 00:00 2024-04-21T08:31:59.783 INFO:teuthology.orchestra.run.smithi136.stdout:(71/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 1.3 MB/s | 4.1 MB 00:03 2024-04-21T08:31:59.802 INFO:teuthology.orchestra.run.smithi057.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.9 MB/s | 142 kB 00:00 2024-04-21T08:31:59.903 INFO:teuthology.orchestra.run.smithi057.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 8.6 MB/s | 3.1 MB 00:00 2024-04-21T08:31:59.975 INFO:teuthology.orchestra.run.smithi136.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 457 kB/s | 87 kB 00:00 2024-04-21T08:32:00.261 INFO:teuthology.orchestra.run.smithi057.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 7.7 MB/s | 3.5 MB 00:00 2024-04-21T08:32:00.353 INFO:teuthology.orchestra.run.smithi057.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 2.7 MB/s | 249 kB 00:00 2024-04-21T08:32:00.437 INFO:teuthology.orchestra.run.smithi057.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 592 kB/s | 49 kB 00:00 2024-04-21T08:32:00.520 INFO:teuthology.orchestra.run.smithi057.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 197 kB/s | 16 kB 00:00 2024-04-21T08:32:00.613 INFO:teuthology.orchestra.run.smithi057.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 2.9 MB/s | 273 kB 00:00 2024-04-21T08:32:00.667 INFO:teuthology.orchestra.run.smithi136.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 725 kB/s | 1.0 MB 00:01 2024-04-21T08:32:00.813 INFO:teuthology.orchestra.run.smithi057.stdout:(36/150): ceph-mgr-diskprediction-local-19.0.0- 8.1 MB/s | 7.4 MB 00:00 2024-04-21T08:32:00.838 INFO:teuthology.orchestra.run.smithi057.stdout:(37/150): cephadm-19.0.0-3154.ga630765c.el9.noa 3.3 MB/s | 762 kB 00:00 2024-04-21T08:32:00.871 INFO:teuthology.orchestra.run.smithi057.stdout:(38/150): cephfs-top-19.0.0-3154.ga630765c.el9. 313 kB/s | 18 kB 00:00 2024-04-21T08:32:00.900 INFO:teuthology.orchestra.run.smithi136.stdout:(74/150): postgresql-server-13.14-1.el9.x86_64. 2.1 MB/s | 5.8 MB 00:02 2024-04-21T08:32:00.905 INFO:teuthology.orchestra.run.smithi057.stdout:(39/150): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 80 kB 00:00 2024-04-21T08:32:00.981 INFO:teuthology.orchestra.run.smithi057.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-21T08:32:01.014 INFO:teuthology.orchestra.run.smithi057.stdout:(41/150): libgfortran-11.4.1-3.el9.x86_64.rpm 5.5 MB/s | 803 kB 00:00 2024-04-21T08:32:01.017 INFO:teuthology.orchestra.run.smithi136.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-04-21T08:32:01.060 INFO:teuthology.orchestra.run.smithi136.stdout:(76/150): python3-devel-3.9.18-3.el9.x86_64.rpm 623 kB/s | 244 kB 00:00 2024-04-21T08:32:01.118 INFO:teuthology.orchestra.run.smithi136.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 255 kB/s | 26 kB 00:00 2024-04-21T08:32:01.151 INFO:teuthology.orchestra.run.smithi136.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 209 kB/s | 19 kB 00:00 2024-04-21T08:32:01.230 INFO:teuthology.orchestra.run.smithi136.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-04-21T08:32:01.244 INFO:teuthology.orchestra.run.smithi136.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 377 kB/s | 35 kB 00:00 2024-04-21T08:32:01.514 INFO:teuthology.orchestra.run.smithi057.stdout:(42/150): ceph-test-19.0.0-3154.ga630765c.el9.x 15 MB/s | 49 MB 00:03 2024-04-21T08:32:01.539 INFO:teuthology.orchestra.run.smithi057.stdout:(43/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 453 kB/s | 253 kB 00:00 2024-04-21T08:32:01.586 INFO:teuthology.orchestra.run.smithi136.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 734 kB/s | 250 kB 00:00 2024-04-21T08:32:01.598 INFO:teuthology.orchestra.run.smithi057.stdout:(44/150): python3-cryptography-36.0.1-4.el9.x86 2.1 MB/s | 1.2 MB 00:00 2024-04-21T08:32:01.623 INFO:teuthology.orchestra.run.smithi057.stdout:(45/150): python3-ply-3.11-14.el9.noarch.rpm 983 kB/s | 106 kB 00:00 2024-04-21T08:32:01.649 INFO:teuthology.orchestra.run.smithi057.stdout:(46/150): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-21T08:32:01.691 INFO:teuthology.orchestra.run.smithi057.stdout:(47/150): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-04-21T08:32:01.741 INFO:teuthology.orchestra.run.smithi057.stdout:(48/150): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-21T08:32:01.928 INFO:teuthology.orchestra.run.smithi136.stdout:(82/150): python3-oauthlib-3.1.1-5.el9.noarch.r 650 kB/s | 222 kB 00:00 2024-04-21T08:32:02.120 INFO:teuthology.orchestra.run.smithi136.stdout:(83/150): python3-packaging-20.9-5.el9.noarch.r 402 kB/s | 77 kB 00:00 2024-04-21T08:32:02.220 INFO:teuthology.orchestra.run.smithi136.stdout:(84/150): python3-prettytable-0.7.2-27.el9.noar 423 kB/s | 42 kB 00:00 2024-04-21T08:32:02.233 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi057.stdout:(49/150): boost-program-options-1.75.0-8.el9.x8 144 kB/s | 107 kB 00:00 2024-04-21T08:32:02.679 INFO:teuthology.orchestra.run.smithi136.stdout:(85/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.5 MB/s | 5.1 MB 00:01 2024-04-21T08:32:02.704 INFO:teuthology.orchestra.run.smithi136.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 557 kB/s | 269 kB 00:00 2024-04-21T08:32:02.804 INFO:teuthology.orchestra.run.smithi136.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-21T08:32:02.938 INFO:teuthology.orchestra.run.smithi136.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 383 kB/s | 51 kB 00:00 2024-04-21T08:32:03.049 INFO:teuthology.orchestra.run.smithi057.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 38 kB/s | 21 kB 00:00 2024-04-21T08:32:03.063 INFO:teuthology.orchestra.run.smithi136.stdout:(89/150): python3-requests-oauthlib-1.3.0-12.el 430 kB/s | 54 kB 00:00 2024-04-21T08:32:03.133 INFO:teuthology.orchestra.run.smithi057.stdout:(51/150): automake-1.16.2-8.el9.noarch.rpm 464 kB/s | 689 kB 00:01 2024-04-21T08:32:03.280 INFO:teuthology.orchestra.run.smithi136.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 484 kB/s | 279 kB 00:00 2024-04-21T08:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 417 kB/s | 42 kB 00:00 2024-04-21T08:32:03.458 INFO:teuthology.orchestra.run.smithi057.stdout:(52/150): bison-3.7.4-5.el9.x86_64.rpm 533 kB/s | 943 kB 00:01 2024-04-21T08:32:03.481 INFO:teuthology.orchestra.run.smithi136.stdout:(92/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-04-21T08:32:03.758 INFO:teuthology.orchestra.run.smithi057.stdout:(53/150): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-04-21T08:32:03.823 INFO:teuthology.orchestra.run.smithi136.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 646 kB/s | 221 kB 00:00 2024-04-21T08:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout:(94/150): qatlib-service-24.02.0-1.el9.x86_64.r 330 kB/s | 36 kB 00:00 2024-04-21T08:32:04.032 INFO:teuthology.orchestra.run.smithi136.stdout:(95/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-04-21T08:32:04.117 INFO:teuthology.orchestra.run.smithi057.stdout:(54/150): flex-2.6.4-9.el9.x86_64.rpm 294 kB/s | 313 kB 00:01 2024-04-21T08:32:04.301 INFO:teuthology.orchestra.run.smithi057.stdout:(55/150): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-04-21T08:32:04.402 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:32:04.402 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-21T08:32:04.412 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-21T08:32:04.413 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-21T08:32:04.414 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-21T08:32:04.415 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-21T08:32:04.416 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-21T08:32:04.419 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-21T08:32:04.420 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-21T08:32:04.457 INFO:teuthology.orchestra.run.smithi136.stdout:(96/150): socat-1.7.4.1-5.el9.x86_64.rpm 717 kB/s | 305 kB 00:00 2024-04-21T08:32:04.479 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-04-21T08:32:04.494 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-04-21T08:32:04.494 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-3154.ga630765c.el9 ceph 6.6 k 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 890 k 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3154.ga630765c.el9 ceph 145 k 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-3154.ga630765c.el9 ceph 1.5 M 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 142 k 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 3.5 M 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 7.4 M 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 49 k 2024-04-21T08:32:04.495 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 11 M 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-3154.ga630765c.el9 ceph 49 M 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 273 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 762 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 225 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell x86_64 2:19.0.0-3154.ga630765c.el9 ceph 27 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 18 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 32 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 746 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-3154.ga630765c.el9 ceph 127 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-21T08:32:04.496 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-3154.ga630765c.el9 ceph 162 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-3154.ga630765c.el9 ceph 321 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 298 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-3154.ga630765c.el9 ceph 100 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 85 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 171 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-04-21T08:32:04.497 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.4 M 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 3.1 M 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 21 M 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 34 k 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-3154.ga630765c.el9 ceph 2.4 M 2024-04-21T08:32:04.498 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 249 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-3154.ga630765c.el9 ceph 4.7 M 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-3154.ga630765c.el9 ceph 17 M 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3154.ga630765c.el9 ceph-noarch 16 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-3154.ga630765c.el9 ceph 25 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-21T08:32:04.499 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-3154.ga630765c.el9 ceph 163 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 501 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-3154.ga630765c.el9 ceph 5.4 M 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-21T08:32:04.500 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-21T08:32:04.501 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.0.0-3154.ga630765c.el9 ceph 45 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-3154.ga630765c.el9 ceph 135 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-21T08:32:04.502 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-21T08:32:04.503 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-21T08:32:04.504 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-21T08:32:04.505 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-21T08:32:04.506 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout:Install 148 Packages 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-04-21T08:32:04.507 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:32:04.508 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 218 M 2024-04-21T08:32:04.508 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-04-21T08:32:04.582 INFO:teuthology.orchestra.run.smithi136.stdout:(97/150): uuid-1.6.2-55.el9.x86_64.rpm 463 kB/s | 58 kB 00:00 2024-04-21T08:32:04.626 INFO:teuthology.orchestra.run.smithi057.stdout:(56/150): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-04-21T08:32:04.783 INFO:teuthology.orchestra.run.smithi136.stdout:(98/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 468 kB/s | 94 kB 00:00 2024-04-21T08:32:04.983 INFO:teuthology.orchestra.run.smithi136.stdout:(99/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 318 kB/s | 64 kB 00:00 2024-04-21T08:32:05.083 INFO:teuthology.orchestra.run.smithi136.stdout:(100/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 448 kB/s | 45 kB 00:00 2024-04-21T08:32:05.109 INFO:teuthology.orchestra.run.smithi057.stdout:(57/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 198 kB/s | 160 kB 00:00 2024-04-21T08:32:05.278 INFO:teuthology.orchestra.run.smithi057.stdout:(58/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 69 kB/s | 45 kB 00:00 2024-04-21T08:32:05.475 INFO:teuthology.orchestra.run.smithi136.stdout:(101/150): lua-devel-5.4.4-4.el9.x86_64.rpm 57 kB/s | 22 kB 00:00 2024-04-21T08:32:05.617 INFO:teuthology.orchestra.run.smithi136.stdout:(102/150): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-04-21T08:32:05.892 INFO:teuthology.orchestra.run.smithi136.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-21T08:32:06.076 INFO:teuthology.orchestra.run.smithi136.stdout:(104/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.4 MB/s | 16 MB 00:03 2024-04-21T08:32:06.101 INFO:teuthology.orchestra.run.smithi136.stdout:(105/150): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-04-21T08:32:06.235 INFO:teuthology.orchestra.run.smithi136.stdout:(106/150): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-04-21T08:32:06.318 INFO:teuthology.orchestra.run.smithi136.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 313 kB/s | 26 kB 00:00 2024-04-21T08:32:06.402 INFO:teuthology.orchestra.run.smithi136.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-04-21T08:32:06.469 INFO:teuthology.orchestra.run.smithi136.stdout:(109/150): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-21T08:32:06.494 INFO:teuthology.orchestra.run.smithi136.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-04-21T08:32:06.578 INFO:teuthology.orchestra.run.smithi136.stdout:(111/150): python3-babel-2.9.1-2.el9.noarch.rpm 925 kB/s | 6.0 MB 00:06 2024-04-21T08:32:06.586 INFO:teuthology.orchestra.run.smithi057.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 449 kB/s | 662 kB 00:01 2024-04-21T08:32:06.595 INFO:teuthology.orchestra.run.smithi136.stdout:(112/150): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-21T08:32:06.612 INFO:teuthology.orchestra.run.smithi057.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 437 kB/s | 582 kB 00:01 2024-04-21T08:32:06.687 INFO:teuthology.orchestra.run.smithi136.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-04-21T08:32:06.754 INFO:teuthology.orchestra.run.smithi136.stdout:(114/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-04-21T08:32:06.779 INFO:teuthology.orchestra.run.smithi136.stdout:(115/150): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-04-21T08:32:06.996 INFO:teuthology.orchestra.run.smithi136.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 62 kB/s | 15 kB 00:00 2024-04-21T08:32:07.021 INFO:teuthology.orchestra.run.smithi136.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 710 kB/s | 172 kB 00:00 2024-04-21T08:32:07.047 INFO:teuthology.orchestra.run.smithi136.stdout:(118/150): python3-asyncssh-2.13.2-5.el9.noarch 1.1 MB/s | 548 kB 00:00 2024-04-21T08:32:07.147 INFO:teuthology.orchestra.run.smithi136.stdout:(119/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-21T08:32:07.170 INFO:teuthology.orchestra.run.smithi057.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-04-21T08:32:07.172 INFO:teuthology.orchestra.run.smithi136.stdout:(120/150): python3-cherrypy-18.6.1-2.el9.noarch 2.0 MB/s | 358 kB 00:00 2024-04-21T08:32:07.198 INFO:teuthology.orchestra.run.smithi136.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 234 kB/s | 35 kB 00:00 2024-04-21T08:32:07.240 INFO:teuthology.orchestra.run.smithi136.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 2.1 MB/s | 203 kB 00:00 2024-04-21T08:32:07.245 INFO:teuthology.orchestra.run.smithi057.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 75 kB/s | 47 kB 00:00 2024-04-21T08:32:07.282 INFO:teuthology.orchestra.run.smithi136.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-21T08:32:07.332 INFO:teuthology.orchestra.run.smithi136.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-21T08:32:07.382 INFO:teuthology.orchestra.run.smithi136.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-21T08:32:07.424 INFO:teuthology.orchestra.run.smithi136.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-21T08:32:07.466 INFO:teuthology.orchestra.run.smithi136.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-21T08:32:07.500 INFO:teuthology.orchestra.run.smithi136.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-21T08:32:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:(129/150): python3-grpcio-1.46.7-10.el9.x86_64. 5.0 MB/s | 2.0 MB 00:00 2024-04-21T08:32:07.608 INFO:teuthology.orchestra.run.smithi136.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-04-21T08:32:07.642 INFO:teuthology.orchestra.run.smithi136.stdout:(131/150): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-21T08:32:07.667 INFO:teuthology.orchestra.run.smithi136.stdout:(132/150): python3-more-itertools-8.12.0-2.el9. 943 kB/s | 79 kB 00:00 2024-04-21T08:32:07.734 INFO:teuthology.orchestra.run.smithi136.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 460 kB/s | 58 kB 00:00 2024-04-21T08:32:07.771 INFO:teuthology.orchestra.run.smithi057.stdout:(63/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 710 kB/s | 3.0 MB 00:04 2024-04-21T08:32:07.793 INFO:teuthology.orchestra.run.smithi136.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-04-21T08:32:07.818 INFO:teuthology.orchestra.run.smithi136.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-04-21T08:32:07.844 INFO:teuthology.orchestra.run.smithi136.stdout:(136/150): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-21T08:32:07.902 INFO:teuthology.orchestra.run.smithi136.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-04-21T08:32:07.919 INFO:teuthology.orchestra.run.smithi136.stdout:(138/150): python3-pyperclip-1.8.2-2.el9.noarch 224 kB/s | 28 kB 00:00 2024-04-21T08:32:07.945 INFO:teuthology.orchestra.run.smithi136.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-21T08:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 878 kB/s | 59 kB 00:00 2024-04-21T08:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-21T08:32:08.079 INFO:teuthology.orchestra.run.smithi136.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-21T08:32:08.104 INFO:teuthology.orchestra.run.smithi136.stdout:(143/150): python3-typing-extensions-4.4.0-2.el 320 kB/s | 51 kB 00:00 2024-04-21T08:32:08.113 INFO:teuthology.orchestra.run.smithi057.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 217 kB/s | 188 kB 00:00 2024-04-21T08:32:08.129 INFO:teuthology.orchestra.run.smithi136.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 670 kB/s | 90 kB 00:00 2024-04-21T08:32:08.163 INFO:teuthology.orchestra.run.smithi057.stdout:(65/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 295 kB/s | 292 kB 00:00 2024-04-21T08:32:08.171 INFO:teuthology.orchestra.run.smithi136.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-21T08:32:08.197 INFO:teuthology.orchestra.run.smithi136.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-04-21T08:32:08.222 INFO:teuthology.orchestra.run.smithi136.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-21T08:32:08.330 INFO:teuthology.orchestra.run.smithi136.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-21T08:32:08.397 INFO:teuthology.orchestra.run.smithi057.stdout:(66/150): openblas-0.3.21-2.el9.x86_64.rpm 56 kB/s | 35 kB 00:00 2024-04-21T08:32:08.714 INFO:teuthology.orchestra.run.smithi136.stdout:(149/150): librados2-19.0.0-3154.ga630765c.el9. 6.6 MB/s | 3.4 MB 00:00 2024-04-21T08:32:08.826 INFO:teuthology.orchestra.run.smithi129.stdout:(1/150): ceph-19.0.0-3154.ga630765c.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-21T08:32:09.139 INFO:teuthology.orchestra.run.smithi136.stdout:(150/150): librbd1-19.0.0-3154.ga630765c.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-04-21T08:32:09.142 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:32:09.142 INFO:teuthology.orchestra.run.smithi136.stdout:Total 9.3 MB/s | 218 MB 00:23 2024-04-21T08:32:09.335 INFO:teuthology.orchestra.run.smithi129.stdout:(2/150): ceph-fuse-19.0.0-3154.ga630765c.el9.x8 1.7 MB/s | 890 kB 00:00 2024-04-21T08:32:09.426 INFO:teuthology.orchestra.run.smithi129.stdout:(3/150): ceph-immutable-object-cache-19.0.0-315 1.6 MB/s | 145 kB 00:00 2024-04-21T08:32:10.185 INFO:teuthology.orchestra.run.smithi129.stdout:(4/150): ceph-mds-19.0.0-3154.ga630765c.el9.x86 3.1 MB/s | 2.4 MB 00:00 2024-04-21T08:32:10.220 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-04-21T08:32:10.346 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-04-21T08:32:10.347 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-04-21T08:32:10.627 INFO:teuthology.orchestra.run.smithi129.stdout:(5/150): ceph-mgr-19.0.0-3154.ga630765c.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-04-21T08:32:11.122 INFO:teuthology.orchestra.run.smithi057.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 603 kB/s | 1.6 MB 00:02 2024-04-21T08:32:11.635 INFO:teuthology.orchestra.run.smithi129.stdout:(6/150): ceph-mon-19.0.0-3154.ga630765c.el9.x86 4.7 MB/s | 4.7 MB 00:01 2024-04-21T08:32:11.894 INFO:teuthology.orchestra.run.smithi129.stdout:(7/150): ceph-base-19.0.0-3154.ga630765c.el9.x8 1.6 MB/s | 5.4 MB 00:03 2024-04-21T08:32:12.042 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-04-21T08:32:12.042 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-04-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi057.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 502 kB/s | 911 kB 00:01 2024-04-21T08:32:13.697 INFO:teuthology.orchestra.run.smithi057.stdout:(69/150): postgresql-private-libs-13.14-1.el9.x 176 kB/s | 133 kB 00:00 2024-04-21T08:32:14.105 INFO:teuthology.orchestra.run.smithi057.stdout:(70/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 712 kB/s | 4.1 MB 00:05 2024-04-21T08:32:14.680 INFO:teuthology.orchestra.run.smithi057.stdout:(71/150): openblas-openmp-0.3.21-2.el9.x86_64.r 739 kB/s | 4.7 MB 00:06 2024-04-21T08:32:15.381 INFO:teuthology.orchestra.run.smithi057.stdout:(72/150): python3-attrs-20.3.0-7.el9.noarch.rpm 125 kB/s | 87 kB 00:00 2024-04-21T08:32:15.844 INFO:teuthology.orchestra.run.smithi129.stdout:(8/150): ceph-osd-19.0.0-3154.ga630765c.el9.x86 4.0 MB/s | 17 MB 00:04 2024-04-21T08:32:15.902 INFO:teuthology.orchestra.run.smithi129.stdout:(9/150): ceph-selinux-19.0.0-3154.ga630765c.el9 433 kB/s | 25 kB 00:00 2024-04-21T08:32:16.014 INFO:teuthology.orchestra.run.smithi057.stdout:(73/150): protobuf-3.14.0-13.el9.x86_64.rpm 541 kB/s | 1.0 MB 00:01 2024-04-21T08:32:16.948 INFO:teuthology.orchestra.run.smithi057.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 262 kB/s | 244 kB 00:00 2024-04-21T08:32:16.952 INFO:teuthology.orchestra.run.smithi129.stdout:(10/150): ceph-radosgw-19.0.0-3154.ga630765c.el 2.1 MB/s | 11 MB 00:05 2024-04-21T08:32:17.094 INFO:teuthology.orchestra.run.smithi129.stdout:(11/150): cephfs-mirror-19.0.0-3154.ga630765c.e 1.6 MB/s | 225 kB 00:00 2024-04-21T08:32:17.161 INFO:teuthology.orchestra.run.smithi129.stdout:(12/150): cephfs-shell-19.0.0-3154.ga630765c.el 406 kB/s | 27 kB 00:00 2024-04-21T08:32:17.245 INFO:teuthology.orchestra.run.smithi129.stdout:(13/150): libcephfs-devel-19.0.0-3154.ga630765c 380 kB/s | 32 kB 00:00 2024-04-21T08:32:17.711 INFO:teuthology.orchestra.run.smithi129.stdout:(14/150): libcephfs2-19.0.0-3154.ga630765c.el9. 1.6 MB/s | 746 kB 00:00 2024-04-21T08:32:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:(15/150): libcephsqlite-19.0.0-3154.ga630765c.e 1.4 MB/s | 163 kB 00:00 2024-04-21T08:32:17.873 INFO:teuthology.orchestra.run.smithi057.stdout:(75/150): python3-jinja2-2.11.3-5.el9.noarch.rp 268 kB/s | 248 kB 00:00 2024-04-21T08:32:18.012 INFO:teuthology.orchestra.run.smithi129.stdout:(16/150): librados-devel-19.0.0-3154.ga630765c. 692 kB/s | 127 kB 00:00 2024-04-21T08:32:18.044 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-04-21T08:32:18.117 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:32:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:32:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:32:18.496 INFO:teuthology.orchestra.run.smithi129.stdout:(17/150): ceph-common-19.0.0-3154.ga630765c.el9 2.1 MB/s | 21 MB 00:10 2024-04-21T08:32:18.498 INFO:teuthology.orchestra.run.smithi057.stdout:(76/150): python3-jsonpatch-1.21-16.el9.noarch. 41 kB/s | 26 kB 00:00 2024-04-21T08:32:18.529 INFO:teuthology.orchestra.run.smithi129.stdout:(18/150): libradosstriper1-19.0.0-3154.ga630765 968 kB/s | 501 kB 00:00 2024-04-21T08:32:18.646 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:32:18.655 INFO:teuthology.orchestra.run.smithi129.stdout:(19/150): python3-ceph-argparse-19.0.0-3154.ga6 361 kB/s | 45 kB 00:00 2024-04-21T08:32:18.768 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:18.863 INFO:teuthology.orchestra.run.smithi129.stdout:(20/150): python3-ceph-common-19.0.0-3154.ga630 647 kB/s | 135 kB 00:00 2024-04-21T08:32:18.880 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:18.902 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:18.985 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:19.027 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:19.056 INFO:teuthology.orchestra.run.smithi057.stdout:(77/150): python3-jsonpointer-2.0-4.el9.noarch. 34 kB/s | 19 kB 00:00 2024-04-21T08:32:19.057 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:32:19.063 INFO:teuthology.orchestra.run.smithi129.stdout:(21/150): python3-cephfs-19.0.0-3154.ga630765c. 809 kB/s | 162 kB 00:00 2024-04-21T08:32:19.082 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:32:19.296 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:32:19.324 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:19.389 INFO:teuthology.orchestra.run.smithi129.stdout:(22/150): python3-rados-19.0.0-3154.ga630765c.e 988 kB/s | 321 kB 00:00 2024-04-21T08:32:19.427 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:19.450 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:19.535 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:19.558 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:19.681 INFO:teuthology.orchestra.run.smithi129.stdout:(23/150): python3-rbd-19.0.0-3154.ga630765c.el9 1.0 MB/s | 298 kB 00:00 2024-04-21T08:32:19.744 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:19.776 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:19.781 INFO:teuthology.orchestra.run.smithi129.stdout:(24/150): python3-rgw-19.0.0-3154.ga630765c.el9 995 kB/s | 100 kB 00:00 2024-04-21T08:32:19.882 INFO:teuthology.orchestra.run.smithi129.stdout:(25/150): rbd-fuse-19.0.0-3154.ga630765c.el9.x8 851 kB/s | 85 kB 00:00 2024-04-21T08:32:19.883 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:32:19.890 INFO:teuthology.orchestra.run.smithi057.stdout:(78/150): python3-mako-1.1.4-6.el9.noarch.rpm 207 kB/s | 172 kB 00:00 2024-04-21T08:32:19.979 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:32:20.009 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:32:20.043 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:32:20.108 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:32:20.143 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:32:20.186 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:32:20.219 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:32:20.247 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:32:20.315 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:32:20.369 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:32:20.400 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:32:20.435 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:32:20.490 INFO:teuthology.orchestra.run.smithi057.stdout:(79/150): python3-markupsafe-1.1.1-12.el9.x86_6 58 kB/s | 35 kB 00:00 2024-04-21T08:32:20.511 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:32:20.555 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:32:20.589 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:32:20.619 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:32:20.700 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:32:20.742 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:32:20.818 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:32:20.952 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:32:21.003 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:32:21.374 INFO:teuthology.orchestra.run.smithi129.stdout:(26/150): librgw2-19.0.0-3154.ga630765c.el9.x86 1.9 MB/s | 5.4 MB 00:02 2024-04-21T08:32:21.524 INFO:teuthology.orchestra.run.smithi129.stdout:(27/150): rbd-nbd-19.0.0-3154.ga630765c.el9.x86 1.1 MB/s | 171 kB 00:00 2024-04-21T08:32:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:32:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:32:21.649 INFO:teuthology.orchestra.run.smithi129.stdout:(28/150): ceph-grafana-dashboards-19.0.0-3154.g 273 kB/s | 34 kB 00:00 2024-04-21T08:32:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:32:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:32:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:32:21.774 INFO:teuthology.orchestra.run.smithi129.stdout:(29/150): ceph-mgr-cephadm-19.0.0-3154.ga630765 1.1 MB/s | 142 kB 00:00 2024-04-21T08:32:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:32:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:32:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:32:21.917 INFO:teuthology.orchestra.run.smithi129.stdout:(30/150): rbd-mirror-19.0.0-3154.ga630765c.el9. 1.5 MB/s | 3.1 MB 00:02 2024-04-21T08:32:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:32:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:32:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:32:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:32:22.207 INFO:teuthology.orchestra.run.smithi057.stdout:(80/150): postgresql-server-13.14-1.el9.x86_64. 695 kB/s | 5.8 MB 00:08 2024-04-21T08:32:22.286 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:32:22.328 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:32:22.358 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:32:22.400 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:32:22.495 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:32:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:32:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:32:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:32:22.615 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:32:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:32:22.819 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:32:22.856 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:32:22.884 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:32:22.919 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:32:22.964 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:32:23.001 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:32:23.041 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:32:23.071 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:32:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:32:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:32:23.157 INFO:teuthology.orchestra.run.smithi057.stdout:(81/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 264 kB/s | 250 kB 00:00 2024-04-21T08:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:32:23.310 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:32:23.353 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:32:23.384 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:32:23.432 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:32:23.433 INFO:teuthology.orchestra.run.smithi129.stdout:(31/150): ceph-mgr-dashboard-19.0.0-3154.ga6307 2.1 MB/s | 3.5 MB 00:01 2024-04-21T08:32:23.531 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:32:23.587 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:32:23.600 INFO:teuthology.orchestra.run.smithi129.stdout:(32/150): ceph-mgr-modules-core-19.0.0-3154.ga6 1.5 MB/s | 249 kB 00:00 2024-04-21T08:32:23.709 INFO:teuthology.orchestra.run.smithi129.stdout:(33/150): ceph-mgr-rook-19.0.0-3154.ga630765c.e 455 kB/s | 49 kB 00:00 2024-04-21T08:32:23.791 INFO:teuthology.orchestra.run.smithi057.stdout:(82/150): python3-babel-2.9.1-2.el9.noarch.rpm 726 kB/s | 6.0 MB 00:08 2024-04-21T08:32:23.792 INFO:teuthology.orchestra.run.smithi129.stdout:(34/150): ceph-prometheus-alerts-19.0.0-3154.ga 197 kB/s | 16 kB 00:00 2024-04-21T08:32:23.959 INFO:teuthology.orchestra.run.smithi129.stdout:(35/150): ceph-volume-19.0.0-3154.ga630765c.el9 1.6 MB/s | 273 kB 00:00 2024-04-21T08:32:24.099 INFO:teuthology.orchestra.run.smithi057.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 236 kB/s | 222 kB 00:00 2024-04-21T08:32:24.321 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:32:24.359 INFO:teuthology.orchestra.run.smithi129.stdout:(36/150): cephadm-19.0.0-3154.ga630765c.el9.noa 1.9 MB/s | 762 kB 00:00 2024-04-21T08:32:24.367 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:32:24.401 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:32:24.451 INFO:teuthology.orchestra.run.smithi129.stdout:(37/150): cephfs-top-19.0.0-3154.ga630765c.el9. 200 kB/s | 18 kB 00:00 2024-04-21T08:32:24.505 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:32:24.507 INFO:teuthology.orchestra.run.smithi057.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 108 kB/s | 77 kB 00:00 2024-04-21T08:32:24.510 INFO:teuthology.orchestra.run.smithi129.stdout:(38/150): fuse-2.9.9-15.el9.x86_64.rpm 1.4 MB/s | 80 kB 00:00 2024-04-21T08:32:24.559 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:32:24.610 INFO:teuthology.orchestra.run.smithi129.stdout:(39/150): libgfortran-11.4.1-3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2024-04-21T08:32:24.657 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:32:24.685 INFO:teuthology.orchestra.run.smithi129.stdout:(40/150): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-21T08:32:24.695 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:32:24.733 INFO:teuthology.orchestra.run.smithi057.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 67 kB/s | 42 kB 00:00 2024-04-21T08:32:24.761 INFO:teuthology.orchestra.run.smithi129.stdout:(41/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-21T08:32:24.871 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:32:24.878 INFO:teuthology.orchestra.run.smithi129.stdout:(42/150): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-21T08:32:24.901 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:32:24.945 INFO:teuthology.orchestra.run.smithi129.stdout:(43/150): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-21T08:32:24.979 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:32:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:(44/150): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-21T08:32:25.095 INFO:teuthology.orchestra.run.smithi129.stdout:(45/150): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-21T08:32:25.171 INFO:teuthology.orchestra.run.smithi129.stdout:(46/150): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-21T08:32:25.521 INFO:teuthology.orchestra.run.smithi129.stdout:(47/150): ceph-mgr-diskprediction-local-19.0.0- 2.1 MB/s | 7.4 MB 00:03 2024-04-21T08:32:25.555 INFO:teuthology.orchestra.run.smithi129.stdout:(48/150): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 689 kB 00:00 2024-04-21T08:32:25.616 INFO:teuthology.orchestra.run.smithi057.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 243 kB/s | 269 kB 00:01 2024-04-21T08:32:25.632 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:32:25.633 INFO:teuthology.orchestra.run.smithi057.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 182 kB/s | 159 kB 00:00 2024-04-21T08:32:25.688 INFO:teuthology.orchestra.run.smithi129.stdout:(49/150): bison-3.7.4-5.el9.x86_64.rpm 5.5 MB/s | 943 kB 00:00 2024-04-21T08:32:25.743 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:32:25.747 INFO:teuthology.orchestra.run.smithi129.stdout:(50/150): elfutils-libelf-devel-0.190-2.el9.x86 369 kB/s | 21 kB 00:00 2024-04-21T08:32:25.772 INFO:teuthology.orchestra.run.smithi129.stdout:(51/150): boost-program-options-1.75.0-8.el9.x8 492 kB/s | 107 kB 00:00 2024-04-21T08:32:25.831 INFO:teuthology.orchestra.run.smithi129.stdout:(52/150): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-21T08:32:25.856 INFO:teuthology.orchestra.run.smithi129.stdout:(53/150): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-04-21T08:32:25.915 INFO:teuthology.orchestra.run.smithi129.stdout:(54/150): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-21T08:32:25.973 INFO:teuthology.orchestra.run.smithi129.stdout:(55/150): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-04-21T08:32:26.074 INFO:teuthology.orchestra.run.smithi129.stdout:(56/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-21T08:32:26.140 INFO:teuthology.orchestra.run.smithi129.stdout:(57/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 679 kB/s | 45 kB 00:00 2024-04-21T08:32:26.199 INFO:teuthology.orchestra.run.smithi129.stdout:(58/150): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.1 MB/s | 3.0 MB 00:00 2024-04-21T08:32:26.266 INFO:teuthology.orchestra.run.smithi129.stdout:(59/150): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-21T08:32:26.284 INFO:teuthology.orchestra.run.smithi057.stdout:(88/150): python3-pytz-2021.1-5.el9.noarch.rpm 78 kB/s | 51 kB 00:00 2024-04-21T08:32:26.300 INFO:teuthology.orchestra.run.smithi129.stdout:(60/150): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-21T08:32:26.342 INFO:teuthology.orchestra.run.smithi129.stdout:(61/150): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-04-21T08:32:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:(62/150): libzstd-devel-1.5.1-2.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-04-21T08:32:26.417 INFO:teuthology.orchestra.run.smithi129.stdout:(63/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-21T08:32:26.442 INFO:teuthology.orchestra.run.smithi129.stdout:(64/150): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-21T08:32:26.484 INFO:teuthology.orchestra.run.smithi129.stdout:(65/150): openblas-0.3.21-2.el9.x86_64.rpm 526 kB/s | 35 kB 00:00 2024-04-21T08:32:26.567 INFO:teuthology.orchestra.run.smithi057.stdout:(89/150): python3-pyasn1-modules-0.4.8-6.el9.no 293 kB/s | 279 kB 00:00 2024-04-21T08:32:26.610 INFO:teuthology.orchestra.run.smithi129.stdout:(66/150): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-04-21T08:32:26.718 INFO:teuthology.orchestra.run.smithi129.stdout:(67/150): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-21T08:32:26.793 INFO:teuthology.orchestra.run.smithi129.stdout:(68/150): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-21T08:32:26.852 INFO:teuthology.orchestra.run.smithi129.stdout:(69/150): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-04-21T08:32:26.894 INFO:teuthology.orchestra.run.smithi129.stdout:(70/150): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-21T08:32:26.959 INFO:teuthology.orchestra.run.smithi057.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 79 kB/s | 54 kB 00:00 2024-04-21T08:32:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:32:27.037 INFO:teuthology.orchestra.run.smithi129.stdout:(71/150): postgresql-server-13.14-1.el9.x86_64. 32 MB/s | 5.8 MB 00:00 2024-04-21T08:32:27.069 INFO:teuthology.orchestra.run.smithi129.stdout:(72/150): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-04-21T08:32:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:32:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:32:27.120 INFO:teuthology.orchestra.run.smithi129.stdout:(73/150): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-04-21T08:32:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:32:27.221 INFO:teuthology.orchestra.run.smithi129.stdout:(74/150): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-21T08:32:27.296 INFO:teuthology.orchestra.run.smithi129.stdout:(75/150): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-04-21T08:32:27.346 INFO:teuthology.orchestra.run.smithi129.stdout:(76/150): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-04-21T08:32:27.371 INFO:teuthology.orchestra.run.smithi129.stdout:(77/150): python3-jsonpatch-1.21-16.el9.noarch. 341 kB/s | 26 kB 00:00 2024-04-21T08:32:27.413 INFO:teuthology.orchestra.run.smithi129.stdout:(78/150): python3-jsonpointer-2.0-4.el9.noarch. 285 kB/s | 19 kB 00:00 2024-04-21T08:32:27.439 INFO:teuthology.orchestra.run.smithi129.stdout:(79/150): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-04-21T08:32:27.457 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:32:27.480 INFO:teuthology.orchestra.run.smithi129.stdout:(80/150): python3-markupsafe-1.1.1-12.el9.x86_6 520 kB/s | 35 kB 00:00 2024-04-21T08:32:27.497 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:27.606 INFO:teuthology.orchestra.run.smithi129.stdout:(81/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 5.1 MB 00:00 2024-04-21T08:32:27.609 INFO:teuthology.orchestra.run.smithi057.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-04-21T08:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:27.631 INFO:teuthology.orchestra.run.smithi129.stdout:(82/150): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-04-21T08:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:32:27.681 INFO:teuthology.orchestra.run.smithi129.stdout:(83/150): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-21T08:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:32:27.723 INFO:teuthology.orchestra.run.smithi129.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 838 kB/s | 77 kB 00:00 2024-04-21T08:32:27.749 INFO:teuthology.orchestra.run.smithi129.stdout:(85/150): python3-prettytable-0.7.2-27.el9.noar 632 kB/s | 42 kB 00:00 2024-04-21T08:32:27.799 INFO:teuthology.orchestra.run.smithi129.stdout:(86/150): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-21T08:32:27.824 INFO:teuthology.orchestra.run.smithi129.stdout:(87/150): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-21T08:32:27.943 INFO:teuthology.orchestra.run.smithi057.stdout:(92/150): python3-numpy-1.20.1-5.el9.x86_64.rpm 700 kB/s | 5.1 MB 00:07 2024-04-21T08:32:28.182 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:32:28.217 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:28.260 INFO:teuthology.orchestra.run.smithi057.stdout:(93/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 68 kB/s | 44 kB 00:00 2024-04-21T08:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:32:28.333 INFO:teuthology.orchestra.run.smithi129.stdout:(88/150): ceph-test-19.0.0-3154.ga630765c.el9.x 3.9 MB/s | 49 MB 00:12 2024-04-21T08:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout:(89/150): python3-pytz-2021.1-5.el9.noarch.rpm 95 kB/s | 51 kB 00:00 2024-04-21T08:32:28.383 INFO:teuthology.orchestra.run.smithi129.stdout:(90/150): python3-pyasn1-modules-0.4.8-6.el9.no 477 kB/s | 279 kB 00:00 2024-04-21T08:32:28.409 INFO:teuthology.orchestra.run.smithi129.stdout:(91/150): python3-requests-oauthlib-1.3.0-12.el 714 kB/s | 54 kB 00:00 2024-04-21T08:32:28.459 INFO:teuthology.orchestra.run.smithi129.stdout:(92/150): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-04-21T08:32:28.592 INFO:teuthology.orchestra.run.smithi129.stdout:(93/150): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-21T08:32:28.618 INFO:teuthology.orchestra.run.smithi129.stdout:(94/150): python3-wcwidth-0.2.5-8.el9.noarch.rp 212 kB/s | 44 kB 00:00 2024-04-21T08:32:28.660 INFO:teuthology.orchestra.run.smithi129.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 537 kB/s | 36 kB 00:00 2024-04-21T08:32:28.868 INFO:teuthology.orchestra.run.smithi057.stdout:(94/150): qatlib-24.02.0-1.el9.x86_64.rpm 239 kB/s | 221 kB 00:00 2024-04-21T08:32:28.885 INFO:teuthology.orchestra.run.smithi129.stdout:(96/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-04-21T08:32:28.894 INFO:teuthology.orchestra.run.smithi057.stdout:(95/150): qatlib-service-24.02.0-1.el9.x86_64.r 57 kB/s | 36 kB 00:00 2024-04-21T08:32:28.910 INFO:teuthology.orchestra.run.smithi129.stdout:(97/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 162 kB/s | 47 kB 00:00 2024-04-21T08:32:28.952 INFO:teuthology.orchestra.run.smithi129.stdout:(98/150): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-04-21T08:32:28.977 INFO:teuthology.orchestra.run.smithi129.stdout:(99/150): uuid-1.6.2-55.el9.x86_64.rpm 630 kB/s | 58 kB 00:00 2024-04-21T08:32:29.003 INFO:teuthology.orchestra.run.smithi129.stdout:(100/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-04-21T08:32:29.027 INFO:teuthology.orchestra.run.smithi129.stdout:(101/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 846 kB/s | 64 kB 00:00 2024-04-21T08:32:29.055 INFO:teuthology.orchestra.run.smithi129.stdout:(102/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 578 kB/s | 45 kB 00:00 2024-04-21T08:32:29.080 INFO:teuthology.orchestra.run.smithi129.stdout:(103/150): lua-devel-5.4.4-4.el9.x86_64.rpm 288 kB/s | 22 kB 00:00 2024-04-21T08:32:29.122 INFO:teuthology.orchestra.run.smithi129.stdout:(104/150): protobuf-compiler-3.14.0-13.el9.x86_ 8.9 MB/s | 863 kB 00:00 2024-04-21T08:32:29.364 INFO:teuthology.orchestra.run.smithi129.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-04-21T08:32:29.514 INFO:teuthology.orchestra.run.smithi129.stdout:(106/150): gperftools-libs-2.9.1-2.el9.x86_64.r 712 kB/s | 309 kB 00:00 2024-04-21T08:32:29.536 INFO:teuthology.orchestra.run.smithi057.stdout:(96/150): qatzip-libs-1.2.0-1.el9.x86_64.rpm 71 kB/s | 47 kB 00:00 2024-04-21T08:32:29.539 INFO:teuthology.orchestra.run.smithi129.stdout:(107/150): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-21T08:32:29.590 INFO:teuthology.orchestra.run.smithi129.stdout:(108/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-04-21T08:32:29.615 INFO:teuthology.orchestra.run.smithi129.stdout:(109/150): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-04-21T08:32:29.665 INFO:teuthology.orchestra.run.smithi129.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-04-21T08:32:29.690 INFO:teuthology.orchestra.run.smithi129.stdout:(111/150): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-21T08:32:29.841 INFO:teuthology.orchestra.run.smithi129.stdout:(112/150): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-04-21T08:32:29.875 INFO:teuthology.orchestra.run.smithi129.stdout:(113/150): python3-asyncssh-2.13.2-5.el9.noarch 2.9 MB/s | 548 kB 00:00 2024-04-21T08:32:29.900 INFO:teuthology.orchestra.run.smithi129.stdout:(114/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-04-21T08:32:29.925 INFO:teuthology.orchestra.run.smithi129.stdout:(115/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 520 kB/s | 43 kB 00:00 2024-04-21T08:32:29.936 INFO:teuthology.orchestra.run.smithi057.stdout:(97/150): socat-1.7.4.1-5.el9.x86_64.rpm 293 kB/s | 305 kB 00:01 2024-04-21T08:32:29.950 INFO:teuthology.orchestra.run.smithi129.stdout:(116/150): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-04-21T08:32:29.976 INFO:teuthology.orchestra.run.smithi129.stdout:(117/150): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-04-21T08:32:30.001 INFO:teuthology.orchestra.run.smithi129.stdout:(118/150): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-21T08:32:30.043 INFO:teuthology.orchestra.run.smithi129.stdout:(119/150): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-21T08:32:30.068 INFO:teuthology.orchestra.run.smithi129.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-21T08:32:30.093 INFO:teuthology.orchestra.run.smithi129.stdout:(121/150): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-21T08:32:30.127 INFO:teuthology.orchestra.run.smithi129.stdout:(122/150): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-04-21T08:32:30.160 INFO:teuthology.orchestra.run.smithi129.stdout:(123/150): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-21T08:32:30.194 INFO:teuthology.orchestra.run.smithi129.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-04-21T08:32:30.211 INFO:teuthology.orchestra.run.smithi057.stdout:(98/150): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-04-21T08:32:30.236 INFO:teuthology.orchestra.run.smithi129.stdout:(125/150): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-21T08:32:30.261 INFO:teuthology.orchestra.run.smithi129.stdout:(126/150): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-21T08:32:30.286 INFO:teuthology.orchestra.run.smithi129.stdout:(127/150): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-21T08:32:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:(128/150): python3-jaraco-functools-3.5.0-2.el9 289 kB/s | 19 kB 00:00 2024-04-21T08:32:30.329 INFO:teuthology.orchestra.run.smithi129.stdout:(129/150): python3-jaraco-text-3.2.0-6.el9.noar 292 kB/s | 20 kB 00:00 2024-04-21T08:32:30.370 INFO:teuthology.orchestra.run.smithi129.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 692 kB/s | 46 kB 00:00 2024-04-21T08:32:30.396 INFO:teuthology.orchestra.run.smithi129.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-04-21T08:32:30.429 INFO:teuthology.orchestra.run.smithi129.stdout:(132/150): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-04-21T08:32:30.446 INFO:teuthology.orchestra.run.smithi129.stdout:(133/150): python3-natsort-7.1.1-5.el9.noarch.r 763 kB/s | 58 kB 00:00 2024-04-21T08:32:30.472 INFO:teuthology.orchestra.run.smithi129.stdout:(134/150): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-21T08:32:30.489 INFO:teuthology.orchestra.run.smithi129.stdout:(135/150): python3-portend-3.1.0-2.el9.noarch.r 277 kB/s | 16 kB 00:00 2024-04-21T08:32:30.514 INFO:teuthology.orchestra.run.smithi129.stdout:(136/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-21T08:32:30.531 INFO:teuthology.orchestra.run.smithi129.stdout:(137/150): python3-pyperclip-1.8.2-2.el9.noarch 479 kB/s | 28 kB 00:00 2024-04-21T08:32:30.548 INFO:teuthology.orchestra.run.smithi129.stdout:(138/150): python3-repoze-lru-0.7-16.el9.noarch 522 kB/s | 31 kB 00:00 2024-04-21T08:32:30.550 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:30.590 INFO:teuthology.orchestra.run.smithi129.stdout:(139/150): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-21T08:32:30.590 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:30.607 INFO:teuthology.orchestra.run.smithi129.stdout:(140/150): python3-rsa-4.9-2.el9.noarch.rpm 782 kB/s | 59 kB 00:00 2024-04-21T08:32:30.624 INFO:teuthology.orchestra.run.smithi129.stdout:(141/150): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-04-21T08:32:30.649 INFO:teuthology.orchestra.run.smithi129.stdout:(142/150): python3-typing-extensions-4.4.0-2.el 866 kB/s | 51 kB 00:00 2024-04-21T08:32:30.661 INFO:teuthology.orchestra.run.smithi057.stdout:(99/150): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 129 kB/s | 94 kB 00:00 2024-04-21T08:32:30.682 INFO:teuthology.orchestra.run.smithi129.stdout:(143/150): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-21T08:32:30.716 INFO:teuthology.orchestra.run.smithi129.stdout:(144/150): python3-websocket-client-1.2.3-2.el9 972 kB/s | 90 kB 00:00 2024-04-21T08:32:30.741 INFO:teuthology.orchestra.run.smithi129.stdout:(145/150): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-21T08:32:30.766 INFO:teuthology.orchestra.run.smithi129.stdout:(146/150): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-21T08:32:30.792 INFO:teuthology.orchestra.run.smithi129.stdout:(147/150): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-21T08:32:30.842 INFO:teuthology.orchestra.run.smithi129.stdout:(148/150): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-21T08:32:30.878 INFO:teuthology.orchestra.run.smithi057.stdout:(100/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-04-21T08:32:31.132 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:31.168 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:31.262 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:31.262 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:32:31.262 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:32:31.262 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:(101/150): zlib-devel-1.2.11-41.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-04-21T08:32:31.295 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:31.501 INFO:teuthology.orchestra.run.smithi129.stdout:(149/150): librados2-19.0.0-3154.ga630765c.el9. 4.6 MB/s | 3.4 MB 00:00 2024-04-21T08:32:31.512 INFO:teuthology.orchestra.run.smithi057.stdout:(102/150): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-04-21T08:32:32.234 INFO:teuthology.orchestra.run.smithi129.stdout:(150/150): librbd1-19.0.0-3154.ga630765c.el9.x8 2.2 MB/s | 3.1 MB 00:01 2024-04-21T08:32:32.242 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:32:32.242 INFO:teuthology.orchestra.run.smithi129.stdout:Total 7.8 MB/s | 218 MB 00:27 2024-04-21T08:32:32.263 INFO:teuthology.orchestra.run.smithi057.stdout:(103/150): abseil-cpp-20211102.0-3.el9.x86_64.r 731 kB/s | 548 kB 00:00 2024-04-21T08:32:32.412 INFO:teuthology.orchestra.run.smithi057.stdout:(104/150): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-04-21T08:32:32.504 INFO:teuthology.orchestra.run.smithi057.stdout:(105/150): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-04-21T08:32:32.896 INFO:teuthology.orchestra.run.smithi057.stdout:(106/150): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-21T08:32:32.988 INFO:teuthology.orchestra.run.smithi057.stdout:(107/150): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-04-21T08:32:33.080 INFO:teuthology.orchestra.run.smithi057.stdout:(108/150): liboath-2.6.7-2.el9.x86_64.rpm 531 kB/s | 49 kB 00:00 2024-04-21T08:32:33.181 INFO:teuthology.orchestra.run.smithi057.stdout:(109/150): protobuf-compiler-3.14.0-13.el9.x86_ 456 kB/s | 863 kB 00:01 2024-04-21T08:32:33.206 INFO:teuthology.orchestra.run.smithi057.stdout:(110/150): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-04-21T08:32:33.297 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-04-21T08:32:33.314 INFO:teuthology.orchestra.run.smithi057.stdout:(111/150): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-04-21T08:32:33.414 INFO:teuthology.orchestra.run.smithi057.stdout:(112/150): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-04-21T08:32:33.421 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-04-21T08:32:33.421 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-04-21T08:32:33.507 INFO:teuthology.orchestra.run.smithi057.stdout:(113/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 474 kB/s | 43 kB 00:00 2024-04-21T08:32:33.599 INFO:teuthology.orchestra.run.smithi057.stdout:(114/150): python3-cachetools-4.2.4-1.el9.noarc 352 kB/s | 32 kB 00:00 2024-04-21T08:32:33.623 INFO:teuthology.orchestra.run.smithi057.stdout:(115/150): luarocks-3.9.2-1.el9.noarch.rpm 341 kB/s | 151 kB 00:00 2024-04-21T08:32:33.691 INFO:teuthology.orchestra.run.smithi057.stdout:(116/150): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-21T08:32:33.799 INFO:teuthology.orchestra.run.smithi057.stdout:(117/150): python3-cheroot-8.6.0-4.el9.noarch.r 979 kB/s | 172 kB 00:00 2024-04-21T08:32:33.825 INFO:teuthology.orchestra.run.smithi057.stdout:(118/150): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-04-21T08:32:33.925 INFO:teuthology.orchestra.run.smithi057.stdout:(119/150): python3-colorama-0.4.4-12.el9.noarch 354 kB/s | 35 kB 00:00 2024-04-21T08:32:34.000 INFO:teuthology.orchestra.run.smithi057.stdout:(120/150): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.5 MB/s | 303 kB 00:00 2024-04-21T08:32:34.142 INFO:teuthology.orchestra.run.smithi057.stdout:(121/150): python3-google-auth-2.28.2-1.el9.noa 935 kB/s | 203 kB 00:00 2024-04-21T08:32:34.234 INFO:teuthology.orchestra.run.smithi057.stdout:(122/150): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-21T08:32:34.301 INFO:teuthology.orchestra.run.smithi057.stdout:(123/150): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-04-21T08:32:34.326 INFO:teuthology.orchestra.run.smithi057.stdout:(124/150): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-21T08:32:34.393 INFO:teuthology.orchestra.run.smithi057.stdout:(125/150): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-21T08:32:34.418 INFO:teuthology.orchestra.run.smithi057.stdout:(126/150): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-21T08:32:34.485 INFO:teuthology.orchestra.run.smithi057.stdout:(127/150): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-21T08:32:34.511 INFO:teuthology.orchestra.run.smithi057.stdout:(128/150): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-21T08:32:34.611 INFO:teuthology.orchestra.run.smithi057.stdout:(129/150): python3-kubernetes-26.1.0-2.el9.noar 8.2 MB/s | 1.0 MB 00:00 2024-04-21T08:32:34.636 INFO:teuthology.orchestra.run.smithi057.stdout:(130/150): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-04-21T08:32:34.703 INFO:teuthology.orchestra.run.smithi057.stdout:(131/150): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-04-21T08:32:34.728 INFO:teuthology.orchestra.run.smithi057.stdout:(132/150): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-04-21T08:32:34.804 INFO:teuthology.orchestra.run.smithi057.stdout:(133/150): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-04-21T08:32:34.829 INFO:teuthology.orchestra.run.smithi057.stdout:(134/150): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-21T08:32:34.896 INFO:teuthology.orchestra.run.smithi057.stdout:(135/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 981 kB/s | 90 kB 00:00 2024-04-21T08:32:34.921 INFO:teuthology.orchestra.run.smithi057.stdout:(136/150): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-04-21T08:32:34.996 INFO:teuthology.orchestra.run.smithi057.stdout:(137/150): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-21T08:32:35.022 INFO:teuthology.orchestra.run.smithi057.stdout:(138/150): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-21T08:32:35.089 INFO:teuthology.orchestra.run.smithi057.stdout:(139/150): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-21T08:32:35.114 INFO:teuthology.orchestra.run.smithi057.stdout:(140/150): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-04-21T08:32:35.125 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-04-21T08:32:35.126 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-04-21T08:32:35.181 INFO:teuthology.orchestra.run.smithi057.stdout:(141/150): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-04-21T08:32:35.206 INFO:teuthology.orchestra.run.smithi057.stdout:(142/150): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-21T08:32:35.281 INFO:teuthology.orchestra.run.smithi057.stdout:(143/150): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-04-21T08:32:35.315 INFO:teuthology.orchestra.run.smithi057.stdout:(144/150): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-04-21T08:32:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:(145/150): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-04-21T08:32:35.407 INFO:teuthology.orchestra.run.smithi057.stdout:(146/150): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-21T08:32:35.549 INFO:teuthology.orchestra.run.smithi057.stdout:(147/150): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-04-21T08:32:35.741 INFO:teuthology.orchestra.run.smithi057.stdout:(148/150): librados2-19.0.0-3154.ga630765c.el9. 10 MB/s | 3.4 MB 00:00 2024-04-21T08:32:35.966 INFO:teuthology.orchestra.run.smithi057.stdout:(149/150): librbd1-19.0.0-3154.ga630765c.el9.x8 7.5 MB/s | 3.1 MB 00:00 2024-04-21T08:32:40.915 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-04-21T08:32:40.995 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:32:41.045 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:32:41.088 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:32:41.473 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:32:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:41.675 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:41.705 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:41.797 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:41.839 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:41.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:32:41.894 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:32:42.047 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:32:42.099 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:32:42.127 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:42.222 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:42.245 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:42.291 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:42.313 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:42.336 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:42.512 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:42.598 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-04-21T08:32:42.599 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-04-21T08:32:42.600 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:42.621 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:32:42.725 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:32:42.753 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:32:42.776 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:42.788 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:42.862 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:32:42.895 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:32:42.938 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:32:42.971 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:32:42.999 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:32:43.070 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:32:43.130 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:32:43.160 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:32:43.195 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:32:43.272 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:32:43.306 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:32:43.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:32:43.378 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:32:43.449 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:32:43.493 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:32:43.568 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:32:43.702 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:32:43.754 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:32:44.290 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:32:44.296 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:44.321 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:32:44.322 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:44.322 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:32:44.322 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:44.322 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:32:44.322 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:44.375 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:32:44.403 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:32:44.446 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:32:44.484 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:32:44.517 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:32:44.554 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:44.573 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:32:44.578 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:32:44.578 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:44.578 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:32:44.578 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:44.578 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:32:44.579 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:44.645 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:32:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:32:44.713 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:32:44.759 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:32:44.796 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:32:44.839 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:32:44.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:32:45.022 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:45.027 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:32:45.045 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:32:45.045 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:45.046 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:32:45.046 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:45.046 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:32:45.046 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:45.086 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:32:45.141 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:32:45.152 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:32:45.183 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:32:45.209 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:32:45.264 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:32:45.278 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:32:45.308 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:32:45.348 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:32:45.373 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:32:45.398 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:32:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:32:45.473 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:32:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:32:45.576 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:32:45.602 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:32:45.639 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:32:45.664 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:32:45.666 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:32:45.701 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:32:45.745 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:32:45.783 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:32:45.822 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:32:45.848 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:32:45.852 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:32:45.879 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:32:45.880 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:32:45.908 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:32:45.911 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:32:45.953 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:32:45.987 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:32:46.076 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:32:46.127 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:32:46.158 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:32:46.205 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:32:46.313 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:32:46.352 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:46.368 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:32:46.390 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:32:46.423 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:32:46.485 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:32:46.514 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:47.107 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:32:47.156 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:32:47.190 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:32:47.290 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:32:47.339 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:32:47.428 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:32:47.483 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:32:47.490 INFO:teuthology.orchestra.run.smithi057.stdout:(150/150): python3-scipy-1.6.2-8.el9.x86_64.rpm 795 kB/s | 16 MB 00:20 2024-04-21T08:32:47.495 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:32:47.495 INFO:teuthology.orchestra.run.smithi057.stdout:Total 3.8 MB/s | 218 MB 00:57 2024-04-21T08:32:47.548 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:32:47.582 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:32:47.697 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:32:47.776 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:32:48.432 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:32:48.538 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-04-21T08:32:48.538 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:32:48.550 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:32:48.574 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:48.677 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-04-21T08:32:48.677 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-04-21T08:32:48.678 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:32:48.847 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:32:48.885 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:48.908 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:32:48.908 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:48.908 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:32:48.908 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:48.908 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:32:48.908 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:48.952 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:48.964 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:32:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:32:49.109 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:32:49.793 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:32:49.875 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:32:49.904 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:32:49.928 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:32:50.030 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:32:50.063 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:50.086 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:32:50.086 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:50.086 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:32:50.086 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:50.086 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:32:50.086 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:50.130 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:50.153 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:32:50.153 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:50.153 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:32:50.153 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:50.198 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:50.220 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:32:50.221 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:50.221 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:32:50.221 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:50.221 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:32:50.221 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:50.241 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:32:50.283 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:50.373 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:32:50.406 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:32:50.440 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:32:50.496 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-04-21T08:32:50.496 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-04-21T08:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:32:50.919 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:32:50.944 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:51.007 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:32:51.029 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:32:53.293 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:53.342 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:53.880 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:32:53.920 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:54.002 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:32:54.002 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:32:54.002 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:32:54.002 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:32:54.029 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:32:55.219 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:32:55.260 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:32:55.292 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:32:55.319 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:32:55.501 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:32:55.549 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:32:55.579 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:32:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:56.244 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:56.264 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:32:56.411 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-04-21T08:32:56.501 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/152 2024-04-21T08:32:56.526 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2024-04-21T08:32:56.569 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/152 2024-04-21T08:32:56.922 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2024-04-21T08:32:57.026 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:57.047 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:32:57.133 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:32:57.134 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 5/152 2024-04-21T08:32:57.161 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:57.174 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:32:57.253 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:32:57.273 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:32:57.295 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:32:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:32:57.304 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:32:57.323 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/152 2024-04-21T08:32:57.349 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/152 2024-04-21T08:32:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:32:57.388 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:32:57.511 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 10/152 2024-04-21T08:32:57.564 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-13.14-1.el9.x86_64 11/152 2024-04-21T08:32:57.591 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:57.696 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:32:57.725 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:57.790 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 13/152 2024-04-21T08:32:57.810 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:57.830 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 14/152 2024-04-21T08:32:57.986 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:58.026 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 15/152 2024-04-21T08:32:58.137 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : re2-1:20211101-3.el9.x86_64 16/152 2024-04-21T08:32:58.236 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 17/152 2024-04-21T08:32:58.284 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 18/152 2024-04-21T08:32:58.320 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.7-2.el9.x86_64 19/152 2024-04-21T08:32:58.396 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 20/152 2024-04-21T08:32:58.435 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 21/152 2024-04-21T08:32:58.479 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 22/152 2024-04-21T08:32:58.511 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 23/152 2024-04-21T08:32:58.538 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-5.4.4-4.el9.x86_64 24/152 2024-04-21T08:32:58.612 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 25/152 2024-04-21T08:32:58.671 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 26/152 2024-04-21T08:32:58.700 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 27/152 2024-04-21T08:32:58.743 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 28/152 2024-04-21T08:32:58.824 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 29/152 2024-04-21T08:32:58.863 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 30/152 2024-04-21T08:32:58.898 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 31/152 2024-04-21T08:32:58.926 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 32/152 2024-04-21T08:32:59.003 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 33/152 2024-04-21T08:32:59.049 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : luarocks-3.9.2-1.el9.noarch 34/152 2024-04-21T08:32:59.128 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 35/152 2024-04-21T08:32:59.276 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/152 2024-04-21T08:32:59.328 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/152 2024-04-21T08:32:59.895 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/152 2024-04-21T08:32:59.985 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/152 2024-04-21T08:33:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/152 2024-04-21T08:33:00.069 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 41/152 2024-04-21T08:33:00.107 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 42/152 2024-04-21T08:33:00.140 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/152 2024-04-21T08:33:00.197 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/152 2024-04-21T08:33:00.272 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/152 2024-04-21T08:33:00.307 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/152 2024-04-21T08:33:00.361 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/152 2024-04-21T08:33:00.408 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/152 2024-04-21T08:33:00.443 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/152 2024-04-21T08:33:00.487 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/152 2024-04-21T08:33:00.529 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/152 2024-04-21T08:33:00.677 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/152 2024-04-21T08:33:00.718 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/152 2024-04-21T08:33:00.747 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/152 2024-04-21T08:33:00.790 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/152 2024-04-21T08:33:00.891 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/152 2024-04-21T08:33:00.930 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 57/152 2024-04-21T08:33:00.970 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/152 2024-04-21T08:33:00.995 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/152 2024-04-21T08:33:01.020 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/152 2024-04-21T08:33:01.099 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 61/152 2024-04-21T08:33:01.227 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 62/152 2024-04-21T08:33:01.269 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 63/152 2024-04-21T08:33:01.297 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 64/152 2024-04-21T08:33:01.323 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 65/152 2024-04-21T08:33:01.368 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : uuid-1.6.2-55.el9.x86_64 66/152 2024-04-21T08:33:01.405 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 67/152 2024-04-21T08:33:01.445 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 68/152 2024-04-21T08:33:01.474 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 69/152 2024-04-21T08:33:01.508 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 70/152 2024-04-21T08:33:01.542 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 71/152 2024-04-21T08:33:01.584 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 72/152 2024-04-21T08:33:01.626 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 73/152 2024-04-21T08:33:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 74/152 2024-04-21T08:33:01.765 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 75/152 2024-04-21T08:33:01.813 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 76/152 2024-04-21T08:33:01.861 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 77/152 2024-04-21T08:33:01.975 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-packaging-20.9-5.el9.noarch 78/152 2024-04-21T08:33:02.032 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 79/152 2024-04-21T08:33:02.816 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 80/152 2024-04-21T08:33:02.861 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 81/152 2024-04-21T08:33:02.894 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 82/152 2024-04-21T08:33:03.003 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 83/152 2024-04-21T08:33:03.062 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 84/152 2024-04-21T08:33:03.153 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 85/152 2024-04-21T08:33:03.188 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 86/152 2024-04-21T08:33:03.373 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.21-2.el9.x86_64 87/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:33:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:33:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:33:03.398 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:33:03.399 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:33:03.400 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:33:03.401 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:33:03.402 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:33:03.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:33:03.404 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:33:03.405 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:33:03.406 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:33:03.407 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:33:03.408 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 88/152 2024-04-21T08:33:03.483 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 89/152 2024-04-21T08:33:04.182 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 90/152 2024-04-21T08:33:04.307 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 91/152 2024-04-21T08:33:05.562 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:05.562 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.563 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.564 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:33:05.565 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:33:05.566 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:33:05.567 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:33:05.568 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:33:05.569 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:33:05.570 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:33:05.571 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:33:05.572 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:05.573 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:33:05.574 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-04-21T08:33:05.637 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/152 2024-04-21T08:33:05.714 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 93/152 2024-04-21T08:33:05.742 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 94/152 2024-04-21T08:33:05.775 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 95/152 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-04-21T08:33:05.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:05.960 DEBUG:teuthology.parallel:result is None 2024-04-21T08:33:05.991 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:33:06.006 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:33:06.007 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:06.007 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:33:06.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:06.103 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/152 2024-04-21T08:33:06.145 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:33:06.237 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 97/152 2024-04-21T08:33:06.267 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 98/152 2024-04-21T08:33:06.304 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/152 2024-04-21T08:33:06.810 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 100/152 2024-04-21T08:33:06.840 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:33:06.912 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 101/152 2024-04-21T08:33:06.942 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 102/152 2024-04-21T08:33:07.477 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:33:07.502 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:33:07.502 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:07.502 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:33:07.502 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:33:07.503 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:33:07.503 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:07.719 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:33:07.741 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:33:07.741 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:07.741 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:33:07.741 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:33:07.742 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:33:07.742 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:08.186 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:33:08.209 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:33:08.209 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:08.209 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:33:08.209 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:33:08.209 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:33:08.209 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:08.316 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:33:08.373 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:33:08.429 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:33:08.614 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:33:08.662 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:33:08.739 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:33:08.828 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:33:08.957 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:33:09.003 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:33:09.035 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:33:09.063 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:33:09.297 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:33:09.330 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:33:09.457 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:33:09.486 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:33:09.518 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:33:09.580 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:33:09.609 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:33:09.904 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 103/152 2024-04-21T08:33:09.941 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:33:10.037 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 104/152 2024-04-21T08:33:10.037 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-21T08:33:10.037 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-21T08:33:10.037 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:10.067 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:33:10.639 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:33:10.669 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:33:11.627 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:33:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:33:11.762 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:33:11.933 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:33:11.963 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:33:11.986 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:33:11.986 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:11.987 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:33:11.987 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:33:11.987 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:33:11.987 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:12.031 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:33:12.042 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:33:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:33:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:33:13.105 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:33:13.133 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:33:13.155 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:33:13.155 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:13.156 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:33:13.156 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:33:13.156 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:33:13.156 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:13.200 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:33:13.222 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:33:13.222 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:13.222 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:33:13.222 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:13.267 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:33:13.290 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:33:13.291 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:13.291 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:33:13.291 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:33:13.291 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:33:13.291 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:13.602 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:33:13.624 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:33:13.625 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:13.625 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:33:13.625 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:33:13.625 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:33:13.625 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:18.189 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:33:18.230 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:33:18.270 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:33:18.303 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:33:18.330 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:33:18.511 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:33:18.551 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:33:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:33:19.174 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:19.211 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:19.231 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:20.006 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:33:20.090 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:33:20.123 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:33:20.211 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:33:20.244 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:33:20.244 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:33:20.319 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:33:20.320 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 105/152 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /root 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2024-04-21T08:33:21.855 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:22.044 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:33:22.060 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 106/152 2024-04-21T08:33:22.060 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:22.061 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-21T08:33:22.061 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:23.598 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:33:23.624 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 107/152 2024-04-21T08:33:23.624 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:23.624 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-21T08:33:23.624 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:33:23.625 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-21T08:33:23.625 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:23.856 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:33:23.880 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 108/152 2024-04-21T08:33:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-21T08:33:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:33:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-21T08:33:23.880 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:24.340 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:33:24.364 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 109/152 2024-04-21T08:33:24.364 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:24.364 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-21T08:33:24.364 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:33:24.364 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-21T08:33:24.364 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:24.471 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : automake-1.16.2-8.el9.noarch 110/152 2024-04-21T08:33:24.529 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ply-3.11-14.el9.noarch 111/152 2024-04-21T08:33:24.592 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 112/152 2024-04-21T08:33:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 113/152 2024-04-21T08:33:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 114/152 2024-04-21T08:33:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 115/152 2024-04-21T08:33:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 116/152 2024-04-21T08:33:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 117/152 2024-04-21T08:33:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 118/152 2024-04-21T08:33:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 119/152 2024-04-21T08:33:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.9-15.el9.x86_64 120/152 2024-04-21T08:33:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:33:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 121/152 2024-04-21T08:33:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 122/152 2024-04-21T08:33:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 123/152 2024-04-21T08:33:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:33:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:33:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:33:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:33:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:33:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:33:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:33:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:33:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:33:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:33:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:33:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:33:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:33:26.965 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 124/152 2024-04-21T08:33:26.998 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:33:28.011 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 125/152 2024-04-21T08:33:28.040 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:33:28.147 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 126/152 2024-04-21T08:33:28.278 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:28.278 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:28.278 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-04-21T08:33:28.278 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.278 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.279 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.280 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:33:28.281 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.282 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:33:28.283 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:33:28.284 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:33:28.285 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:33:28.286 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.287 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:33:28.288 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:33:28.289 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:33:28.290 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:33:28.290 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-04-21T08:33:28.290 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-04-21T08:33:28.311 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 127/152 2024-04-21T08:33:28.342 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:33:28.366 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 128/152 2024-04-21T08:33:28.366 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:28.366 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-21T08:33:28.366 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:33:28.366 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-21T08:33:28.366 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:28.418 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:33:28.429 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 129/152 2024-04-21T08:33:28.531 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 130/152 2024-04-21T08:33:28.576 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 131/152 2024-04-21T08:33:28.624 DEBUG:teuthology.parallel:result is None 2024-04-21T08:33:29.558 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libtool-2.4.6-45.el9.x86_64 132/152 2024-04-21T08:33:29.587 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:33:29.611 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 133/152 2024-04-21T08:33:29.611 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:29.611 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-21T08:33:29.611 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:33:29.611 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-21T08:33:29.611 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:29.663 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:33:29.686 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3154.ga6307 134/152 2024-04-21T08:33:29.686 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:29.686 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-21T08:33:29.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:29.738 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:33:29.762 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 135/152 2024-04-21T08:33:29.762 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:29.762 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-21T08:33:29.762 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:33:29.762 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-21T08:33:29.762 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:30.081 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:33:30.104 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 136/152 2024-04-21T08:33:30.104 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-21T08:33:30.104 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-21T08:33:30.104 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:33:30.104 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-21T08:33:30.105 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:34.943 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 137/152 2024-04-21T08:33:34.993 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 138/152 2024-04-21T08:33:35.034 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 139/152 2024-04-21T08:33:35.065 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 140/152 2024-04-21T08:33:35.092 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 141/152 2024-04-21T08:33:35.285 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 142/152 2024-04-21T08:33:35.338 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 143/152 2024-04-21T08:33:35.368 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 144/152 2024-04-21T08:33:36.011 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:36.057 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:36.078 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 145/152 2024-04-21T08:33:36.945 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 146/152 2024-04-21T08:33:37.036 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 147/152 2024-04-21T08:33:37.078 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 148/152 2024-04-21T08:33:37.162 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flex-2.6.4-9.el9.x86_64 149/152 2024-04-21T08:33:37.198 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : bison-3.7.4-5.el9.x86_64 150/152 2024-04-21T08:33:37.199 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:33:37.265 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2024-04-21T08:33:37.266 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:42.968 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:19.0.0-3154.ga630765c.el9.x86_64 1/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 3/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 4/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3154.ga6307 5/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 6/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 7/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 8/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 9/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 10/152 2024-04-21T08:33:42.969 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 11/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 12/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 13/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 14/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_ 15/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 16/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 17/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:19.0.0-3154.ga630765c.el9.x86_6 18/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86 19/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 20/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3154.ga630765c.el 21/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:19.0.0-3154.ga630765c.el9. 22/152 2024-04-21T08:33:42.970 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_6 23/152 2024-04-21T08:33:42.971 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 24/152 2024-04-21T08:33:42.971 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 25/152 2024-04-21T08:33:42.971 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 26/152 2024-04-21T08:33:42.971 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 27/152 2024-04-21T08:33:42.971 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 28/152 2024-04-21T08:33:42.971 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 29/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3154.ga630765c. 30/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noa 31/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.n 32/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3154.ga63 33/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el 34/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 35/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.e 36/152 2024-04-21T08:33:42.982 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 37/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:19.0.0-3154.ga630765c.el9.noarch 38/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 39/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/152 2024-04-21T08:33:42.983 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/152 2024-04-21T08:33:42.984 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 68/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/152 2024-04-21T08:33:42.985 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/152 2024-04-21T08:33:42.986 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/152 2024-04-21T08:33:42.987 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/152 2024-04-21T08:33:42.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 122/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/152 2024-04-21T08:33:42.989 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/152 2024-04-21T08:33:42.990 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/152 2024-04-21T08:33:42.991 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/152 2024-04-21T08:33:42.991 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:19.0.0-3154.ga630765c.el9.x86_64 149/152 2024-04-21T08:33:42.991 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2024-04-21T08:33:42.991 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 151/152 2024-04-21T08:33:45.076 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2024-04-21T08:33:45.076 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:45.076 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: automake-1.16.2-8.el9.noarch 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: bison-3.7.4-5.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.077 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.078 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-shell-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top-2:19.0.0-3154.ga630765c.el9.noarch 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: flex-2.6.4-9.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-21T08:33:45.079 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-21T08:33:45.080 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: lua-5.4.4-4.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-13.14-1.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-21T08:33:45.081 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-21T08:33:45.082 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-21T08:33:45.083 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply-3.11-14.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-21T08:33:45.084 INFO:teuthology.orchestra.run.smithi057.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.085 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-21T08:33:45.086 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:19.0.0-3154.ga630765c.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: re2-1:20211101-3.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-21T08:33:45.087 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-21T08:33:45.088 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:33:45.088 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-04-21T08:33:45.441 DEBUG:teuthology.parallel:result is None 2024-04-21T08:33:45.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:45.672 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:33:45.694 INFO:teuthology.orchestra.run.smithi057.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:33:45.694 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:33:45.694 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:33:45.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:45.839 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:33:45.866 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:33:45.867 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:33:45.867 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:33:45.869 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:46.099 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-21T08:33:46.124 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-3154.ga630765c.el9 2024-04-21T08:33:46.125 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154.ga630765c.el9 2024-04-21T08:33:46.125 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154.ga630765c is installed. 2024-04-21T08:33:46.127 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-21T08:33:46.127 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:33:46.127 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:33:46.154 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:33:46.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:33:46.185 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:33:46.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-21T08:33:46.215 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-21T08:33:46.215 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:33:46.215 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:33:46.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:33:46.304 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:33:46.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:33:46.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:33:46.401 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:33:46.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-04-21T08:33:46.437 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-21T08:33:46.503 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-21T08:33:46.503 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:33:46.503 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:33:46.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:33:46.594 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:33:46.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:33:46.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:33:46.687 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:33:46.687 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-21T08:33:46.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-21T08:33:46.775 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-21T08:33:46.776 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:33:46.776 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:33:46.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:33:46.866 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:33:46.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:33:46.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:33:46.959 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:33:46.959 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-04-21T08:33:46.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-21T08:33:47.046 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-21T08:33:47.143 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-21T08:33:47.144 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:47.144 INFO:tasks.cephadm:Cluster fsid is df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:33:47.144 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-21T08:33:47.145 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.57', 'mon.b': '172.21.15.129', 'mon.c': '172.21.15.136'} 2024-04-21T08:33:47.145 INFO:tasks.cephadm:First mon is mon.a on smithi057 2024-04-21T08:33:47.145 INFO:tasks.cephadm:First mgr is x 2024-04-21T08:33:47.145 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-21T08:33:47.145 DEBUG:teuthology.orchestra.run.smithi057:> sudo hostname $(hostname -s) 2024-04-21T08:33:47.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-04-21T08:33:47.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-04-21T08:33:47.221 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-21T08:33:47.221 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:47.436 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a630765c1f93c6e61d9b40c4b76437611d5ef29e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78740/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3154-ga630765c', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3154.ga630765c'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-19 11:13:53.331234', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-21T08:33:47.649 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 a630765c1f93c6e61d9b40c4b76437611d5ef29e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:33:47.650 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm 2024-04-21T08:33:47.650 DEBUG:teuthology.orchestra.run.smithi057:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:48.737 INFO:teuthology.orchestra.run.smithi057.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:33 /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:48.738 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:49.923 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:33 /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:49.923 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/a630765c1f93c6e61d9b40c4b76437611d5ef29e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:51.091 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 781532 Apr 21 08:33 /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:51.091 DEBUG:teuthology.orchestra.run.smithi057:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:51.111 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:51.131 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-21T08:33:51.165 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e on all hosts... 2024-04-21T08:33:51.165 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:33:51.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:33:51.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e pull 2024-04-21T08:33:52.711 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:52.725 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:33:52.844 INFO:teuthology.orchestra.run.smithi057.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:34:20.700 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-21T08:34:20.700 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:34:20.700 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:34:20.700 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-04-21T08:34:20.700 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:34:20.701 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-04-21T08:34:20.701 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-21T08:34:21.319 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-04-21T08:34:21.319 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:34:21.320 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:34:21.320 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-04-21T08:34:21.320 INFO:teuthology.orchestra.run.smithi129.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:34:21.320 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-04-21T08:34:21.320 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-04-21T08:34:28.676 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:34:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:34:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: "image_id": "b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192", 2024-04-21T08:34:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: "repo_digests": [ 2024-04-21T08:34:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda" 2024-04-21T08:34:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:34:28.677 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:34:28.703 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph 2024-04-21T08:34:28.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-04-21T08:34:28.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-04-21T08:34:28.884 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /etc/ceph 2024-04-21T08:34:28.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-04-21T08:34:28.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-04-21T08:34:28.962 INFO:tasks.cephadm:Writing seed config... 2024-04-21T08:34:28.963 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-21T08:34:28.963 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-21T08:34:28.963 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-21T08:34:28.963 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-21T08:34:28.963 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-21T08:34:28.963 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-21T08:34:28.964 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-21T08:34:28.965 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-21T08:34:28.966 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-21T08:34:28.967 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:34:28.967 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-21T08:34:28.982 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = df0fb3ae-ffb9-11ee-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-21T08:34:28.983 DEBUG:teuthology.orchestra.run.smithi057:mon.a> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a.service 2024-04-21T08:34:29.025 DEBUG:teuthology.orchestra.run.smithi057:mgr.x> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x.service 2024-04-21T08:34:29.068 INFO:tasks.cephadm:Bootstrapping... 2024-04-21T08:34:29.068 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e -v bootstrap --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.57 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:34:29.264 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-04-21T08:34:29.264 INFO:teuthology.orchestra.run.smithi057.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e', '-v', 'bootstrap', '--fsid', 'df0fb3ae-ffb9-11ee-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.57', '--skip-admin-label'] 2024-04-21T08:34:29.264 INFO:teuthology.orchestra.run.smithi057.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-21T08:34:29.265 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying podman|docker is present... 2024-04-21T08:34:29.291 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:34:29.291 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying lvm2 is present... 2024-04-21T08:34:29.291 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying time synchronization is in place... 2024-04-21T08:34:29.299 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:34:29.299 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:34:29.305 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:34:29.305 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2024-04-21T08:34:29.312 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout enabled 2024-04-21T08:34:29.318 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout active 2024-04-21T08:34:29.319 INFO:teuthology.orchestra.run.smithi057.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:34:29.319 INFO:teuthology.orchestra.run.smithi057.stdout:Repeating the final host check... 2024-04-21T08:34:29.345 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-21T08:34:29.345 INFO:teuthology.orchestra.run.smithi057.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-21T08:34:29.345 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl is present 2024-04-21T08:34:29.345 INFO:teuthology.orchestra.run.smithi057.stdout:lvcreate is present 2024-04-21T08:34:29.351 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-21T08:34:29.351 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-21T08:34:29.357 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-21T08:34:29.357 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2024-04-21T08:34:29.364 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout enabled 2024-04-21T08:34:29.371 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout active 2024-04-21T08:34:29.371 INFO:teuthology.orchestra.run.smithi057.stdout:Unit chronyd.service is enabled and running 2024-04-21T08:34:29.371 INFO:teuthology.orchestra.run.smithi057.stdout:Host looks OK 2024-04-21T08:34:29.371 INFO:teuthology.orchestra.run.smithi057.stdout:Cluster fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:29.371 INFO:teuthology.orchestra.run.smithi057.stdout:Acquiring lock 140137320631456 on /run/cephadm/df0fb3ae-ffb9-11ee-bc93-c7b262605968.lock 2024-04-21T08:34:29.371 INFO:teuthology.orchestra.run.smithi057.stdout:Lock 140137320631456 acquired on /run/cephadm/df0fb3ae-ffb9-11ee-bc93-c7b262605968.lock 2024-04-21T08:34:29.372 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying IP 172.21.15.57 port 3300 ... 2024-04-21T08:34:29.372 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying IP 172.21.15.57 port 6789 ... 2024-04-21T08:34:29.372 INFO:teuthology.orchestra.run.smithi057.stdout:Base mon IP(s) is [172.21.15.57:3300, 172.21.15.57:6789], mon addrv is [v2:172.21.15.57:3300,v1:172.21.15.57:6789] 2024-04-21T08:34:29.376 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.57 metric 100 2024-04-21T08:34:29.376 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.57 metric 100 2024-04-21T08:34:29.378 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-21T08:34:29.378 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-21T08:34:29.379 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-21T08:34:29.381 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-21T08:34:29.381 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-21T08:34:29.381 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:34:29.381 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-21T08:34:29.381 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2024-04-21T08:34:29.382 INFO:teuthology.orchestra.run.smithi057.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-21T08:34:29.383 INFO:teuthology.orchestra.run.smithi057.stdout:Mon IP `172.21.15.57` is in CIDR network `172.21.0.0/20` 2024-04-21T08:34:29.383 INFO:teuthology.orchestra.run.smithi057.stdout:Mon IP `172.21.15.57` is in CIDR network `172.21.0.0/20` 2024-04-21T08:34:29.383 INFO:teuthology.orchestra.run.smithi057.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-21T08:34:29.383 INFO:teuthology.orchestra.run.smithi057.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-21T08:34:29.383 INFO:teuthology.orchestra.run.smithi057.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stdout b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e... 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Getting image source signatures 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Copying blob sha256:7eab8a2da10d06ac88aea1fd8998a2559b4de9d07c585e3b1e6ee6c4d35fc119 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Copying config sha256:b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 2024-04-21T08:34:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-21T08:34:30.941 INFO:teuthology.orchestra.run.smithi057.stdout:ceph: stdout ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:34:30.941 INFO:teuthology.orchestra.run.smithi057.stdout:Ceph version: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev) 2024-04-21T08:34:30.941 INFO:teuthology.orchestra.run.smithi057.stdout:Extracting ceph user uid/gid from container image... 2024-04-21T08:34:31.788 INFO:teuthology.orchestra.run.smithi057.stdout:stat: stdout 167 167 2024-04-21T08:34:31.788 INFO:teuthology.orchestra.run.smithi057.stdout:Creating initial keys... 2024-04-21T08:34:32.760 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph-authtool: stdout AQCYzyRmTIhsBxAAGZIAjl8Buzx6RmuuKiky1Q== 2024-04-21T08:34:33.566 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph-authtool: stdout AQCZzyRmCp7TBBAAClkS4fMtkCsOJsEQvp4PGQ== 2024-04-21T08:34:41.728 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph-authtool: stdout AQCbzyRmN7UdDRAAtMJdG/lNo6g+rJUCY2LSzA== 2024-04-21T08:34:41.730 INFO:teuthology.orchestra.run.smithi057.stdout:Creating initial monmap... 2024-04-21T08:34:46.417 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:34:46.417 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-21T08:34:46.417 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:46.417 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:34:46.417 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool for a [v2:172.21.15.57:3300,v1:172.21.15.57:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-21T08:34:46.418 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = quincy 2024-04-21T08:34:46.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: set fsid to df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:46.418 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-21T08:34:46.418 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:34:46.418 INFO:teuthology.orchestra.run.smithi057.stdout:Creating mon... 2024-04-21T08:34:47.989 INFO:teuthology.orchestra.run.smithi057.stdout:create mon.a on 2024-04-21T08:34:48.203 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-21T08:34:48.352 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-21T08:34:48.530 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968.target. 2024-04-21T08:34:48.530 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968.target → /etc/systemd/system/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968.target. 2024-04-21T08:34:48.886 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:34:48.886 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to reset failed state of unit ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a.service: Unit ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a.service not loaded. 2024-04-21T08:34:49.063 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968.target.wants/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@.service. 2024-04-21T08:34:49.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 podman[27574]: 2024-04-21 08:34:49.218415426 +0000 UTC m=+0.026260017 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:34:49.595 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:34:49.595 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2024-04-21T08:34:49.602 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:34:49.602 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2024-04-21T08:34:49.602 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2024-04-21T08:34:49.602 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:34:49.603 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mon to start... 2024-04-21T08:34:49.603 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mon... 2024-04-21T08:34:49.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 podman[27574]: 2024-04-21 08:34:49.499247987 +0000 UTC m=+0.307092550 container init d4cfda37146ff4bdd3699553f29631c029526baf9cd2c58eacf3bc67b9de1367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T08:34:49.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 podman[27574]: 2024-04-21 08:34:49.502227275 +0000 UTC m=+0.310071838 container start d4cfda37146ff4bdd3699553f29631c029526baf9cd2c58eacf3bc67b9de1367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-21T08:34:49.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 ceph-mon[27593]: mkfs df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:49.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 ceph-mon[27593]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T08:34:49.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 bash[27574]: d4cfda37146ff4bdd3699553f29631c029526baf9cd2c58eacf3bc67b9de1367 2024-04-21T08:34:49.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:49 smithi057 systemd[1]: Started Ceph mon.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:50.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout cluster: 2024-04-21T08:34:50.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout id: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:50.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-21T08:34:50.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:50.647 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout services: 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.569932s) 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout data: 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-21T08:34:50.648 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout pgs: 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.036+0000 7f5eb7f53700 1 Processor -- start 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.036+0000 7f5eb7f53700 1 -- start start 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.036+0000 7f5eb7f53700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb01087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.036+0000 7f5eb7f53700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb0108da0 con 0x7f5eb01083b0 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.036+0000 7f5eb5cef700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb01087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb5cef700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb01087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59268/0 (socket says 172.21.15.57:59268) 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb5cef700 1 -- 172.21.15.57:0/3682152305 learned_addr learned my addr 172.21.15.57:0/3682152305 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb5cef700 1 -- 172.21.15.57:0/3682152305 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb0109560 con 0x7f5eb01083b0 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb5cef700 1 --2- 172.21.15.57:0/3682152305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb01087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098a0 tx=0x7f5eac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10917da3bd594e68 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb4ced700 1 -- 172.21.15.57:0/3682152305 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eac004160 con 0x7f5eb01083b0 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb4ced700 1 -- 172.21.15.57:0/3682152305 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eac004300 con 0x7f5eb01083b0 2024-04-21T08:34:50.649 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.037+0000 7f5eb4ced700 1 -- 172.21.15.57:0/3682152305 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eac0036a0 con 0x7f5eb01083b0 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 -- 172.21.15.57:0/3682152305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 msgr2=0x7f5eb01087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 --2- 172.21.15.57:0/3682152305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb01087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5eac0098a0 tx=0x7f5eac009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 -- 172.21.15.57:0/3682152305 shutdown_connections 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 --2- 172.21.15.57:0/3682152305 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb01087d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 -- 172.21.15.57:0/3682152305 >> 172.21.15.57:0/3682152305 conn(0x7f5eb01039b0 msgr2=0x7f5eb0105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 -- 172.21.15.57:0/3682152305 shutdown_connections 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 -- 172.21.15.57:0/3682152305 wait complete. 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.038+0000 7f5eb7f53700 1 Processor -- start 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb7f53700 1 -- start start 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb7f53700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb019b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb7f53700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb0109a90 con 0x7f5eb01083b0 2024-04-21T08:34:50.650 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb5cef700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb019b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb5cef700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb019b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59282/0 (socket says 172.21.15.57:59282) 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb5cef700 1 -- 172.21.15.57:0/1601736503 learned_addr learned my addr 172.21.15.57:0/1601736503 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb5cef700 1 -- 172.21.15.57:0/1601736503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb019bf00 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5eb5cef700 1 --2- 172.21.15.57:0/1601736503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb019b9c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5eac000c00 tx=0x7f5eac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eac003db0 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.039+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eac011630 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.040+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb019c190 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.040+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5eac011950 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.040+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb019ee80 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.040+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5eac011af0 con 0x7f5eb01083b0 2024-04-21T08:34:50.651 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.040+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5eac01c450 con 0x7f5eb01083b0 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.040+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb00640a0 con 0x7f5eb01083b0 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.043+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5eac004260 con 0x7f5eb01083b0 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.116+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f5eb002e100 con 0x7f5eb01083b0 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5ea6ffd700 1 -- 172.21.15.57:0/1601736503 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f5eac01b450 con 0x7f5eb01083b0 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 msgr2=0x7f5eb019b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 --2- 172.21.15.57:0/1601736503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb019b9c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5eac000c00 tx=0x7f5eac004750 comp rx=0 tx=0).stop 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 shutdown_connections 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 --2- 172.21.15.57:0/1601736503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5eb01083b0 0x7f5eb019b9c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 >> 172.21.15.57:0/1601736503 conn(0x7f5eb01039b0 msgr2=0x7f5eb01925c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 shutdown_connections 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:50.117+0000 7f5eb7f53700 1 -- 172.21.15.57:0/1601736503 wait complete. 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:mon is available 2024-04-21T08:34:50.652 INFO:teuthology.orchestra.run.smithi057.stdout:Assimilating anything we can from ceph.conf... 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: monmap epoch 1 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: last_changed 2024-04-21T08:34:44.528184+0000 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: min_mon_release 19 (squid) 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: election_strategy: 1 2024-04-21T08:34:50.738 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:34:50.739 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: fsmap 2024-04-21T08:34:50.739 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T08:34:50.739 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: mgrmap e1: no daemons active 2024-04-21T08:34:50.739 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:50 smithi057 ceph-mon[27593]: from='client.? 172.21.15.57:0/1601736503' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-21T08:34:51.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:51 smithi057 ceph-mon[27593]: from='client.? 172.21.15.57:0/2967632288' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:34:51.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:51 smithi057 ceph-mon[27593]: from='client.? 172.21.15.57:0/2967632288' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-21T08:34:52.629 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout fsid = df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.57:3300,v1:172.21.15.57:6789] 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:52.630 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.070+0000 7fe3d3113700 1 Processor -- start 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.070+0000 7fe3d3113700 1 -- start start 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.070+0000 7fe3d3113700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc106a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.070+0000 7fe3d3113700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe3cc107040 con 0x7fe3cc106650 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3d0eaf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:52.631 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3d0eaf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc106a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59286/0 (socket says 172.21.15.57:59286) 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3d0eaf700 1 -- 172.21.15.57:0/3030668854 learned_addr learned my addr 172.21.15.57:0/3030668854 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3d0eaf700 1 -- 172.21.15.57:0/3030668854 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3cc107370 con 0x7fe3cc106650 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3d0eaf700 1 --2- 172.21.15.57:0/3030668854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc106a70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00098a0 tx=0x7fe3c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71881c8777f3525e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3cb7fe700 1 -- 172.21.15.57:0/3030668854 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0004160 con 0x7fe3cc106650 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3cb7fe700 1 -- 172.21.15.57:0/3030668854 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c0004300 con 0x7fe3cc106650 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.071+0000 7fe3cb7fe700 1 -- 172.21.15.57:0/3030668854 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c00036a0 con 0x7fe3cc106650 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 -- 172.21.15.57:0/3030668854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 msgr2=0x7fe3cc106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 --2- 172.21.15.57:0/3030668854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc106a70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00098a0 tx=0x7fe3c0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 -- 172.21.15.57:0/3030668854 shutdown_connections 2024-04-21T08:34:52.632 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 --2- 172.21.15.57:0/3030668854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc106a70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 -- 172.21.15.57:0/3030668854 >> 172.21.15.57:0/3030668854 conn(0x7fe3cc101fe0 msgr2=0x7fe3cc104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 -- 172.21.15.57:0/3030668854 shutdown_connections 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 -- 172.21.15.57:0/3030668854 wait complete. 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 Processor -- start 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.072+0000 7fe3d3113700 1 -- start start 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d3113700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d3113700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe3cc107840 con 0x7fe3cc106650 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d0eaf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d0eaf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc19dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59294/0 (socket says 172.21.15.57:59294) 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d0eaf700 1 -- 172.21.15.57:0/2967632288 learned_addr learned my addr 172.21.15.57:0/2967632288 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d0eaf700 1 -- 172.21.15.57:0/2967632288 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3cc19e110 con 0x7fe3cc106650 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d0eaf700 1 --2- 172.21.15.57:0/2967632288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc19dbd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0000c00 tx=0x7fe3c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0003db0 con 0x7fe3cc106650 2024-04-21T08:34:52.633 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c0011630 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3cc19e3a0 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0011950 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.073+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3cc1a1090 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.074+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe3c0011af0 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.074+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe3c001c450 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.074+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3cc0640a0 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.077+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe3c0032020 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.147+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe3cc02e190 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.150+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fe3c0004260 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.151+0000 7fe3c9ffb700 1 -- 172.21.15.57:0/2967632288 <== mon.0 v2:172.21.15.57:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe3c001c6a0 con 0x7fe3cc106650 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.151+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 msgr2=0x7fe3cc19dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.151+0000 7fe3d3113700 1 --2- 172.21.15.57:0/2967632288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc19dbd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0000c00 tx=0x7fe3c0004750 comp rx=0 tx=0).stop 2024-04-21T08:34:52.634 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.151+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 shutdown_connections 2024-04-21T08:34:52.635 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.151+0000 7fe3d3113700 1 --2- 172.21.15.57:0/2967632288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3cc106650 0x7fe3cc19dbd0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:52.635 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.152+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 >> 172.21.15.57:0/2967632288 conn(0x7fe3cc101fe0 msgr2=0x7fe3cc1948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:52.635 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.152+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 shutdown_connections 2024-04-21T08:34:52.635 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:51.152+0000 7fe3d3113700 1 -- 172.21.15.57:0/2967632288 wait complete. 2024-04-21T08:34:52.635 INFO:teuthology.orchestra.run.smithi057.stdout:Generating new minimal ceph.conf... 2024-04-21T08:34:53.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:53 smithi057 ceph-mon[27593]: from='client.? 172.21.15.57:0/4225180101' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f806e755700 1 Processor -- start 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f806e755700 1 -- start start 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f806e755700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f80681087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f806e755700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8068108da0 con 0x7f80681083b0 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f8067fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f80681087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f8067fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f80681087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33956/0 (socket says 172.21.15.57:33956) 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f8067fff700 1 -- 172.21.15.57:0/275673534 learned_addr learned my addr 172.21.15.57:0/275673534 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.038+0000 7f8067fff700 1 -- 172.21.15.57:0/275673534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8068109560 con 0x7f80681083b0 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f8067fff700 1 --2- 172.21.15.57:0/275673534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f80681087d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f80580098a0 tx=0x7f8058009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a040dd9f84b516 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:53.669 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f8066ffd700 1 -- 172.21.15.57:0/275673534 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8058004160 con 0x7f80681083b0 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f8066ffd700 1 -- 172.21.15.57:0/275673534 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8058004300 con 0x7f80681083b0 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f8066ffd700 1 -- 172.21.15.57:0/275673534 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8058011af0 con 0x7f80681083b0 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f806e755700 1 -- 172.21.15.57:0/275673534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 msgr2=0x7f80681087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f806e755700 1 --2- 172.21.15.57:0/275673534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f80681087d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f80580098a0 tx=0x7f8058009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f806e755700 1 -- 172.21.15.57:0/275673534 shutdown_connections 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f806e755700 1 --2- 172.21.15.57:0/275673534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f80681087d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.039+0000 7f806e755700 1 -- 172.21.15.57:0/275673534 >> 172.21.15.57:0/275673534 conn(0x7f80681039b0 msgr2=0x7f8068105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.040+0000 7f806e755700 1 -- 172.21.15.57:0/275673534 shutdown_connections 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.040+0000 7f806e755700 1 -- 172.21.15.57:0/275673534 wait complete. 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.040+0000 7f806e755700 1 Processor -- start 2024-04-21T08:34:53.670 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.040+0000 7f806e755700 1 -- start start 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f806e755700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f806819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f806e755700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f806810c910 con 0x7f80681083b0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f8067fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f806819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f8067fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f806819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33958/0 (socket says 172.21.15.57:33958) 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f8067fff700 1 -- 172.21.15.57:0/4225180101 learned_addr learned my addr 172.21.15.57:0/4225180101 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f8067fff700 1 -- 172.21.15.57:0/4225180101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806819e940 con 0x7f80681083b0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f8067fff700 1 --2- 172.21.15.57:0/4225180101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f806819e400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8058000c00 tx=0x7f8058011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8058003b20 con 0x7f80681083b0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f805801a8c0 con 0x7f80681083b0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806819ebd0 con 0x7f80681083b0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.041+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8058024940 con 0x7f80681083b0 2024-04-21T08:34:53.671 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.042+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80681a18c0 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.042+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8058022020 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.042+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f80580193c0 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.043+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80680640a0 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.045+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f805802dde0 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.116+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f80681a1e00 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.117+0000 7f80657fa700 1 -- 172.21.15.57:0/4225180101 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f805801aa70 con 0x7f80681083b0 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.117+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 msgr2=0x7f806819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.117+0000 7f806e755700 1 --2- 172.21.15.57:0/4225180101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f806819e400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8058000c00 tx=0x7f8058011b70 comp rx=0 tx=0).stop 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.118+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 shutdown_connections 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.118+0000 7f806e755700 1 --2- 172.21.15.57:0/4225180101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80681083b0 0x7f806819e400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.118+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 >> 172.21.15.57:0/4225180101 conn(0x7f80681039b0 msgr2=0x7f8068104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:53.672 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.118+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 shutdown_connections 2024-04-21T08:34:53.673 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:53.118+0000 7f806e755700 1 -- 172.21.15.57:0/4225180101 wait complete. 2024-04-21T08:34:53.673 INFO:teuthology.orchestra.run.smithi057.stdout:Restarting the monitor... 2024-04-21T08:34:54.066 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:53 smithi057 systemd[1]: Stopping Ceph mon.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:34:54.066 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:53 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a[27589]: 2024-04-21T08:34:53.923+0000 7f9512c2f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:34:54.066 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:53 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a[27589]: 2024-04-21T08:34:53.923+0000 7f9512c2f700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-21T08:34:54.333 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 podman[27838]: 2024-04-21 08:34:54.067260567 +0000 UTC m=+0.228886489 container died d4cfda37146ff4bdd3699553f29631c029526baf9cd2c58eacf3bc67b9de1367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:34:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 podman[27838]: 2024-04-21 08:34:54.381617795 +0000 UTC m=+0.543243775 container cleanup d4cfda37146ff4bdd3699553f29631c029526baf9cd2c58eacf3bc67b9de1367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-21T08:34:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 bash[27838]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:34:54.946 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 podman[27850]: 2024-04-21 08:34:54.590555704 +0000 UTC m=+0.520773813 container remove d4cfda37146ff4bdd3699553f29631c029526baf9cd2c58eacf3bc67b9de1367 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T08:34:54.946 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-21T08:34:54.946 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 systemd[1]: Stopped Ceph mon.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:54.947 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 systemd[1]: Starting Ceph mon.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:34:55.201 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 podman[27949]: 2024-04-21 08:34:54.850630661 +0000 UTC m=+0.019137530 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e 2024-04-21T08:34:55.201 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:54 smithi057 podman[27949]: 2024-04-21 08:34:54.949711646 +0000 UTC m=+0.118218508 container create 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-21T08:34:55.201 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 podman[27949]: 2024-04-21 08:34:55.135565308 +0000 UTC m=+0.304072172 container init 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:34:55.201 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 podman[27949]: 2024-04-21 08:34:55.139051726 +0000 UTC m=+0.307558583 container start 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T08:34:55.201 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: pidfile_write: ignore empty --pid-file 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: load: jerasure load: lrc 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Git sha 0 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: DB SUMMARY 2024-04-21T08:34:55.202 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: DB Session ID: 57X624G45OZCZK6WJEMC 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104534 ; 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:34:55.203 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.env: 0x55f34a6facc0 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.info_log: 0x55f34c289be0 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.statistics: (nil) 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.use_fsync: 0 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:34:55.204 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.db_log_dir: 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.wal_dir: 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:34:55.205 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.write_buffer_manager: 0x55f34c21d0e0 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:34:55.206 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.unordered_write: 0 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.row_cache: None 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.wal_filter: None 2024-04-21T08:34:55.207 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.wal_compression: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:34:55.208 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:34:55.209 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_open_files: -1 2024-04-21T08:34:55.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:34:55.211 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Compression algorithms supported: 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kZSTD supported: 0 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:34:55.212 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.merge_operator: 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_filter: None 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:34:55.213 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f34c289dc0) 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: index_type: 0 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: data_block_index_type: 0 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: index_shortening: 1 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: checksum: 4 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: no_block_cache: 0 2024-04-21T08:34:55.214 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache: 0x55f34c24f1f0 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache_options: 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: capacity : 536870912 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: num_shard_bits : 4 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: strict_capacity_limit : 0 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_cache_compressed: (nil) 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: persistent_cache: (nil) 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_size: 4096 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_size_deviation: 10 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_restart_interval: 16 2024-04-21T08:34:55.215 INFO:journalctl@ceph.mon.a.smithi057.stdout: index_block_restart_interval: 1 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: metadata_block_size: 4096 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: partition_filters: 0 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: use_delta_encoding: 1 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: filter_policy: bloomfilter 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: whole_key_filtering: 1 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: verify_compression: 0 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: format_version: 5 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: enable_index_compression: 1 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: block_align: 0 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: max_auto_readahead_size: 262144 2024-04-21T08:34:55.216 INFO:journalctl@ceph.mon.a.smithi057.stdout: prepopulate_block_cache: 0 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression: NoCompression 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.num_levels: 7 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:34:55.217 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:34:55.218 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:34:55.219 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:34:55.220 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:34:55.221 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:34:55.222 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.ttl: 2592000 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:34:55.223 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7385695f-0fa1-4fa4-8868-e6ca902d5723 2024-04-21T08:34:55.224 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688495168082, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688495169301, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99761, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97643, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14028, "raw_average_key_size": 49, "raw_value_size": 90001, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688495, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7385695f-0fa1-4fa4-8868-e6ca902d5723", "db_session_id": "57X624G45OZCZK6WJEMC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688495169386, "job": 1, "event": "recovery_finished"} 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f34c364000 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: DB pointer 0x55f34c34e000 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** DB Stats ** 2024-04-21T08:34:55.225 INFO:journalctl@ceph.mon.a.smithi057.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** Compaction Stats [default] ** 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:34:55.226 INFO:journalctl@ceph.mon.a.smithi057.stdout: L0 2/0 99.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: Sum 2/0 99.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 124.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** Compaction Stats [default] ** 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 124.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2024-04-21T08:34:55.227 INFO:journalctl@ceph.mon.a.smithi057.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: Cumulative compaction: 0.00 GB write, 25.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: Interval compaction: 0.00 GB write, 25.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: Block cache BinnedLRUCache@0x55f34c24f1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2024-04-21T08:34:55.228 INFO:journalctl@ceph.mon.a.smithi057.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: starting mon.a rank 0 at public addrs [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] at bind addrs [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???) e1 preinit fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).mds e1 new map 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).mds e1 print_map 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout: e1 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout: legacy client fscid: -1 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout: 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout: No filesystems configured 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:34:55.229 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:34:55.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:34:55.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:34:55.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-21T08:34:55.230 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 bash[27949]: 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 2024-04-21T08:34:55.230 INFO:teuthology.orchestra.run.smithi057.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-21T08:34:55.537 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 systemd[1]: Started Ceph mon.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: monmap epoch 1 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: last_changed 2024-04-21T08:34:44.528184+0000 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: min_mon_release 19 (squid) 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: election_strategy: 1 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: fsmap 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: osdmap e1: 0 total, 0 up, 0 in 2024-04-21T08:34:55.538 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:55 smithi057 ceph-mon[27968]: mgrmap e1: no daemons active 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.644+0000 7f72e4b34700 1 Processor -- start 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.644+0000 7f72e4b34700 1 -- start start 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.644+0000 7f72e4b34700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.644+0000 7f72e4b34700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f72e007c8c0 con 0x7f72e007de40 2024-04-21T08:34:56.239 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.644+0000 7f72de59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72de59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33964/0 (socket says 172.21.15.57:33964) 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72de59c700 1 -- 172.21.15.57:0/3396632033 learned_addr learned my addr 172.21.15.57:0/3396632033 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72de59c700 1 -- 172.21.15.57:0/3396632033 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e007ca40 con 0x7f72e007de40 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72de59c700 1 --2- 172.21.15.57:0/3396632033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e007c2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72d00098a0 tx=0x7f72d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca15d52b9688d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72dd59a700 1 -- 172.21.15.57:0/3396632033 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d0004160 con 0x7f72e007de40 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72dd59a700 1 -- 172.21.15.57:0/3396632033 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72d0004300 con 0x7f72e007de40 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.645+0000 7f72dd59a700 1 -- 172.21.15.57:0/3396632033 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d0011af0 con 0x7f72e007de40 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 -- 172.21.15.57:0/3396632033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 msgr2=0x7f72e007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 --2- 172.21.15.57:0/3396632033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e007c2f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f72d00098a0 tx=0x7f72d0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 -- 172.21.15.57:0/3396632033 shutdown_connections 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 --2- 172.21.15.57:0/3396632033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e007c2f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 -- 172.21.15.57:0/3396632033 >> 172.21.15.57:0/3396632033 conn(0x7f72e01039b0 msgr2=0x7f72e0105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 -- 172.21.15.57:0/3396632033 shutdown_connections 2024-04-21T08:34:56.240 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.646+0000 7f72e4b34700 1 -- 172.21.15.57:0/3396632033 wait complete. 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72e4b34700 1 Processor -- start 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72e4b34700 1 -- start start 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72e4b34700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72e4b34700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010e900 con 0x7f72e007de40 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72de59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72de59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33978/0 (socket says 172.21.15.57:33978) 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72de59c700 1 -- 172.21.15.57:0/857742827 learned_addr learned my addr 172.21.15.57:0/857742827 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.647+0000 7f72de59c700 1 -- 172.21.15.57:0/857742827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e019e920 con 0x7f72e007de40 2024-04-21T08:34:56.241 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72de59c700 1 --2- 172.21.15.57:0/857742827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e019e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f72d0011ac0 tx=0x7f72d0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d0003b20 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72d001a8c0 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e019ebb0 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d00248d0 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e01a18a0 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f72d0022020 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.648+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f72d0019610 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.649+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e00640a0 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.651+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f72d0042e50 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.723+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f72e01a1a70 con 0x7f72e007de40 2024-04-21T08:34:56.242 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72cf7fe700 1 -- 172.21.15.57:0/857742827 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f72d0031020 con 0x7f72e007de40 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 msgr2=0x7f72e019e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 --2- 172.21.15.57:0/857742827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e019e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f72d0011ac0 tx=0x7f72d0011ba0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 shutdown_connections 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 --2- 172.21.15.57:0/857742827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72e007de40 0x7f72e019e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 >> 172.21.15.57:0/857742827 conn(0x7f72e01039b0 msgr2=0x7f72e0104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 shutdown_connections 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:55.730+0000 7f72e4b34700 1 -- 172.21.15.57:0/857742827 wait complete. 2024-04-21T08:34:56.243 INFO:teuthology.orchestra.run.smithi057.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-21T08:34:56.264 INFO:teuthology.orchestra.run.smithi057.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:34:56.264 INFO:teuthology.orchestra.run.smithi057.stdout:Creating mgr... 2024-04-21T08:34:56.265 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-21T08:34:56.265 INFO:teuthology.orchestra.run.smithi057.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-21T08:34:56.534 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:34:56.534 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Failed to reset failed state of unit ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x.service: Unit ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-21T08:34:56.684 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968.target.wants/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@.service. 2024-04-21T08:34:56.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:56 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/857742827' entity='client.admin' 2024-04-21T08:34:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:34:57.167 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2024-04-21T08:34:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:34:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2024-04-21T08:34:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2024-04-21T08:34:57.175 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to enable service . firewalld.service is not available 2024-04-21T08:34:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:34:57.184 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2024-04-21T08:34:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:34:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2024-04-21T08:34:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2024-04-21T08:34:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-21T08:34:57.192 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr to start... 2024-04-21T08:34:57.193 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr... 2024-04-21T08:34:57.263 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:57 smithi057 systemd[1]: Started Ceph mgr.x for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:34:57.536 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:57 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:57.264+0000 7f991ebaf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:34:57.537 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:57 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:57.455+0000 7f991ebaf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:34:57.813 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:57 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:57.569+0000 7f991ebaf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:34:58.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:34:57 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1323713724' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:34:58.090 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:57 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:57.902+0000 7f991ebaf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:34:58.090 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:57 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:57.994+0000 7f991ebaf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:34:58.570 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:34:58.572 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:58.573 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:34:58.574 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:34:58.582 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:34:58.583 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:49.550788+0000", 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:34:58.584 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.852+0000 7f11de39d700 1 Processor -- start 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.852+0000 7f11de39d700 1 -- start start 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.852+0000 7f11de39d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d810b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.852+0000 7f11de39d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f11d807be60 con 0x7f11d8108df0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d810b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d810b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34010/0 (socket says 172.21.15.57:34010) 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d7fff700 1 -- 172.21.15.57:0/3940345427 learned_addr learned my addr 172.21.15.57:0/3940345427 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d7fff700 1 -- 172.21.15.57:0/3940345427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d810b720 con 0x7f11d8108df0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d7fff700 1 --2- 172.21.15.57:0/3940345427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d810b1e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0098a0 tx=0x7f11cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc220158488881d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d6ffd700 1 -- 172.21.15.57:0/3940345427 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11cc004160 con 0x7f11d8108df0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d6ffd700 1 -- 172.21.15.57:0/3940345427 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f11cc004300 con 0x7f11d8108df0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.853+0000 7f11d6ffd700 1 -- 172.21.15.57:0/3940345427 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11cc011af0 con 0x7f11d8108df0 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 -- 172.21.15.57:0/3940345427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 msgr2=0x7f11d810b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 --2- 172.21.15.57:0/3940345427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d810b1e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0098a0 tx=0x7f11cc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.585 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 -- 172.21.15.57:0/3940345427 shutdown_connections 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 --2- 172.21.15.57:0/3940345427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d810b1e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 -- 172.21.15.57:0/3940345427 >> 172.21.15.57:0/3940345427 conn(0x7f11d8102820 msgr2=0x7f11d8104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 -- 172.21.15.57:0/3940345427 shutdown_connections 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.854+0000 7f11de39d700 1 -- 172.21.15.57:0/3940345427 wait complete. 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11de39d700 1 Processor -- start 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11de39d700 1 -- start start 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11de39d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d819c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11de39d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f11d810c6e0 con 0x7f11d8108df0 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d819c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11d7fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d819c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34016/0 (socket says 172.21.15.57:34016) 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.855+0000 7f11d7fff700 1 -- 172.21.15.57:0/1323713724 learned_addr learned my addr 172.21.15.57:0/1323713724 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11d7fff700 1 -- 172.21.15.57:0/1323713724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d819c740 con 0x7f11d8108df0 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11d7fff700 1 --2- 172.21.15.57:0/1323713724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d819c200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f11cc003f40 tx=0x7f11cc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11cc0038b0 con 0x7f11d8108df0 2024-04-21T08:34:58.586 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d819c9d0 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f11cc003a50 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11cc01a5c0 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.856+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d819f6c0 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.857+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f11cc01ac50 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.857+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f11cc019700 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.857+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d80640a0 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.859+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f11cc0199a0 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.932+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f11d819fc10 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.933+0000 7f11d57fa700 1 -- 172.21.15.57:0/1323713724 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f11cc019b80 con 0x7f11d8108df0 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 msgr2=0x7f11d819c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 --2- 172.21.15.57:0/1323713724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d819c200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f11cc003f40 tx=0x7f11cc011890 comp rx=0 tx=0).stop 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 shutdown_connections 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 --2- 172.21.15.57:0/1323713724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11d8108df0 0x7f11d819c200 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:34:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 >> 172.21.15.57:0/1323713724 conn(0x7f11d8102820 msgr2=0x7f11d81032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:34:58.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 shutdown_connections 2024-04-21T08:34:58.588 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:34:57.934+0000 7f11de39d700 1 -- 172.21.15.57:0/1323713724 wait complete. 2024-04-21T08:34:58.588 INFO:teuthology.orchestra.run.smithi057.stdout:mgr not available, waiting (1/15)... 2024-04-21T08:34:59.449 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:59 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:59.275+0000 7f991ebaf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:34:59.838 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:59 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:59.459+0000 7f991ebaf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:34:59.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:59 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:59.639+0000 7f991ebaf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:35:00.254 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:34:59 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:34:59.942+0000 7f991ebaf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:35:00.255 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:00.039+0000 7f991ebaf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:35:00.255 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:00.255+0000 7f991ebaf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:35:00.588 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:00.456+0000 7f991ebaf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:01.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:01 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3123147513' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:35:01.339 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:01 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:01.104+0000 7f991ebaf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:35:01.691 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:35:01.694 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:35:01.695 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:35:01.703 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:35:01.704 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:01.705 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:49.550788+0000", 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.018+0000 7f7ff4f24700 1 Processor -- start 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.019+0000 7f7ff4f24700 1 -- start start 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.019+0000 7f7ff4f24700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff0108760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.019+0000 7f7ff4f24700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff0108d30 con 0x7f7ff0108340 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.019+0000 7f7fee59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff0108760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:01.706 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.019+0000 7f7fee59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff0108760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34032/0 (socket says 172.21.15.57:34032) 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.019+0000 7f7fee59c700 1 -- 172.21.15.57:0/168448618 learned_addr learned my addr 172.21.15.57:0/168448618 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.020+0000 7f7fee59c700 1 -- 172.21.15.57:0/168448618 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff0109540 con 0x7f7ff0108340 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.020+0000 7f7fee59c700 1 --2- 172.21.15.57:0/168448618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff0108760 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7fe00098a0 tx=0x7f7fe0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ee107c41413001 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.020+0000 7f7fed59a700 1 -- 172.21.15.57:0/168448618 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe0004160 con 0x7f7ff0108340 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.020+0000 7f7fed59a700 1 -- 172.21.15.57:0/168448618 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7fe0004300 con 0x7f7ff0108340 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.020+0000 7f7fed59a700 1 -- 172.21.15.57:0/168448618 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe0011af0 con 0x7f7ff0108340 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 -- 172.21.15.57:0/168448618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 msgr2=0x7f7ff0108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 --2- 172.21.15.57:0/168448618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff0108760 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7fe00098a0 tx=0x7f7fe0009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 -- 172.21.15.57:0/168448618 shutdown_connections 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 --2- 172.21.15.57:0/168448618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff0108760 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 -- 172.21.15.57:0/168448618 >> 172.21.15.57:0/168448618 conn(0x7f7ff01038c0 msgr2=0x7f7ff0105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 -- 172.21.15.57:0/168448618 shutdown_connections 2024-04-21T08:35:01.707 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.021+0000 7f7ff4f24700 1 -- 172.21.15.57:0/168448618 wait complete. 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7ff4f24700 1 Processor -- start 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7ff4f24700 1 -- start start 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7ff4f24700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff007db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7ff4f24700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff010c8f0 con 0x7f7ff0108340 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7fee59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff007db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7fee59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff007db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34042/0 (socket says 172.21.15.57:34042) 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7fee59c700 1 -- 172.21.15.57:0/3123147513 learned_addr learned my addr 172.21.15.57:0/3123147513 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7fee59c700 1 -- 172.21.15.57:0/3123147513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff007e0a0 con 0x7f7ff0108340 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.022+0000 7f7fee59c700 1 --2- 172.21.15.57:0/3123147513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff007db60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0000c00 tx=0x7f7fe0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe0003b20 con 0x7f7ff0108340 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7fe001a8c0 con 0x7f7ff0108340 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff007c290 con 0x7f7ff0108340 2024-04-21T08:35:01.708 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe0024940 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff007c7d0 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7fe0022020 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.023+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7fe002ddd0 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.024+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ff00640a0 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.026+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7fe0042e50 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.125+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7ff007ca80 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.126+0000 7f7fdf7fe700 1 -- 172.21.15.57:0/3123147513 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7fe0031030 con 0x7f7ff0108340 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 msgr2=0x7f7ff007db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 --2- 172.21.15.57:0/3123147513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff007db60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7fe0000c00 tx=0x7f7fe0011b70 comp rx=0 tx=0).stop 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 shutdown_connections 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 --2- 172.21.15.57:0/3123147513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ff0108340 0x7f7ff007db60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 >> 172.21.15.57:0/3123147513 conn(0x7f7ff01038c0 msgr2=0x7f7ff0104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 shutdown_connections 2024-04-21T08:35:01.709 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:01.127+0000 7f7ff4f24700 1 -- 172.21.15.57:0/3123147513 wait complete. 2024-04-21T08:35:01.710 INFO:teuthology.orchestra.run.smithi057.stdout:mgr not available, waiting (2/15)... 2024-04-21T08:35:01.838 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:01 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:01.557+0000 7f991ebaf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:35:01.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:01 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:01.658+0000 7f991ebaf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:35:02.338 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:01 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:01.868+0000 7f991ebaf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:35:02.762 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:02.417+0000 7f991ebaf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:35:02.762 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:02.511+0000 7f991ebaf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:35:03.088 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:02.762+0000 7f991ebaf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:35:03.089 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:02.899+0000 7f991ebaf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:35:03.089 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:02.997+0000 7f991ebaf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:35:03.441 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:03 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:03.116+0000 7f991ebaf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:03.441 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:03 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:03.441+0000 7f991ebaf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:35:04.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:04 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4256085290' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:35:04.339 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:04 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:04.073+0000 7f991ebaf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:35:04.339 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:04 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:04.303+0000 7f991ebaf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:35:04.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:35:04.791 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:04.792 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:35:04.793 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:35:04.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:35:04.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:35:04.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:35:04.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.801 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:04.802 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.803 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:49.550788+0000", 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f671b49c700 1 Processor -- start 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f671b49c700 1 -- start start 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f671b49c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671410b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f671b49c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f671407be00 con 0x7f6714108e50 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f6719238700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671410b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f6719238700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671410b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56838/0 (socket says 172.21.15.57:56838) 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.119+0000 7f6719238700 1 -- 172.21.15.57:0/1404169201 learned_addr learned my addr 172.21.15.57:0/1404169201 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:04.804 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f6719238700 1 -- 172.21.15.57:0/1404169201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671410b780 con 0x7f6714108e50 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f6719238700 1 --2- 172.21.15.57:0/1404169201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671410b240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67100098a0 tx=0x7f6710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfdf619165afeec server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f670bfff700 1 -- 172.21.15.57:0/1404169201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710004160 con 0x7f6714108e50 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f670bfff700 1 -- 172.21.15.57:0/1404169201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6710004300 con 0x7f6714108e50 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f670bfff700 1 -- 172.21.15.57:0/1404169201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710011af0 con 0x7f6714108e50 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f671b49c700 1 -- 172.21.15.57:0/1404169201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 msgr2=0x7f671410b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.120+0000 7f671b49c700 1 --2- 172.21.15.57:0/1404169201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671410b240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67100098a0 tx=0x7f6710009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 -- 172.21.15.57:0/1404169201 shutdown_connections 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 --2- 172.21.15.57:0/1404169201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671410b240 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 -- 172.21.15.57:0/1404169201 >> 172.21.15.57:0/1404169201 conn(0x7f6714102820 msgr2=0x7f6714104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 -- 172.21.15.57:0/1404169201 shutdown_connections 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 -- 172.21.15.57:0/1404169201 wait complete. 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 Processor -- start 2024-04-21T08:35:04.805 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 -- start start 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.121+0000 7f671b49c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f671410c6e0 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f6719238700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f6719238700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671419c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56848/0 (socket says 172.21.15.57:56848) 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f6719238700 1 -- 172.21.15.57:0/4256085290 learned_addr learned my addr 172.21.15.57:0/4256085290 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f6719238700 1 -- 172.21.15.57:0/4256085290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671419c730 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f6719238700 1 --2- 172.21.15.57:0/4256085290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671419c1f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6710011ac0 tx=0x7f6710011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710003b20 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f671001a8c0 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671419c9c0 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67100248d0 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.122+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671419f6b0 con 0x7f6714108e50 2024-04-21T08:35:04.806 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.123+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6710022020 con 0x7f6714108e50 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.123+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f67100195d0 con 0x7f6714108e50 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.123+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67140640a0 con 0x7f6714108e50 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.126+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6710042e50 con 0x7f6714108e50 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.200+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f671419fbf0 con 0x7f6714108e50 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.200+0000 7f670a7fc700 1 -- 172.21.15.57:0/4256085290 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6710031020 con 0x7f6714108e50 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 msgr2=0x7f671419c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 --2- 172.21.15.57:0/4256085290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671419c1f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6710011ac0 tx=0x7f6710011ba0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 shutdown_connections 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 --2- 172.21.15.57:0/4256085290 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6714108e50 0x7f671419c1f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 >> 172.21.15.57:0/4256085290 conn(0x7f6714102820 msgr2=0x7f67141032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 shutdown_connections 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:04.201+0000 7f671b49c700 1 -- 172.21.15.57:0/4256085290 wait complete. 2024-04-21T08:35:04.807 INFO:teuthology.orchestra.run.smithi057.stdout:mgr not available, waiting (3/15)... 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: Activating manager daemon x 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: mgrmap e2: x(active, starting, since 0.00333461s) 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: Manager daemon x is now available 2024-04-21T08:35:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:35:05.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:35:05.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' 2024-04-21T08:35:05.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' 2024-04-21T08:35:05.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:05 smithi057 ceph-mon[27968]: from='mgr.14100 172.21.15.57:0/828484896' entity='mgr.x' 2024-04-21T08:35:06.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:06 smithi057 ceph-mon[27968]: mgrmap e3: x(active, since 1.00785s) 2024-04-21T08:35:07.504 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:07 smithi057 ceph-mon[27968]: mgrmap e4: x(active, since 2s) 2024-04-21T08:35:08.303 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:35:08.303 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:08.303 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:35:08.303 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "health": { 2024-04-21T08:35:08.303 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-21T08:35:08.303 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 0 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "a" 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.304 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-21T08:35:08.305 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-21T08:35:08.306 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "restful" 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ], 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "modified": "2024-04-21T08:34:49.550788+0000", 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout }, 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a72725700 1 Processor -- start 2024-04-21T08:35:08.307 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a72725700 1 -- start start 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a72725700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a72725700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c108da0 con 0x7f2a6c1083b0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a6bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a6bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56900/0 (socket says 172.21.15.57:56900) 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.201+0000 7f2a6bfff700 1 -- 172.21.15.57:0/3599246818 learned_addr learned my addr 172.21.15.57:0/3599246818 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.202+0000 7f2a6bfff700 1 -- 172.21.15.57:0/3599246818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c109560 con 0x7f2a6c1083b0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.202+0000 7f2a6bfff700 1 --2- 172.21.15.57:0/3599246818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c1087d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2a540098a0 tx=0x7f2a54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a3fb84e31dd252 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.202+0000 7f2a6affd700 1 -- 172.21.15.57:0/3599246818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a54004160 con 0x7f2a6c1083b0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.202+0000 7f2a6affd700 1 -- 172.21.15.57:0/3599246818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a54004300 con 0x7f2a6c1083b0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.202+0000 7f2a6affd700 1 -- 172.21.15.57:0/3599246818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a54011af0 con 0x7f2a6c1083b0 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 -- 172.21.15.57:0/3599246818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 msgr2=0x7f2a6c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 --2- 172.21.15.57:0/3599246818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c1087d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2a540098a0 tx=0x7f2a54009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 -- 172.21.15.57:0/3599246818 shutdown_connections 2024-04-21T08:35:08.308 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 --2- 172.21.15.57:0/3599246818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c1087d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 -- 172.21.15.57:0/3599246818 >> 172.21.15.57:0/3599246818 conn(0x7f2a6c1039b0 msgr2=0x7f2a6c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 -- 172.21.15.57:0/3599246818 shutdown_connections 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 -- 172.21.15.57:0/3599246818 wait complete. 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.203+0000 7f2a72725700 1 Processor -- start 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a72725700 1 -- start start 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a72725700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a72725700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c10c910 con 0x7f2a6c1083b0 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a6bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a6bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c19c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56914/0 (socket says 172.21.15.57:56914) 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a6bfff700 1 -- 172.21.15.57:0/2711583751 learned_addr learned my addr 172.21.15.57:0/2711583751 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a6bfff700 1 -- 172.21.15.57:0/2711583751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c19c790 con 0x7f2a6c1083b0 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.204+0000 7f2a6bfff700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c19c250 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2a54000c00 tx=0x7f2a54011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.205+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a540039e0 con 0x7f2a6c1083b0 2024-04-21T08:35:08.309 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.205+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2a54003b80 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.205+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a54024730 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.205+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6c19ca20 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.205+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6c19f710 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.206+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f2a54022020 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.206+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a6c0640a0 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.206+0000 7f2a697fa700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2a58039620 0x7f2a5803bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.206+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a5404d370 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.206+0000 7f2a6b7fe700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2a58039620 0x7f2a5803bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.206+0000 7f2a6b7fe700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2a58039620 0x7f2a5803bae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c006fd0 tx=0x7f2a5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.211+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2a5402c560 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.505+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2a6c19f8e0 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.505+0000 7f2a697fa700 1 -- 172.21.15.57:0/2711583751 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2a6c19f8e0 con 0x7f2a6c1083b0 2024-04-21T08:35:08.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2a58039620 msgr2=0x7f2a5803bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2a58039620 0x7f2a5803bae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c006fd0 tx=0x7f2a5c006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 msgr2=0x7f2a6c19c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c19c250 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2a54000c00 tx=0x7f2a54011b70 comp rx=0 tx=0).stop 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 shutdown_connections 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2a58039620 0x7f2a5803bae0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 --2- 172.21.15.57:0/2711583751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a6c1083b0 0x7f2a6c19c250 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 >> 172.21.15.57:0/2711583751 conn(0x7f2a6c1039b0 msgr2=0x7f2a6c1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 shutdown_connections 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:07.508+0000 7f2a72725700 1 -- 172.21.15.57:0/2711583751 wait complete. 2024-04-21T08:35:08.311 INFO:teuthology.orchestra.run.smithi057.stdout:mgr is available 2024-04-21T08:35:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:08 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2711583751' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-21T08:35:09.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:09 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2031830745' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [global] 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout fsid = df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 2024-04-21T08:35:09.524 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout [osd] 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2ac00bb700 1 Processor -- start 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2ac00bb700 1 -- start start 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2ac00bb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab8107080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2ac00bb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab807be00 con 0x7f2ab8106c60 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2abde57700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab8107080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2abde57700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab8107080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56930/0 (socket says 172.21.15.57:56930) 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2abde57700 1 -- 172.21.15.57:0/1177749966 learned_addr learned my addr 172.21.15.57:0/1177749966 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.707+0000 7f2abde57700 1 -- 172.21.15.57:0/1177749966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab81075c0 con 0x7f2ab8106c60 2024-04-21T08:35:09.525 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.708+0000 7f2abde57700 1 --2- 172.21.15.57:0/1177749966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab8107080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40098a0 tx=0x7f2ab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d25f8081ae7d493 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.708+0000 7f2abce55700 1 -- 172.21.15.57:0/1177749966 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab4004160 con 0x7f2ab8106c60 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.708+0000 7f2abce55700 1 -- 172.21.15.57:0/1177749966 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ab4004300 con 0x7f2ab8106c60 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.708+0000 7f2abce55700 1 -- 172.21.15.57:0/1177749966 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab4011af0 con 0x7f2ab8106c60 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.708+0000 7f2ac00bb700 1 -- 172.21.15.57:0/1177749966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 msgr2=0x7f2ab8107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.708+0000 7f2ac00bb700 1 --2- 172.21.15.57:0/1177749966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab8107080 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40098a0 tx=0x7f2ab4009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 -- 172.21.15.57:0/1177749966 shutdown_connections 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 --2- 172.21.15.57:0/1177749966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab8107080 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 -- 172.21.15.57:0/1177749966 >> 172.21.15.57:0/1177749966 conn(0x7f2ab8102840 msgr2=0x7f2ab8104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 -- 172.21.15.57:0/1177749966 shutdown_connections 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 -- 172.21.15.57:0/1177749966 wait complete. 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 Processor -- start 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 -- start start 2024-04-21T08:35:09.526 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.709+0000 7f2ac00bb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81085f0 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2abde57700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2abde57700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab819e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56936/0 (socket says 172.21.15.57:56936) 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2abde57700 1 -- 172.21.15.57:0/2031830745 learned_addr learned my addr 172.21.15.57:0/2031830745 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2abde57700 1 -- 172.21.15.57:0/2031830745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab819e9a0 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2abde57700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab819e460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4011ac0 tx=0x7f2ab4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab4003970 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2ab4003b10 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab819ec30 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ab4024610 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.710+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab81a1920 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.711+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f2ab4024840 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.711+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab80640a0 con 0x7f2ab8106c60 2024-04-21T08:35:09.527 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.711+0000 7f2aaaffd700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2aa4039670 0x7f2aa403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.711+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2ab404d9e0 con 0x7f2ab8106c60 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.712+0000 7f2abd656700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2aa4039670 0x7f2aa403bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.712+0000 7f2abd656700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2aa4039670 0x7f2aa403bb30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2aac006fd0 tx=0x7f2aac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.717+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ab404d400 con 0x7f2ab8106c60 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.923+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2ab8076600 con 0x7f2ab8106c60 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.925+0000 7f2aaaffd700 1 -- 172.21.15.57:0/2031830745 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f2ab400bde0 con 0x7f2ab8106c60 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2aa4039670 msgr2=0x7f2aa403bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2aa4039670 0x7f2aa403bb30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2aac006fd0 tx=0x7f2aac006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 msgr2=0x7f2ab819e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab819e460 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4011ac0 tx=0x7f2ab4011ba0 comp rx=0 tx=0).stop 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 shutdown_connections 2024-04-21T08:35:09.528 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f2aa4039670 0x7f2aa403bb30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:09.529 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 --2- 172.21.15.57:0/2031830745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ab8106c60 0x7f2ab819e460 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:09.529 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 >> 172.21.15.57:0/2031830745 conn(0x7f2ab8102840 msgr2=0x7f2ab8195020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:09.529 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 shutdown_connections 2024-04-21T08:35:09.529 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:08.928+0000 7f2ac00bb700 1 -- 172.21.15.57:0/2031830745 wait complete. 2024-04-21T08:35:09.529 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling cephadm module... 2024-04-21T08:35:10.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:10 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3395342673' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-21T08:35:10.487 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: ignoring --setuser ceph since I am not root 2024-04-21T08:35:10.487 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: ignoring --setgroup ceph since I am not root 2024-04-21T08:35:10.795 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:10.499+0000 7ffbfe99d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:35:10.795 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:10.690+0000 7ffbfe99d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:10.795 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:10.796+0000 7ffbfe99d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3196e77700 1 Processor -- start 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3196e77700 1 -- start start 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3196e77700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319010b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3196e77700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f319007be60 con 0x7f3190108e30 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3194c13700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319010b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3194c13700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319010b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56940/0 (socket says 172.21.15.57:56940) 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3194c13700 1 -- 172.21.15.57:0/3476019395 learned_addr learned my addr 172.21.15.57:0/3476019395 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:10.996 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3194c13700 1 -- 172.21.15.57:0/3476019395 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319010b760 con 0x7f3190108e30 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.913+0000 7f3194c13700 1 --2- 172.21.15.57:0/3476019395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319010b220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f3184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f025af6ab3c1874 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f318f7fe700 1 -- 172.21.15.57:0/3476019395 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184004160 con 0x7f3190108e30 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f318f7fe700 1 -- 172.21.15.57:0/3476019395 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3184004300 con 0x7f3190108e30 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f318f7fe700 1 -- 172.21.15.57:0/3476019395 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184011af0 con 0x7f3190108e30 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 -- 172.21.15.57:0/3476019395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 msgr2=0x7f319010b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 --2- 172.21.15.57:0/3476019395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319010b220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f3184009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 -- 172.21.15.57:0/3476019395 shutdown_connections 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 --2- 172.21.15.57:0/3476019395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319010b220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:10.997 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 -- 172.21.15.57:0/3476019395 >> 172.21.15.57:0/3476019395 conn(0x7f3190102860 msgr2=0x7f3190104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 -- 172.21.15.57:0/3476019395 shutdown_connections 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.914+0000 7f3196e77700 1 -- 172.21.15.57:0/3476019395 wait complete. 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3196e77700 1 Processor -- start 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3196e77700 1 -- start start 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3196e77700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319019c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3196e77700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f319010c720 con 0x7f3190108e30 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3194c13700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319019c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:10.998 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3194c13700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319019c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56956/0 (socket says 172.21.15.57:56956) 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.915+0000 7f3194c13700 1 -- 172.21.15.57:0/3395342673 learned_addr learned my addr 172.21.15.57:0/3395342673 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f3194c13700 1 -- 172.21.15.57:0/3395342673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319019c710 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f3194c13700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319019c1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3184003f40 tx=0x7f3184011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f31840038b0 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3184003a50 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f318401a5c0 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319019c9a0 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.916+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f319019f690 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.917+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f3184024430 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.917+0000 7f318dffb700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f3180039670 0x7f318003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.917+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31900640a0 con 0x7f3190108e30 2024-04-21T08:35:10.999 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.917+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f318404daa0 con 0x7f3190108e30 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.917+0000 7f318ffff700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f3180039670 0x7f318003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.918+0000 7f318ffff700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f3180039670 0x7f318003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f317c006fd0 tx=0x7f317c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:09.922+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3184017d70 con 0x7f3190108e30 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.194+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f319019f860 con 0x7f3190108e30 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.324+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f319019f860 con 0x7f3190108e30 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.326+0000 7f318dffb700 1 -- 172.21.15.57:0/3395342673 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f318402d910 con 0x7f3190108e30 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f3180039670 msgr2=0x7f318003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f3180039670 0x7f318003bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f317c006fd0 tx=0x7f317c006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 msgr2=0x7f319019c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319019c1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3184003f40 tx=0x7f3184011890 comp rx=0 tx=0).stop 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 shutdown_connections 2024-04-21T08:35:11.000 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f3180039670 0x7f318003bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.001 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 --2- 172.21.15.57:0/3395342673 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3190108e30 0x7f319019c1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:11.001 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 >> 172.21.15.57:0/3395342673 conn(0x7f3190102860 msgr2=0x7f31901032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:11.001 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 shutdown_connections 2024-04-21T08:35:11.001 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:10.327+0000 7f3196e77700 1 -- 172.21.15.57:0/3395342673 wait complete. 2024-04-21T08:35:11.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:11 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3395342673' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-21T08:35:11.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:11 smithi057 ceph-mon[27968]: mgrmap e5: x(active, since 6s) 2024-04-21T08:35:11.406 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:11 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:11.118+0000 7ffbfe99d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:35:11.406 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:11 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:11.208+0000 7ffbfe99d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:35:12.254 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.409+0000 7f983e22f700 1 Processor -- start 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.409+0000 7f983e22f700 1 -- start start 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f983e22f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983810b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f983e22f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f983807bea0 con 0x7f9838108db0 2024-04-21T08:35:12.255 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f98377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983810b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f98377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983810b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56994/0 (socket says 172.21.15.57:56994) 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f98377fe700 1 -- 172.21.15.57:0/3367856090 learned_addr learned my addr 172.21.15.57:0/3367856090 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f98377fe700 1 -- 172.21.15.57:0/3367856090 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983810b6e0 con 0x7f9838108db0 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.410+0000 7f98377fe700 1 --2- 172.21.15.57:0/3367856090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983810b1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98280098a0 tx=0x7f9828009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b32867f3773df9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f98367fc700 1 -- 172.21.15.57:0/3367856090 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9828004160 con 0x7f9838108db0 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f98367fc700 1 -- 172.21.15.57:0/3367856090 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9828004300 con 0x7f9838108db0 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f98367fc700 1 -- 172.21.15.57:0/3367856090 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9828011af0 con 0x7f9838108db0 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 -- 172.21.15.57:0/3367856090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 msgr2=0x7f983810b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 --2- 172.21.15.57:0/3367856090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983810b1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f98280098a0 tx=0x7f9828009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 -- 172.21.15.57:0/3367856090 shutdown_connections 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 --2- 172.21.15.57:0/3367856090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983810b1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 -- 172.21.15.57:0/3367856090 >> 172.21.15.57:0/3367856090 conn(0x7f9838102820 msgr2=0x7f9838104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 -- 172.21.15.57:0/3367856090 shutdown_connections 2024-04-21T08:35:12.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.411+0000 7f983e22f700 1 -- 172.21.15.57:0/3367856090 wait complete. 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f983e22f700 1 Processor -- start 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f983e22f700 1 -- start start 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f983e22f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983819c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f983e22f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f983810c6b0 con 0x7f9838108db0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f98377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983819c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f98377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983819c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56998/0 (socket says 172.21.15.57:56998) 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.412+0000 7f98377fe700 1 -- 172.21.15.57:0/933944084 learned_addr learned my addr 172.21.15.57:0/933944084 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f98377fe700 1 -- 172.21.15.57:0/933944084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983819c740 con 0x7f9838108db0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f98377fe700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983819c200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9828011ac0 tx=0x7f9828011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9828003b20 con 0x7f9838108db0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f982801a8c0 con 0x7f9838108db0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983819c9d0 con 0x7f9838108db0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98280248d0 con 0x7f9838108db0 2024-04-21T08:35:12.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.413+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f983819f6c0 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.414+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9828022020 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.414+0000 7f9834ff9700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 0x7f982403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.414+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98380640a0 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.414+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f982804d520 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.414+0000 7f9836ffd700 1 -- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 msgr2=0x7f982403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.414+0000 7f9836ffd700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 0x7f982403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.420+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f982802cdf0 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.615+0000 7f9836ffd700 1 -- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 msgr2=0x7f982403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.615+0000 7f9836ffd700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 0x7f982403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.730+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f983819f890 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.730+0000 7f9834ff9700 1 -- 172.21.15.57:0/933944084 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f983819f890 con 0x7f9838108db0 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 msgr2=0x7f982403bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 0x7f982403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 msgr2=0x7f983819c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983819c200 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9828011ac0 tx=0x7f9828011ba0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 shutdown_connections 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7f98240396c0 0x7f982403bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 --2- 172.21.15.57:0/933944084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9838108db0 0x7f983819c200 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.732+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 >> 172.21.15.57:0/933944084 conn(0x7f9838102820 msgr2=0x7f9838103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.733+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 shutdown_connections 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:11.733+0000 7f983e22f700 1 -- 172.21.15.57:0/933944084 wait complete. 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for the mgr to restart... 2024-04-21T08:35:12.259 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr epoch 5... 2024-04-21T08:35:12.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/933944084' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:35:12.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:12 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:12.469+0000 7ffbfe99d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:35:12.923 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:12 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:12.649+0000 7ffbfe99d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:12.923 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:12 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:12.827+0000 7ffbfe99d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:35:13.221 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:13.126+0000 7ffbfe99d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:35:13.221 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:13.222+0000 7ffbfe99d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:35:13.588 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:13.432+0000 7ffbfe99d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:35:14.089 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:13.631+0000 7ffbfe99d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:14.588 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:14.255+0000 7ffbfe99d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:35:15.004 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:14.696+0000 7ffbfe99d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:35:15.004 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:14.796+0000 7ffbfe99d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:35:15.004 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:15.004+0000 7ffbfe99d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:35:15.838 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:15.546+0000 7ffbfe99d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:35:15.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:15.643+0000 7ffbfe99d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:35:16.257 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:15.897+0000 7ffbfe99d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:35:16.257 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:16.037+0000 7ffbfe99d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:35:16.257 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:16.135+0000 7ffbfe99d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:35:16.257 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:16.257+0000 7ffbfe99d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:17.088 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:16.593+0000 7ffbfe99d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:35:17.496 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:17.220+0000 7ffbfe99d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:35:17.496 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:17.444+0000 7ffbfe99d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: Active manager daemon x restarted 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: Activating manager daemon x 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: osdmap e2: 0 total, 0 up, 0 in 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: mgrmap e6: x(active, starting, since 0.00824178s) 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:35:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: Manager daemon x is now available 2024-04-21T08:35:17.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:17.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:17.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:17 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:18.773 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:18 smithi057 ceph-mon[27968]: Found migration_current of "None". Setting to last migration. 2024-04-21T08:35:18.773 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:18 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:18.773 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:18 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:35:18.773 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:18 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:35:18.773 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:18 smithi057 ceph-mon[27968]: mgrmap e7: x(active, since 1.01484s) 2024-04-21T08:35:19.025 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:19.025 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.718+0000 7fa46b79e700 1 Processor -- start 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.718+0000 7fa46b79e700 1 -- start start 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.718+0000 7fa46b79e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46410b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.718+0000 7fa46b79e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa46407be60 con 0x7fa464108df0 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.718+0000 7fa46953a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46410b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa46953a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46410b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58014/0 (socket says 172.21.15.57:58014) 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa46953a700 1 -- 172.21.15.57:0/2141850365 learned_addr learned my addr 172.21.15.57:0/2141850365 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa46953a700 1 -- 172.21.15.57:0/2141850365 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46410b720 con 0x7fa464108df0 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa46953a700 1 --2- 172.21.15.57:0/2141850365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46410b1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa4540098a0 tx=0x7fa454009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ef742e4dad6f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:19.026 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa45bfff700 1 -- 172.21.15.57:0/2141850365 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa454004160 con 0x7fa464108df0 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa45bfff700 1 -- 172.21.15.57:0/2141850365 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa454004300 con 0x7fa464108df0 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.719+0000 7fa45bfff700 1 -- 172.21.15.57:0/2141850365 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa454011af0 con 0x7fa464108df0 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 -- 172.21.15.57:0/2141850365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 msgr2=0x7fa46410b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 --2- 172.21.15.57:0/2141850365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46410b1e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa4540098a0 tx=0x7fa454009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 -- 172.21.15.57:0/2141850365 shutdown_connections 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 --2- 172.21.15.57:0/2141850365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46410b1e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 -- 172.21.15.57:0/2141850365 >> 172.21.15.57:0/2141850365 conn(0x7fa464102820 msgr2=0x7fa464104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 -- 172.21.15.57:0/2141850365 shutdown_connections 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.720+0000 7fa46b79e700 1 -- 172.21.15.57:0/2141850365 wait complete. 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46b79e700 1 Processor -- start 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46b79e700 1 -- start start 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46b79e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46b79e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa46410c6e0 con 0x7fa464108df0 2024-04-21T08:35:19.027 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46953a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46953a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46419c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58030/0 (socket says 172.21.15.57:58030) 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46953a700 1 -- 172.21.15.57:0/1873805232 learned_addr learned my addr 172.21.15.57:0/1873805232 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.721+0000 7fa46953a700 1 -- 172.21.15.57:0/1873805232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46419c730 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.722+0000 7fa46953a700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46419c1f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa454011ac0 tx=0x7fa454011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.722+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa454003b20 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.722+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa45401a8c0 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.722+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46419c9c0 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.722+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4540248d0 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.722+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46419f6b0 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.723+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fa454022020 con 0x7fa464108df0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.723+0000 7fa45a7fc700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.723+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 --> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa45003c290 con 0x7fa4500396c0 2024-04-21T08:35:19.028 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.723+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa45404d530 con 0x7fa464108df0 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.723+0000 7fa468d39700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 msgr2=0x7fa45003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.723+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.924+0000 7fa468d39700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 msgr2=0x7fa45003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:12.924+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.325+0000 7fa468d39700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 msgr2=0x7fa45003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:13.325+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.126+0000 7fa468d39700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 msgr2=0x7fa45003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:14.126+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.727+0000 7fa468d39700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 msgr2=0x7fa45003bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3383203302 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:15.727+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.452+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fa45402d3f0 con 0x7fa464108df0 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.452+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 msgr2=0x7fa45003bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:17.452+0000 7fa45a7fc700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.029 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.458+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fa45404f0e0 con 0x7fa464108df0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.458+0000 7fa45a7fc700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa45003cfb0 0x7fa45003f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.458+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa454029e50 con 0x7fa45003cfb0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.458+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa45003cfb0 0x7fa45003f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.459+0000 7fa468d39700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa45003cfb0 0x7fa45003f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa460003a10 tx=0x7fa4600092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.459+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fa454029e50 con 0x7fa45003cfb0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa464069b50 con 0x7fa45003cfb0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa45a7fc700 1 -- 172.21.15.57:0/1873805232 <== mgr.14120 v2:172.21.15.57:6800/2928520021 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fa464069b50 con 0x7fa45003cfb0 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa45003cfb0 msgr2=0x7fa45003f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa45003cfb0 0x7fa45003f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa460003a10 tx=0x7fa4600092b0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 msgr2=0x7fa46419c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46419c1f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa454011ac0 tx=0x7fa454011ba0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 shutdown_connections 2024-04-21T08:35:19.030 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa45003cfb0 0x7fa45003f3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa464108df0 0x7fa46419c1f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 --2- 172.21.15.57:0/1873805232 >> [v2:172.21.15.57:6800/3383203302,v1:172.21.15.57:6801/3383203302] conn(0x7fa4500396c0 0x7fa45003bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.466+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 >> 172.21.15.57:0/1873805232 conn(0x7fa464102820 msgr2=0x7fa464103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.467+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 shutdown_connections 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:18.467+0000 7fa46b79e700 1 -- 172.21.15.57:0/1873805232 wait complete. 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:mgr epoch 5 is available 2024-04-21T08:35:19.031 INFO:teuthology.orchestra.run.smithi057.stdout:Setting orchestrator backend to cephadm... 2024-04-21T08:35:19.970 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:35:19.970 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:35:19.970 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:19.970 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:19.970 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:19] ENGINE Bus STARTING 2024-04-21T08:35:19.971 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:19.971 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:19.971 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:19 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:20.279 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf4579700 1 Processor -- start 2024-04-21T08:35:20.279 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf4579700 1 -- start start 2024-04-21T08:35:20.279 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf4579700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec108780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:20.279 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf4579700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fddec108d50 con 0x7fddec108360 2024-04-21T08:35:20.279 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf2315700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf2315700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58108/0 (socket says 172.21.15.57:58108) 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf2315700 1 -- 172.21.15.57:0/1782870590 learned_addr learned my addr 172.21.15.57:0/1782870590 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.412+0000 7fddf2315700 1 -- 172.21.15.57:0/1782870590 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec109560 con 0x7fddec108360 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.413+0000 7fddf2315700 1 --2- 172.21.15.57:0/1782870590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec108780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdde80098a0 tx=0x7fdde8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7e642667ca613d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.413+0000 7fddf1313700 1 -- 172.21.15.57:0/1782870590 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde8004160 con 0x7fddec108360 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.413+0000 7fddf1313700 1 -- 172.21.15.57:0/1782870590 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdde8004300 con 0x7fddec108360 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.413+0000 7fddf1313700 1 -- 172.21.15.57:0/1782870590 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde8011af0 con 0x7fddec108360 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.413+0000 7fddf4579700 1 -- 172.21.15.57:0/1782870590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 msgr2=0x7fddec108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.413+0000 7fddf4579700 1 --2- 172.21.15.57:0/1782870590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec108780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdde80098a0 tx=0x7fdde8009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 -- 172.21.15.57:0/1782870590 shutdown_connections 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 --2- 172.21.15.57:0/1782870590 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec108780 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:20.280 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 -- 172.21.15.57:0/1782870590 >> 172.21.15.57:0/1782870590 conn(0x7fddec1038e0 msgr2=0x7fddec105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 -- 172.21.15.57:0/1782870590 shutdown_connections 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 -- 172.21.15.57:0/1782870590 wait complete. 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 Processor -- start 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 -- start start 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec07db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.414+0000 7fddf4579700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fddec10c910 con 0x7fddec108360 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf2315700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec07db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf2315700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec07db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58122/0 (socket says 172.21.15.57:58122) 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf2315700 1 -- 172.21.15.57:0/2839366421 learned_addr learned my addr 172.21.15.57:0/2839366421 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf2315700 1 -- 172.21.15.57:0/2839366421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec07e0a0 con 0x7fddec108360 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf2315700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec07db60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdde8003f40 tx=0x7fdde8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde80038b0 con 0x7fddec108360 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdde8003a50 con 0x7fddec108360 2024-04-21T08:35:20.281 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddec07c290 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde801a5c0 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.415+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddec07c7d0 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.416+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fdde8024430 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.416+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddec0640a0 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.416+0000 7fdde37fe700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fddd8039510 0x7fddd803b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.416+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdde804d910 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.416+0000 7fddf1b14700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fddd8039510 0x7fddd803b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.417+0000 7fddf1b14700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fddd8039510 0x7fddd803b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdddc006fd0 tx=0x7fdddc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.422+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdde802d910 con 0x7fddec108360 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.648+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fddec104ab0 con 0x7fddd8039510 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.658+0000 7fdde37fe700 1 -- 172.21.15.57:0/2839366421 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fddec104ab0 con 0x7fddd8039510 2024-04-21T08:35:20.282 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fddd8039510 msgr2=0x7fddd803b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fddd8039510 0x7fddd803b9d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdddc006fd0 tx=0x7fdddc006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 msgr2=0x7fddec07db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec07db60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdde8003f40 tx=0x7fdde8011890 comp rx=0 tx=0).stop 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 shutdown_connections 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fddd8039510 0x7fddd803b9d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 --2- 172.21.15.57:0/2839366421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fddec108360 0x7fddec07db60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 >> 172.21.15.57:0/2839366421 conn(0x7fddec1038e0 msgr2=0x7fddec1043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.660+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 shutdown_connections 2024-04-21T08:35:20.283 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:19.661+0000 7fddf4579700 1 -- 172.21.15.57:0/2839366421 wait complete. 2024-04-21T08:35:20.959 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:20 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:19] ENGINE Serving on https://172.21.15.57:7150 2024-04-21T08:35:20.959 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:20 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:19] ENGINE Serving on http://172.21.15.57:8765 2024-04-21T08:35:20.959 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:20 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:19] ENGINE Bus STARTED 2024-04-21T08:35:20.959 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:20 smithi057 ceph-mon[27968]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:20.959 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:20 smithi057 ceph-mon[27968]: mgrmap e8: x(active, since 2s) 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.739+0000 7fdaeb74a700 1 Processor -- start 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.739+0000 7fdaeb74a700 1 -- start start 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdaeb74a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae410b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdaeb74a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdae407bea0 con 0x7fdae4108db0 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae94e6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae410b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae94e6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae410b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58126/0 (socket says 172.21.15.57:58126) 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae94e6700 1 -- 172.21.15.57:0/1914251121 learned_addr learned my addr 172.21.15.57:0/1914251121 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae94e6700 1 -- 172.21.15.57:0/1914251121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae410b6e0 con 0x7fdae4108db0 2024-04-21T08:35:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae94e6700 1 --2- 172.21.15.57:0/1914251121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae410b1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009e80 tx=0x7fdadc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f841885b4bc0ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae3fff700 1 -- 172.21.15.57:0/1914251121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdadc017070 con 0x7fdae4108db0 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae3fff700 1 -- 172.21.15.57:0/1914251121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdadc00b8c0 con 0x7fdae4108db0 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.740+0000 7fdae3fff700 1 -- 172.21.15.57:0/1914251121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdadc004490 con 0x7fdae4108db0 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 -- 172.21.15.57:0/1914251121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 msgr2=0x7fdae410b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 --2- 172.21.15.57:0/1914251121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae410b1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009e80 tx=0x7fdadc00b330 comp rx=0 tx=0).stop 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 -- 172.21.15.57:0/1914251121 shutdown_connections 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 --2- 172.21.15.57:0/1914251121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae410b1a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 -- 172.21.15.57:0/1914251121 >> 172.21.15.57:0/1914251121 conn(0x7fdae4102820 msgr2=0x7fdae4104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 -- 172.21.15.57:0/1914251121 shutdown_connections 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.741+0000 7fdaeb74a700 1 -- 172.21.15.57:0/1914251121 wait complete. 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdaeb74a700 1 Processor -- start 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdaeb74a700 1 -- start start 2024-04-21T08:35:21.494 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdaeb74a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae419e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdaeb74a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdae410c6b0 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdae94e6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae419e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdae94e6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae419e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58132/0 (socket says 172.21.15.57:58132) 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdae94e6700 1 -- 172.21.15.57:0/3294664886 learned_addr learned my addr 172.21.15.57:0/3294664886 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.742+0000 7fdae94e6700 1 -- 172.21.15.57:0/3294664886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae419e940 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.743+0000 7fdae94e6700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae419e400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00b7b0 tx=0x7fdadc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.743+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdadc020020 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.743+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdadc012430 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.743+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdadc017040 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.743+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae419ebd0 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.743+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae41a18c0 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.744+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdadc023020 con 0x7fdae4108db0 2024-04-21T08:35:21.495 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.744+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae4069b50 con 0x7fdae4108db0 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.744+0000 7fdae27fc700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdacc039670 0x7fdacc03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.744+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdadc00f050 con 0x7fdae4108db0 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.744+0000 7fdae8ce5700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdacc039670 0x7fdacc03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.745+0000 7fdae8ce5700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdacc039670 0x7fdacc03bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdad8006fd0 tx=0x7fdad8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.749+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdadc02a020 con 0x7fdae4108db0 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.959+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fdae41039c0 con 0x7fdacc039670 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.960+0000 7fdae27fc700 1 -- 172.21.15.57:0/3294664886 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fdae41039c0 con 0x7fdacc039670 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.962+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdacc039670 msgr2=0x7fdacc03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.962+0000 7fdaeb74a700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdacc039670 0x7fdacc03bb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdad8006fd0 tx=0x7fdad8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.962+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 msgr2=0x7fdae419e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.962+0000 7fdaeb74a700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae419e400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00b7b0 tx=0x7fdadc00be20 comp rx=0 tx=0).stop 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.962+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 shutdown_connections 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.963+0000 7fdaeb74a700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdacc039670 0x7fdacc03bb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.963+0000 7fdaeb74a700 1 --2- 172.21.15.57:0/3294664886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdae4108db0 0x7fdae419e400 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:21.496 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.963+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 >> 172.21.15.57:0/3294664886 conn(0x7fdae4102820 msgr2=0x7fdae41032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:21.497 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.963+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 shutdown_connections 2024-04-21T08:35:21.497 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:20.963+0000 7fdaeb74a700 1 -- 172.21.15.57:0/3294664886 wait complete. 2024-04-21T08:35:21.497 INFO:teuthology.orchestra.run.smithi057.stdout:Generating ssh key... 2024-04-21T08:35:22.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:21 smithi057 ceph-mon[27968]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: Generating public/private rsa key pair. 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: Your identification has been saved in /tmp/tmpk_t4ffgr/key. 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: Your public key has been saved in /tmp/tmpk_t4ffgr/key.pub. 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: The key fingerprint is: 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: SHA256:Y2tbAm+Faukgi0yBRrqBxtv3dQzezF9V9ZdvoiTrRuY ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: The key's randomart image is: 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: +---[RSA 3072]----+ 2024-04-21T08:35:22.832 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: | .| 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: | +| 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: | . .+| 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: |* . +| 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: |== . S.o . . +| 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: |oo+ *.+O+ . o.| 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: |.o....+ **o*. . | 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: |o. o.+.o.=E . . | 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: |... ..... . | 2024-04-21T08:35:22.833 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: +----[SHA256]-----+ 2024-04-21T08:35:23.032 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.909+0000 7f7992467700 1 Processor -- start 2024-04-21T08:35:23.032 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.909+0000 7f7992467700 1 -- start start 2024-04-21T08:35:23.032 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.909+0000 7f7992467700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:23.032 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.909+0000 7f7992467700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f798c108da0 con 0x7f798c1083b0 2024-04-21T08:35:23.032 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58138/0 (socket says 172.21.15.57:58138) 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798bfff700 1 -- 172.21.15.57:0/1636501697 learned_addr learned my addr 172.21.15.57:0/1636501697 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798bfff700 1 -- 172.21.15.57:0/1636501697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c109560 con 0x7f798c1083b0 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798bfff700 1 --2- 172.21.15.57:0/1636501697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c1087d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f797c0098a0 tx=0x7f797c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4bcbf5ed4bab03 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798affd700 1 -- 172.21.15.57:0/1636501697 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f797c004160 con 0x7f798c1083b0 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798affd700 1 -- 172.21.15.57:0/1636501697 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f797c004300 con 0x7f798c1083b0 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.910+0000 7f798affd700 1 -- 172.21.15.57:0/1636501697 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f797c011af0 con 0x7f798c1083b0 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 -- 172.21.15.57:0/1636501697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 msgr2=0x7f798c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 --2- 172.21.15.57:0/1636501697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c1087d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f797c0098a0 tx=0x7f797c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 -- 172.21.15.57:0/1636501697 shutdown_connections 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 --2- 172.21.15.57:0/1636501697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c1087d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.033 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 -- 172.21.15.57:0/1636501697 >> 172.21.15.57:0/1636501697 conn(0x7f798c1039b0 msgr2=0x7f798c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 -- 172.21.15.57:0/1636501697 shutdown_connections 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 -- 172.21.15.57:0/1636501697 wait complete. 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 Processor -- start 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.911+0000 7f7992467700 1 -- start start 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f7992467700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f7992467700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f798c10c910 con 0x7f798c1083b0 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f798bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f798bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c19e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58150/0 (socket says 172.21.15.57:58150) 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f798bfff700 1 -- 172.21.15.57:0/3699566939 learned_addr learned my addr 172.21.15.57:0/3699566939 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f798bfff700 1 -- 172.21.15.57:0/3699566939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c19e940 con 0x7f798c1083b0 2024-04-21T08:35:23.034 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.912+0000 7f798bfff700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c19e400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f797c000c00 tx=0x7f797c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.913+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f797c003b20 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.913+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798c19ebd0 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.913+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f797c01a8c0 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.913+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f797c024940 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.913+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798c1a18c0 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.913+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f797c022020 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.914+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798c0640a0 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.914+0000 7f79897fa700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7974039670 0x7f797403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.914+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f797c04d860 con 0x7f798c1083b0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.914+0000 7f798b7fe700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7974039670 0x7f797403bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:23.035 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.914+0000 7f798b7fe700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7974039670 0x7f797403bb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7980006fd0 tx=0x7f7980006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:21.919+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f797c00be50 con 0x7f798c1083b0 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.127+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f798c104b20 con 0x7f7974039670 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.557+0000 7f79897fa700 1 -- 172.21.15.57:0/3699566939 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f798c104b20 con 0x7f7974039670 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7974039670 msgr2=0x7f797403bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7974039670 0x7f797403bb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7980006fd0 tx=0x7f7980006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 msgr2=0x7f798c19e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c19e400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f797c000c00 tx=0x7f797c011b70 comp rx=0 tx=0).stop 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 shutdown_connections 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7974039670 0x7f797403bb30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 --2- 172.21.15.57:0/3699566939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f798c1083b0 0x7f798c19e400 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 >> 172.21.15.57:0/3699566939 conn(0x7f798c1039b0 msgr2=0x7f798c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:23.036 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 shutdown_connections 2024-04-21T08:35:23.037 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:22.559+0000 7f7992467700 1 -- 172.21.15.57:0/3699566939 wait complete. 2024-04-21T08:35:23.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-mon[27968]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:23.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-mon[27968]: Generating ssh key... 2024-04-21T08:35:23.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:23.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:22 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd11c4a700 1 Processor -- start 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd11c4a700 1 -- start start 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd11c4a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd11c4a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c108da0 con 0x7fbd0c1083b0 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44988/0 (socket says 172.21.15.57:44988) 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd0b7fe700 1 -- 172.21.15.57:0/167867193 learned_addr learned my addr 172.21.15.57:0/167867193 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.449+0000 7fbd0b7fe700 1 -- 172.21.15.57:0/167867193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c109560 con 0x7fbd0c1083b0 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd0b7fe700 1 --2- 172.21.15.57:0/167867193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c1087d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0098a0 tx=0x7fbcfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8fd027c5190da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:24.322 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd0a7fc700 1 -- 172.21.15.57:0/167867193 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc004160 con 0x7fbd0c1083b0 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd0a7fc700 1 -- 172.21.15.57:0/167867193 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbcfc004300 con 0x7fbd0c1083b0 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd0a7fc700 1 -- 172.21.15.57:0/167867193 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc011af0 con 0x7fbd0c1083b0 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd11c4a700 1 -- 172.21.15.57:0/167867193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 msgr2=0x7fbd0c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd11c4a700 1 --2- 172.21.15.57:0/167867193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c1087d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0098a0 tx=0x7fbcfc009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd11c4a700 1 -- 172.21.15.57:0/167867193 shutdown_connections 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd11c4a700 1 --2- 172.21.15.57:0/167867193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c1087d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.450+0000 7fbd11c4a700 1 -- 172.21.15.57:0/167867193 >> 172.21.15.57:0/167867193 conn(0x7fbd0c1039b0 msgr2=0x7fbd0c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.451+0000 7fbd11c4a700 1 -- 172.21.15.57:0/167867193 shutdown_connections 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.451+0000 7fbd11c4a700 1 -- 172.21.15.57:0/167867193 wait complete. 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.451+0000 7fbd11c4a700 1 Processor -- start 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.451+0000 7fbd11c4a700 1 -- start start 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.451+0000 7fbd11c4a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:24.323 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.451+0000 7fbd11c4a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c10c910 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd0b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c19c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44992/0 (socket says 172.21.15.57:44992) 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd0b7fe700 1 -- 172.21.15.57:0/3644570745 learned_addr learned my addr 172.21.15.57:0/3644570745 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd0b7fe700 1 -- 172.21.15.57:0/3644570745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c19c790 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd0b7fe700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c19c250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc000c00 tx=0x7fbcfc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc0039e0 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0c19ca20 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbcfc003b80 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcfc024730 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.452+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0c19f710 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.453+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbcfc022020 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.453+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0c0640a0 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.453+0000 7fbd08ff9700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fbcf4039670 0x7fbcf403bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.453+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbcfc04d6d0 con 0x7fbd0c1083b0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.454+0000 7fbd0affd700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fbcf4039670 0x7fbcf403bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.454+0000 7fbd0affd700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fbcf4039670 0x7fbcf403bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbd00006fd0 tx=0x7fbd00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.458+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbcfc00b3e0 con 0x7fbd0c1083b0 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.668+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fbd0c104bb0 con 0x7fbcf4039670 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.669+0000 7fbd08ff9700 1 -- 172.21.15.57:0/3644570745 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fbd0c104bb0 con 0x7fbcf4039670 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fbcf4039670 msgr2=0x7fbcf403bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fbcf4039670 0x7fbcf403bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbd00006fd0 tx=0x7fbd00006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 msgr2=0x7fbd0c19c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c19c250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc000c00 tx=0x7fbcfc011b70 comp rx=0 tx=0).stop 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 shutdown_connections 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fbcf4039670 0x7fbcf403bb30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 --2- 172.21.15.57:0/3644570745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd0c1083b0 0x7fbd0c19c250 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 >> 172.21.15.57:0/3644570745 conn(0x7fbd0c1039b0 msgr2=0x7fbd0c1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 shutdown_connections 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:23.671+0000 7fbd11c4a700 1 -- 172.21.15.57:0/3644570745 wait complete. 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:35:24.325 INFO:teuthology.orchestra.run.smithi057.stdout:Adding key to root@localhost authorized_keys... 2024-04-21T08:35:24.326 INFO:teuthology.orchestra.run.smithi057.stdout:Adding host smithi057... 2024-04-21T08:35:24.581 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:24 smithi057 ceph-mon[27968]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:25.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:25 smithi057 ceph-mon[27968]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi057", "addr": "172.21.15.57", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:26.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:26 smithi057 ceph-mon[27968]: Deploying cephadm binary to smithi057 2024-04-21T08:35:29.886 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Added host 'smithi057' with addr '172.21.15.57' 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.795+0000 7f552253f700 1 Processor -- start 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.795+0000 7f552253f700 1 -- start start 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.795+0000 7f552253f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.795+0000 7f552253f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f551c108da0 con 0x7f551c1083b0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45000/0 (socket says 172.21.15.57:45000) 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551bfff700 1 -- 172.21.15.57:0/4030553455 learned_addr learned my addr 172.21.15.57:0/4030553455 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551bfff700 1 -- 172.21.15.57:0/4030553455 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c109560 con 0x7f551c1083b0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551bfff700 1 --2- 172.21.15.57:0/4030553455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c1087d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f550c0098a0 tx=0x7f550c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1023a644e863b127 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551affd700 1 -- 172.21.15.57:0/4030553455 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550c004160 con 0x7f551c1083b0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551affd700 1 -- 172.21.15.57:0/4030553455 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f550c004300 con 0x7f551c1083b0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.796+0000 7f551affd700 1 -- 172.21.15.57:0/4030553455 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550c011af0 con 0x7f551c1083b0 2024-04-21T08:35:29.887 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 -- 172.21.15.57:0/4030553455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 msgr2=0x7f551c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 --2- 172.21.15.57:0/4030553455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c1087d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f550c0098a0 tx=0x7f550c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 -- 172.21.15.57:0/4030553455 shutdown_connections 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 --2- 172.21.15.57:0/4030553455 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c1087d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 -- 172.21.15.57:0/4030553455 >> 172.21.15.57:0/4030553455 conn(0x7f551c1039b0 msgr2=0x7f551c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 -- 172.21.15.57:0/4030553455 shutdown_connections 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 -- 172.21.15.57:0/4030553455 wait complete. 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 Processor -- start 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.797+0000 7f552253f700 1 -- start start 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f552253f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f552253f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f551c10c910 con 0x7f551c1083b0 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f551bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f551bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c19c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45006/0 (socket says 172.21.15.57:45006) 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f551bfff700 1 -- 172.21.15.57:0/3474299766 learned_addr learned my addr 172.21.15.57:0/3474299766 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f551bfff700 1 -- 172.21.15.57:0/3474299766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c19c790 con 0x7f551c1083b0 2024-04-21T08:35:29.888 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.798+0000 7f551bfff700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c19c250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f550c003f40 tx=0x7f550c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.799+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550c0036a0 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.799+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f550c024430 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.799+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c19ca20 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.799+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f550c01a5c0 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.799+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c19f710 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.800+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f550c01a870 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.800+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c0640a0 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.800+0000 7f55197fa700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f5508039670 0x7f550803bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.800+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f550c04d730 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.800+0000 7f551b7fe700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f5508039670 0x7f550803bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.800+0000 7f551b7fe700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f5508039670 0x7f550803bb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5504006fd0 tx=0x7f5504006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:24.805+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f550c00b3e0 con 0x7f551c1083b0 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:25.019+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi057", "addr": "172.21.15.57", "target": ["mon-mgr", ""]}) -- 0x7f551c104bb0 con 0x7f5508039670 2024-04-21T08:35:29.889 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.273+0000 7f55197fa700 1 -- 172.21.15.57:0/3474299766 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f551c104bb0 con 0x7f5508039670 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.275+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f5508039670 msgr2=0x7f550803bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.275+0000 7f552253f700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f5508039670 0x7f550803bb30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5504006fd0 tx=0x7f5504006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.275+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 msgr2=0x7f551c19c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.275+0000 7f552253f700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c19c250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f550c003f40 tx=0x7f550c004790 comp rx=0 tx=0).stop 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.276+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 shutdown_connections 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.276+0000 7f552253f700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f5508039670 0x7f550803bb30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.276+0000 7f552253f700 1 --2- 172.21.15.57:0/3474299766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f551c1083b0 0x7f551c19c250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.276+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 >> 172.21.15.57:0/3474299766 conn(0x7f551c1039b0 msgr2=0x7f551c1044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.276+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 shutdown_connections 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:29.276+0000 7f552253f700 1 -- 172.21.15.57:0/3474299766 wait complete. 2024-04-21T08:35:29.890 INFO:teuthology.orchestra.run.smithi057.stdout:Deploying unmanaged mon service... 2024-04-21T08:35:30.552 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:30 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:30.552 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:30 smithi057 ceph-mon[27968]: Added host smithi057 2024-04-21T08:35:30.552 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:30 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.277+0000 7fb4cc384700 1 Processor -- start 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.277+0000 7fb4cc384700 1 -- start start 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.277+0000 7fb4cc384700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4108c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.277+0000 7fb4cc384700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c4109210 con 0x7fb4c4108820 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.278+0000 7fb4ca120700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.278+0000 7fb4ca120700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45008/0 (socket says 172.21.15.57:45008) 2024-04-21T08:35:31.286 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.278+0000 7fb4ca120700 1 -- 172.21.15.57:0/2043238719 learned_addr learned my addr 172.21.15.57:0/2043238719 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.278+0000 7fb4ca120700 1 -- 172.21.15.57:0/2043238719 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c4107430 con 0x7fb4c4108820 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.278+0000 7fb4ca120700 1 --2- 172.21.15.57:0/2043238719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4108c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009e80 tx=0x7fb4b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc7b2a862283a28 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.282+0000 7fb4c911e700 1 -- 172.21.15.57:0/2043238719 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4b4017070 con 0x7fb4c4108820 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.282+0000 7fb4c911e700 1 -- 172.21.15.57:0/2043238719 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb4b400b8c0 con 0x7fb4c4108820 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.282+0000 7fb4c911e700 1 -- 172.21.15.57:0/2043238719 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4b4004490 con 0x7fb4c4108820 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 -- 172.21.15.57:0/2043238719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 msgr2=0x7fb4c4108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 --2- 172.21.15.57:0/2043238719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4108c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4009e80 tx=0x7fb4b400b330 comp rx=0 tx=0).stop 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 -- 172.21.15.57:0/2043238719 shutdown_connections 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 --2- 172.21.15.57:0/2043238719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4108c40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 -- 172.21.15.57:0/2043238719 >> 172.21.15.57:0/2043238719 conn(0x7fb4c4101fa0 msgr2=0x7fb4c41043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 -- 172.21.15.57:0/2043238719 shutdown_connections 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.283+0000 7fb4cc384700 1 -- 172.21.15.57:0/2043238719 wait complete. 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4cc384700 1 Processor -- start 2024-04-21T08:35:31.287 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4cc384700 1 -- start start 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4cc384700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4cc384700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c4106320 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4ca120700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4ca120700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45012/0 (socket says 172.21.15.57:45012) 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4ca120700 1 -- 172.21.15.57:0/552509172 learned_addr learned my addr 172.21.15.57:0/552509172 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4ca120700 1 -- 172.21.15.57:0/552509172 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c419a500 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4ca120700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4199fc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4004a60 tx=0x7fb4b400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4b4020020 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb4b4003c60 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.284+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4b4017040 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.285+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c419a790 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.285+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c419d480 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.285+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb4b4023020 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.285+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4c40640a0 con 0x7fb4c4108820 2024-04-21T08:35:31.288 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.286+0000 7fb4c37fe700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fb4ac03b920 0x7fb4ac03dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.286+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb4b400f050 con 0x7fb4c4108820 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.286+0000 7fb4c991f700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fb4ac03b920 0x7fb4ac03dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.287+0000 7fb4c991f700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fb4ac03b920 0x7fb4ac03dde0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8006fd0 tx=0x7fb4b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.295+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4b402a020 con 0x7fb4c4108820 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.552+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fb4c4062e80 con 0x7fb4ac03b920 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.557+0000 7fb4c37fe700 1 -- 172.21.15.57:0/552509172 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb4c4062e80 con 0x7fb4ac03b920 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fb4ac03b920 msgr2=0x7fb4ac03dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fb4ac03b920 0x7fb4ac03dde0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8006fd0 tx=0x7fb4b8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 msgr2=0x7fb4c4199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4199fc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4004a60 tx=0x7fb4b400bd50 comp rx=0 tx=0).stop 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 shutdown_connections 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fb4ac03b920 0x7fb4ac03dde0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 --2- 172.21.15.57:0/552509172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb4c4108820 0x7fb4c4199fc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:31.289 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 >> 172.21.15.57:0/552509172 conn(0x7fb4c4101fa0 msgr2=0x7fb4c4190600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:31.290 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 shutdown_connections 2024-04-21T08:35:31.290 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:30.559+0000 7fb4cc384700 1 -- 172.21.15.57:0/552509172 wait complete. 2024-04-21T08:35:31.290 INFO:teuthology.orchestra.run.smithi057.stdout:Deploying unmanaged mgr service... 2024-04-21T08:35:31.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:31 smithi057 ceph-mon[27968]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:31.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:31 smithi057 ceph-mon[27968]: Saving service mon spec with placement count:5 2024-04-21T08:35:31.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:31 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:33.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:33 smithi057 ceph-mon[27968]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:33.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:33 smithi057 ceph-mon[27968]: Saving service mgr spec with placement count:2 2024-04-21T08:35:33.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:33 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:33.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:33 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:33.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-21T08:35:33.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.768+0000 7f6d55197700 1 Processor -- start 2024-04-21T08:35:33.422 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.768+0000 7f6d55197700 1 -- start start 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.768+0000 7f6d55197700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d50107060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.768+0000 7f6d55197700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5007be20 con 0x7f6d50106c40 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.768+0000 7f6d4ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d50107060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d50107060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45018/0 (socket says 172.21.15.57:45018) 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4ed9d700 1 -- 172.21.15.57:0/2139147707 learned_addr learned my addr 172.21.15.57:0/2139147707 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4ed9d700 1 -- 172.21.15.57:0/2139147707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d501075a0 con 0x7f6d50106c40 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4ed9d700 1 --2- 172.21.15.57:0/2139147707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d50107060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009e80 tx=0x7f6d4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540ed39ae292d8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4dd9b700 1 -- 172.21.15.57:0/2139147707 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d48017070 con 0x7f6d50106c40 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4dd9b700 1 -- 172.21.15.57:0/2139147707 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6d4800b8c0 con 0x7f6d50106c40 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.769+0000 7f6d4dd9b700 1 -- 172.21.15.57:0/2139147707 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d48004490 con 0x7f6d50106c40 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 -- 172.21.15.57:0/2139147707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 msgr2=0x7f6d50107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 --2- 172.21.15.57:0/2139147707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d50107060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009e80 tx=0x7f6d4800b330 comp rx=0 tx=0).stop 2024-04-21T08:35:33.423 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 -- 172.21.15.57:0/2139147707 shutdown_connections 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 --2- 172.21.15.57:0/2139147707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d50107060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 -- 172.21.15.57:0/2139147707 >> 172.21.15.57:0/2139147707 conn(0x7f6d50102820 msgr2=0x7f6d50104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 -- 172.21.15.57:0/2139147707 shutdown_connections 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.770+0000 7f6d55197700 1 -- 172.21.15.57:0/2139147707 wait complete. 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d55197700 1 Processor -- start 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d55197700 1 -- start start 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d55197700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d5019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d55197700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6d501085d0 con 0x7f6d50106c40 2024-04-21T08:35:33.424 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d4ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d5019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d4ed9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d5019e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45028/0 (socket says 172.21.15.57:45028) 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d4ed9d700 1 -- 172.21.15.57:0/3944647439 learned_addr learned my addr 172.21.15.57:0/3944647439 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.771+0000 7f6d4ed9d700 1 -- 172.21.15.57:0/3944647439 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5019e8f0 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.772+0000 7f6d4ed9d700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d5019e3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800b7b0 tx=0x7f6d4800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.772+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d48020020 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.772+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5019eb80 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.772+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6d48012430 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.772+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d48017040 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.772+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d501a1870 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.773+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6d48023020 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.773+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d500640a0 con 0x7f6d50106c40 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.773+0000 7f6d47fff700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f6d3c039620 0x7f6d3c03bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:33.425 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.773+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6d4800f050 con 0x7f6d50106c40 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.773+0000 7f6d4e59c700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f6d3c039620 0x7f6d3c03bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.773+0000 7f6d4e59c700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f6d3c039620 0x7f6d3c03bae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6d38006fd0 tx=0x7f6d38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:31.778+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d4802a020 con 0x7f6d50106c40 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.006+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6d5002e360 con 0x7f6d3c039620 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.011+0000 7f6d47fff700 1 -- 172.21.15.57:0/3944647439 <== mgr.14120 v2:172.21.15.57:6800/2928520021 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6d5002e360 con 0x7f6d3c039620 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f6d3c039620 msgr2=0x7f6d3c03bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f6d3c039620 0x7f6d3c03bae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6d38006fd0 tx=0x7f6d38006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 msgr2=0x7f6d5019e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d5019e3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800b7b0 tx=0x7f6d4800be20 comp rx=0 tx=0).stop 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 shutdown_connections 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f6d3c039620 0x7f6d3c03bae0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 --2- 172.21.15.57:0/3944647439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d50106c40 0x7f6d5019e3b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 >> 172.21.15.57:0/3944647439 conn(0x7f6d50102820 msgr2=0x7f6d50194ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:33.426 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 shutdown_connections 2024-04-21T08:35:33.427 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:32.013+0000 7f6d55197700 1 -- 172.21.15.57:0/3944647439 wait complete. 2024-04-21T08:35:34.753 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:34 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:34.754 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:34 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1100507907' entity='client.admin' 2024-04-21T08:35:35.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.881+0000 7f7eb6687700 1 Processor -- start 2024-04-21T08:35:35.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.881+0000 7f7eb6687700 1 -- start start 2024-04-21T08:35:35.364 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.881+0000 7f7eb6687700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb0107680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.881+0000 7f7eb6687700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb0107c50 con 0x7f7eb0107260 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb0107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb0107680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51180/0 (socket says 172.21.15.57:51180) 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaffff700 1 -- 172.21.15.57:0/2237858341 learned_addr learned my addr 172.21.15.57:0/2237858341 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaffff700 1 -- 172.21.15.57:0/2237858341 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb0108460 con 0x7f7eb0107260 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaffff700 1 --2- 172.21.15.57:0/2237858341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb0107680 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000bb40 tx=0x7f7ea000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce217c6bb85c5d80 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaeffd700 1 -- 172.21.15.57:0/2237858341 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ea000dbe0 con 0x7f7eb0107260 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaeffd700 1 -- 172.21.15.57:0/2237858341 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ea00044e0 con 0x7f7eb0107260 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.882+0000 7f7eaeffd700 1 -- 172.21.15.57:0/2237858341 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ea0010470 con 0x7f7eb0107260 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 -- 172.21.15.57:0/2237858341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 msgr2=0x7f7eb0107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 --2- 172.21.15.57:0/2237858341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb0107680 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000bb40 tx=0x7f7ea000be50 comp rx=0 tx=0).stop 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 -- 172.21.15.57:0/2237858341 shutdown_connections 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 --2- 172.21.15.57:0/2237858341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb0107680 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:35.365 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 -- 172.21.15.57:0/2237858341 >> 172.21.15.57:0/2237858341 conn(0x7f7eb0102820 msgr2=0x7f7eb0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 -- 172.21.15.57:0/2237858341 shutdown_connections 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.883+0000 7f7eb6687700 1 -- 172.21.15.57:0/2237858341 wait complete. 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eb6687700 1 Processor -- start 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eb6687700 1 -- start start 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eb6687700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eb6687700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb007afc0 con 0x7f7eb0107260 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eaffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eaffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51192/0 (socket says 172.21.15.57:51192) 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eaffff700 1 -- 172.21.15.57:0/1100507907 learned_addr learned my addr 172.21.15.57:0/1100507907 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eaffff700 1 -- 172.21.15.57:0/1100507907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb019e930 con 0x7f7eb0107260 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.884+0000 7f7eaffff700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb019e3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0003c30 tx=0x7f7ea0003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.885+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ea0010720 con 0x7f7eb0107260 2024-04-21T08:35:35.366 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.885+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ea001d070 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.885+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb019ebc0 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.885+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb01a18b0 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.885+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ea0021aa0 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.886+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7ea001f030 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.886+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e90005360 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.886+0000 7f7ead7fa700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7e9c039670 0x7f7e9c03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.886+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7ea004e3e0 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.886+0000 7f7eaf7fe700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7e9c039670 0x7f7e9c03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.887+0000 7f7eaf7fe700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7e9c039670 0x7f7e9c03bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e98006fd0 tx=0x7f7e98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:33.891+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ea0015070 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.109+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7e90005650 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.111+0000 7f7ead7fa700 1 -- 172.21.15.57:0/1100507907 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f7ea0021d40 con 0x7f7eb0107260 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7e9c039670 msgr2=0x7f7e9c03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:35.367 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7e9c039670 0x7f7e9c03bb30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7e98006fd0 tx=0x7f7e98006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 msgr2=0x7f7eb019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb019e3f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0003c30 tx=0x7f7ea0003d10 comp rx=0 tx=0).stop 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 shutdown_connections 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f7e9c039670 0x7f7e9c03bb30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 --2- 172.21.15.57:0/1100507907 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7eb0107260 0x7f7eb019e3f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 >> 172.21.15.57:0/1100507907 conn(0x7f7eb0102820 msgr2=0x7f7eb0190620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 shutdown_connections 2024-04-21T08:35:35.368 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:34.114+0000 7f7eb6687700 1 -- 172.21.15.57:0/1100507907 wait complete. 2024-04-21T08:35:36.312 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:36 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:36.312 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:36 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/483289524' entity='client.admin' 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.786+0000 7fd69bc55700 1 Processor -- start 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd69bc55700 1 -- start start 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd69bc55700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69410b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd69bc55700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd69407be60 con 0x7fd694108e00 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd6999f1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69410b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd6999f1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69410b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51198/0 (socket says 172.21.15.57:51198) 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd6999f1700 1 -- 172.21.15.57:0/1259747127 learned_addr learned my addr 172.21.15.57:0/1259747127 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:36.575 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.787+0000 7fd6999f1700 1 -- 172.21.15.57:0/1259747127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69410b730 con 0x7fd694108e00 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.788+0000 7fd6999f1700 1 --2- 172.21.15.57:0/1259747127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69410b1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6900098a0 tx=0x7fd690009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980a57bb3de4b633 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.788+0000 7fd6989ef700 1 -- 172.21.15.57:0/1259747127 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd690004160 con 0x7fd694108e00 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.788+0000 7fd6989ef700 1 -- 172.21.15.57:0/1259747127 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd690004300 con 0x7fd694108e00 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.788+0000 7fd6989ef700 1 -- 172.21.15.57:0/1259747127 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd690011a20 con 0x7fd694108e00 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.788+0000 7fd69bc55700 1 -- 172.21.15.57:0/1259747127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 msgr2=0x7fd69410b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 --2- 172.21.15.57:0/1259747127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69410b1f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6900098a0 tx=0x7fd690009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 -- 172.21.15.57:0/1259747127 shutdown_connections 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 --2- 172.21.15.57:0/1259747127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69410b1f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 -- 172.21.15.57:0/1259747127 >> 172.21.15.57:0/1259747127 conn(0x7fd694102890 msgr2=0x7fd694104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 -- 172.21.15.57:0/1259747127 shutdown_connections 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 -- 172.21.15.57:0/1259747127 wait complete. 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 Processor -- start 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 -- start start 2024-04-21T08:35:36.576 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69419c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.789+0000 7fd69bc55700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd69410c730 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd6999f1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69419c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd6999f1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69419c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51202/0 (socket says 172.21.15.57:51202) 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd6999f1700 1 -- 172.21.15.57:0/483289524 learned_addr learned my addr 172.21.15.57:0/483289524 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd6999f1700 1 -- 172.21.15.57:0/483289524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69419c680 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd6999f1700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69419c140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd690000c00 tx=0x7fd690011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd690003aa0 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69419c910 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd69001b8f0 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd690023870 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.790+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69419f600 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.791+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd690023b70 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.791+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6940640a0 con 0x7fd694108e00 2024-04-21T08:35:36.577 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.791+0000 7fd68affd700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fd680039670 0x7fd68003bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.791+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd69004dea0 con 0x7fd694108e00 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.792+0000 7fd6991f0700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fd680039670 0x7fd68003bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.792+0000 7fd6991f0700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fd680039670 0x7fd68003bb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd684006fd0 tx=0x7fd684006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:35.797+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd69001f030 con 0x7fd694108e00 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.007+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fd69419f7d0 con 0x7fd694108e00 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.014+0000 7fd68affd700 1 -- 172.21.15.57:0/483289524 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fd690018e30 con 0x7fd694108e00 2024-04-21T08:35:36.578 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fd680039670 msgr2=0x7fd68003bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:36.579 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fd680039670 0x7fd68003bb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd684006fd0 tx=0x7fd684006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:36.579 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 msgr2=0x7fd69419c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:36.579 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69419c140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd690000c00 tx=0x7fd690011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:36.579 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 shutdown_connections 2024-04-21T08:35:36.579 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fd680039670 0x7fd68003bb30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:36.579 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 --2- 172.21.15.57:0/483289524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd694108e00 0x7fd69419c140 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:36.580 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 >> 172.21.15.57:0/483289524 conn(0x7fd694102890 msgr2=0x7fd694103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:36.580 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 shutdown_connections 2024-04-21T08:35:36.580 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.017+0000 7fd69bc55700 1 -- 172.21.15.57:0/483289524 wait complete. 2024-04-21T08:35:36.580 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling the dashboard module... 2024-04-21T08:35:37.484 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-mon[27968]: from='mgr.14120 172.21.15.57:0/3977184544' entity='mgr.x' 2024-04-21T08:35:37.485 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/538833762' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-21T08:35:37.485 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: ignoring --setuser ceph since I am not root 2024-04-21T08:35:37.485 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: ignoring --setgroup ceph since I am not root 2024-04-21T08:35:37.802 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:37.503+0000 7f1faa033200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:35:37.802 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:37.694+0000 7f1faa033200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:37.802 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:37.801+0000 7f1faa033200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:35:38.390 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:38 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/538833762' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-21T08:35:38.391 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:38 smithi057 ceph-mon[27968]: mgrmap e9: x(active, since 19s) 2024-04-21T08:35:38.391 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:38 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:38.123+0000 7f1faa033200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:35:38.391 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:38 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:38.212+0000 7f1faa033200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.968+0000 7fa0e8cc4700 1 Processor -- start 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.968+0000 7fa0e8cc4700 1 -- start start 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e8cc4700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e4108c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e8cc4700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e4109220 con 0x7fa0e4108830 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e4108c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e4108c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51208/0 (socket says 172.21.15.57:51208) 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e259c700 1 -- 172.21.15.57:0/2310408633 learned_addr learned my addr 172.21.15.57:0/2310408633 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e259c700 1 -- 172.21.15.57:0/2310408633 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e4107440 con 0x7fa0e4108830 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e259c700 1 --2- 172.21.15.57:0/2310408633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e4108c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009e80 tx=0x7fa0d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613d77de9020c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:38.675 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e159a700 1 -- 172.21.15.57:0/2310408633 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d4017070 con 0x7fa0e4108830 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e159a700 1 -- 172.21.15.57:0/2310408633 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d400b8c0 con 0x7fa0e4108830 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.969+0000 7fa0e159a700 1 -- 172.21.15.57:0/2310408633 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d4004490 con 0x7fa0e4108830 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/2310408633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 msgr2=0x7fa0e4108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 --2- 172.21.15.57:0/2310408633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e4108c50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009e80 tx=0x7fa0d400b330 comp rx=0 tx=0).stop 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/2310408633 shutdown_connections 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 --2- 172.21.15.57:0/2310408633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e4108c50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/2310408633 >> 172.21.15.57:0/2310408633 conn(0x7fa0e4101fb0 msgr2=0x7fa0e41043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/2310408633 shutdown_connections 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.970+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/2310408633 wait complete. 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e8cc4700 1 Processor -- start 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e8cc4700 1 -- start start 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e8cc4700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e41923c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e8cc4700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e410b800 con 0x7fa0e4108830 2024-04-21T08:35:38.676 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e41923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e41923c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51218/0 (socket says 172.21.15.57:51218) 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e259c700 1 -- 172.21.15.57:0/538833762 learned_addr learned my addr 172.21.15.57:0/538833762 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.971+0000 7fa0e259c700 1 -- 172.21.15.57:0/538833762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e4192900 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.972+0000 7fa0e259c700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e41923c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400b7b0 tx=0x7fa0d400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.972+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d4020020 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.972+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0d4003b50 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.972+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e4190af0 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.972+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d4017040 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.972+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e4191030 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.973+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa0d4023020 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.973+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e40640a0 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.973+0000 7fa0d37fe700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa0cc039670 0x7fa0cc03bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.973+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa0d400f050 con 0x7fa0e4108830 2024-04-21T08:35:38.677 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.973+0000 7fa0e1d9b700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa0cc039670 0x7fa0cc03bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.974+0000 7fa0e1d9b700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa0cc039670 0x7fa0cc03bb30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8006fd0 tx=0x7fa0d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:36.978+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0d402a020 con 0x7fa0e4108830 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.277+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa0e4076600 con 0x7fa0e4108830 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.313+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa0d4003d70 con 0x7fa0e4108830 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.315+0000 7fa0d37fe700 1 -- 172.21.15.57:0/538833762 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fa0d4025400 con 0x7fa0e4108830 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa0cc039670 msgr2=0x7fa0cc03bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa0cc039670 0x7fa0cc03bb30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8006fd0 tx=0x7fa0d8006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 msgr2=0x7fa0e41923c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e41923c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa0d400b7b0 tx=0x7fa0d400bd30 comp rx=0 tx=0).stop 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 shutdown_connections 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fa0cc039670 0x7fa0cc03bb30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 --2- 172.21.15.57:0/538833762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0e4108830 0x7fa0e41923c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:38.678 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 >> 172.21.15.57:0/538833762 conn(0x7fa0e4101fb0 msgr2=0x7fa0e4102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:38.679 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 shutdown_connections 2024-04-21T08:35:38.679 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:37.316+0000 7fa0e8cc4700 1 -- 172.21.15.57:0/538833762 wait complete. 2024-04-21T08:35:39.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:39 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/831984828' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-21T08:35:39.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:39.486+0000 7f1faa033200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:35:39.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:39.674+0000 7f1faa033200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:40.175 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:39.851+0000 7f1faa033200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:35:40.175 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:40 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:40.174+0000 7f1faa033200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7abff700 1 Processor -- start 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7abff700 1 -- start start 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7abff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc74107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7abff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdc74107c30 con 0x7fdc74107240 2024-04-21T08:35:40.256 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7899b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc74107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7899b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc74107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51242/0 (socket says 172.21.15.57:51242) 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.248+0000 7fdc7899b700 1 -- 172.21.15.57:0/640933790 learned_addr learned my addr 172.21.15.57:0/640933790 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.249+0000 7fdc7899b700 1 -- 172.21.15.57:0/640933790 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc74108440 con 0x7fdc74107240 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.249+0000 7fdc7899b700 1 --2- 172.21.15.57:0/640933790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc74107660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800bb40 tx=0x7fdc6800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a905b058c7b9f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.249+0000 7fdc737fe700 1 -- 172.21.15.57:0/640933790 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc6800dbe0 con 0x7fdc74107240 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.249+0000 7fdc737fe700 1 -- 172.21.15.57:0/640933790 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc680044e0 con 0x7fdc74107240 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.249+0000 7fdc737fe700 1 -- 172.21.15.57:0/640933790 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc68010470 con 0x7fdc74107240 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 -- 172.21.15.57:0/640933790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 msgr2=0x7fdc74107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 --2- 172.21.15.57:0/640933790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc74107660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800bb40 tx=0x7fdc6800be50 comp rx=0 tx=0).stop 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 -- 172.21.15.57:0/640933790 shutdown_connections 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 --2- 172.21.15.57:0/640933790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc74107660 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 -- 172.21.15.57:0/640933790 >> 172.21.15.57:0/640933790 conn(0x7fdc74102820 msgr2=0x7fdc74104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 -- 172.21.15.57:0/640933790 shutdown_connections 2024-04-21T08:35:40.257 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 -- 172.21.15.57:0/640933790 wait complete. 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 Processor -- start 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.250+0000 7fdc7abff700 1 -- start start 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7abff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc7419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7abff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7407afc0 con 0x7fdc74107240 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7899b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc7419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7899b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc7419e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51246/0 (socket says 172.21.15.57:51246) 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7899b700 1 -- 172.21.15.57:0/831984828 learned_addr learned my addr 172.21.15.57:0/831984828 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7899b700 1 -- 172.21.15.57:0/831984828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7419e930 con 0x7fdc74107240 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7899b700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc7419e3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc68003c30 tx=0x7fdc68003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc68010720 con 0x7fdc74107240 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7419ebc0 con 0x7fdc74107240 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc6801d070 con 0x7fdc74107240 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.251+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc68021aa0 con 0x7fdc74107240 2024-04-21T08:35:40.258 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.252+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc741a18b0 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.252+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fdc6801f030 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.253+0000 7fdc71ffb700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 0x7fdc6403bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.253+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7419f0a0 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.253+0000 7fdc73fff700 1 -- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 msgr2=0x7fdc6403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.253+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdc6804e8a0 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.253+0000 7fdc73fff700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 0x7fdc6403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.258+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc7419f0a0 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.454+0000 7fdc73fff700 1 -- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 msgr2=0x7fdc6403bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.454+0000 7fdc73fff700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 0x7fdc6403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.526+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fdc741a1f30 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.526+0000 7fdc71ffb700 1 -- 172.21.15.57:0/831984828 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fdc68015070 con 0x7fdc74107240 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 msgr2=0x7fdc6403bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 0x7fdc6403bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.259 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 msgr2=0x7fdc7419e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc7419e3f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdc68003c30 tx=0x7fdc68003d10 comp rx=0 tx=0).stop 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 shutdown_connections 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7fdc640396c0 0x7fdc6403bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 --2- 172.21.15.57:0/831984828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc74107240 0x7fdc7419e3f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.528+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 >> 172.21.15.57:0/831984828 conn(0x7fdc74102820 msgr2=0x7fdc74190600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.529+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 shutdown_connections 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:39.529+0000 7fdc7abff700 1 -- 172.21.15.57:0/831984828 wait complete. 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for the mgr to restart... 2024-04-21T08:35:40.260 INFO:teuthology.orchestra.run.smithi057.stdout:Waiting for mgr epoch 9... 2024-04-21T08:35:40.495 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:40 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:40.280+0000 7f1faa033200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:35:40.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:40 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:40.495+0000 7f1faa033200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:35:40.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:40 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:40.699+0000 7f1faa033200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:35:41.588 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:41.325+0000 7f1faa033200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:35:42.077 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:41.764+0000 7f1faa033200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:35:42.078 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:41.864+0000 7f1faa033200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:35:42.078 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:42.077+0000 7f1faa033200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:35:42.993 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:42.641+0000 7f1faa033200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:35:42.993 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:42.739+0000 7f1faa033200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:35:43.339 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:42.992+0000 7f1faa033200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:35:43.339 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:43 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:43.131+0000 7f1faa033200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:35:43.339 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:43 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:43.230+0000 7f1faa033200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:35:43.661 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:43 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:43.350+0000 7f1faa033200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:35:43.662 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:43 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:43.661+0000 7f1faa033200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:35:44.567 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:44.289+0000 7f1faa033200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:35:44.568 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:35:44.514+0000 7f1faa033200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: Active manager daemon x restarted 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: Activating manager daemon x 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: osdmap e3: 0 total, 0 up, 0 in 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: mgrmap e10: x(active, starting, since 0.0056702s) 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: Manager daemon x is now available 2024-04-21T08:35:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:35:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:35:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:45 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:45] ENGINE Bus STARTING 2024-04-21T08:35:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:45 smithi057 ceph-mon[27968]: mgrmap e11: x(active, since 1.00651s) 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout { 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout } 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.719+0000 7f6633968700 1 Processor -- start 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.719+0000 7f6633968700 1 -- start start 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6633968700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6633968700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f662c107c30 con 0x7f662c107240 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6631704700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:46.098 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6631704700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51260/0 (socket says 172.21.15.57:51260) 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6631704700 1 -- 172.21.15.57:0/3651334809 learned_addr learned my addr 172.21.15.57:0/3651334809 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6631704700 1 -- 172.21.15.57:0/3651334809 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c108440 con 0x7f662c107240 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6631704700 1 --2- 172.21.15.57:0/3651334809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c107660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f661c0098a0 tx=0x7f661c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c41fb5d8eb538d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6623fff700 1 -- 172.21.15.57:0/3651334809 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661c004160 con 0x7f662c107240 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6623fff700 1 -- 172.21.15.57:0/3651334809 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f661c004300 con 0x7f662c107240 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.720+0000 7f6623fff700 1 -- 172.21.15.57:0/3651334809 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661c011a20 con 0x7f662c107240 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 -- 172.21.15.57:0/3651334809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 msgr2=0x7f662c107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 --2- 172.21.15.57:0/3651334809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c107660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f661c0098a0 tx=0x7f661c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 -- 172.21.15.57:0/3651334809 shutdown_connections 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 --2- 172.21.15.57:0/3651334809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c107660 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 -- 172.21.15.57:0/3651334809 >> 172.21.15.57:0/3651334809 conn(0x7f662c102820 msgr2=0x7f662c104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 -- 172.21.15.57:0/3651334809 shutdown_connections 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.721+0000 7f6633968700 1 -- 172.21.15.57:0/3651334809 wait complete. 2024-04-21T08:35:46.099 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.722+0000 7f6633968700 1 Processor -- start 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.722+0000 7f6633968700 1 -- start start 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.722+0000 7f6633968700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.722+0000 7f6633968700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f662c07afc0 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6631704700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6631704700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51274/0 (socket says 172.21.15.57:51274) 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6631704700 1 -- 172.21.15.57:0/3236575372 learned_addr learned my addr 172.21.15.57:0/3236575372 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6631704700 1 -- 172.21.15.57:0/3236575372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c19e930 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6631704700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c19e3f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f661c0116e0 tx=0x7f661c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661c0039e0 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f661c01b8f0 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c19ebc0 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c1a18b0 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.723+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f661c023760 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.724+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f661c023a00 con 0x7f662c107240 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.724+0000 7f66227fc700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:46.100 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.724+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 --> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f661803c290 con 0x7f66180396c0 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.724+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f661c01bd50 con 0x7f662c107240 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.724+0000 7f6630f03700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 msgr2=0x7f661803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.724+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.925+0000 7f6630f03700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 msgr2=0x7f661803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:40.925+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:41.326+0000 7f6630f03700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 msgr2=0x7f661803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:41.326+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:42.127+0000 7f6630f03700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 msgr2=0x7f661803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:42.127+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:43.728+0000 7f6630f03700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 msgr2=0x7f661803bb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/2928520021 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:43.728+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:44.520+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f661c02c910 con 0x7f662c107240 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:44.520+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 msgr2=0x7f661803bb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:35:46.101 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:44.520+0000 7f66227fc700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.519+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f661c04fac0 con 0x7f662c107240 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.520+0000 7f66227fc700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f661803cfb0 0x7f661803f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.520+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f661c047f00 con 0x7f661803cfb0 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.520+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f661803cfb0 0x7f661803f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.521+0000 7f6630f03700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f661803cfb0 0x7f661803f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6628003a10 tx=0x7f6628009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.521+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f661c047f00 con 0x7f661803cfb0 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.528+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f662c069b50 con 0x7f661803cfb0 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.528+0000 7f66227fc700 1 -- 172.21.15.57:0/3236575372 <== mgr.14152 v2:172.21.15.57:6800/3002879345 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f662c069b50 con 0x7f661803cfb0 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f661803cfb0 msgr2=0x7f661803f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f661803cfb0 0x7f661803f3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6628003a10 tx=0x7f6628009270 comp rx=0 tx=0).stop 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 msgr2=0x7f662c19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:46.102 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c19e3f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f661c0116e0 tx=0x7f661c011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 shutdown_connections 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f661803cfb0 0x7f661803f3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f662c107240 0x7f662c19e3f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 --2- 172.21.15.57:0/3236575372 >> [v2:172.21.15.57:6800/2928520021,v1:172.21.15.57:6801/2928520021] conn(0x7f66180396c0 0x7f661803bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 >> 172.21.15.57:0/3236575372 conn(0x7f662c102820 msgr2=0x7f662c190600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 shutdown_connections 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:45.529+0000 7f6633968700 1 -- 172.21.15.57:0/3236575372 wait complete. 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:mgr epoch 9 is available 2024-04-21T08:35:46.103 INFO:teuthology.orchestra.run.smithi057.stdout:Generating a dashboard self-signed certificate... 2024-04-21T08:35:46.775 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:46 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:45] ENGINE Serving on https://172.21.15.57:7150 2024-04-21T08:35:46.775 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:46 smithi057 ceph-mon[27968]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-21T08:35:46.775 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:46 smithi057 ceph-mon[27968]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-21T08:35:46.775 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:46 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:45] ENGINE Serving on http://172.21.15.57:8765 2024-04-21T08:35:46.776 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:46 smithi057 ceph-mon[27968]: [21/Apr/2024:08:35:45] ENGINE Bus STARTED 2024-04-21T08:35:47.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:47 smithi057 ceph-mon[27968]: mgrmap e12: x(active, since 2s) 2024-04-21T08:35:47.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:47.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:47 smithi057 ceph-mon[27968]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:47.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:47.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:47.754 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-21T08:35:47.754 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.881+0000 7fec7b474700 1 Processor -- start 2024-04-21T08:35:47.754 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec7b474700 1 -- start start 2024-04-21T08:35:47.754 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec7b474700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec7410b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:47.754 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec7b474700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fec7407bea0 con 0x7fec74108dc0 2024-04-21T08:35:47.754 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec79210700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec7410b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec79210700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec7410b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45442/0 (socket says 172.21.15.57:45442) 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec79210700 1 -- 172.21.15.57:0/8429533 learned_addr learned my addr 172.21.15.57:0/8429533 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.882+0000 7fec79210700 1 -- 172.21.15.57:0/8429533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7410b6f0 con 0x7fec74108dc0 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec79210700 1 --2- 172.21.15.57:0/8429533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec7410b1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec700098a0 tx=0x7fec70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb489a24c9e04afe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec67fff700 1 -- 172.21.15.57:0/8429533 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec70004160 con 0x7fec74108dc0 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec67fff700 1 -- 172.21.15.57:0/8429533 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec70004300 con 0x7fec74108dc0 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec67fff700 1 -- 172.21.15.57:0/8429533 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec70011a20 con 0x7fec74108dc0 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec7b474700 1 -- 172.21.15.57:0/8429533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 msgr2=0x7fec7410b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec7b474700 1 --2- 172.21.15.57:0/8429533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec7410b1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec700098a0 tx=0x7fec70009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec7b474700 1 -- 172.21.15.57:0/8429533 shutdown_connections 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec7b474700 1 --2- 172.21.15.57:0/8429533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec7410b1b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.883+0000 7fec7b474700 1 -- 172.21.15.57:0/8429533 >> 172.21.15.57:0/8429533 conn(0x7fec74102890 msgr2=0x7fec74104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.884+0000 7fec7b474700 1 -- 172.21.15.57:0/8429533 shutdown_connections 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.884+0000 7fec7b474700 1 -- 172.21.15.57:0/8429533 wait complete. 2024-04-21T08:35:47.755 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.884+0000 7fec7b474700 1 Processor -- start 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.884+0000 7fec7b474700 1 -- start start 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec7b474700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec74108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec7b474700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fec7410c700 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec79210700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec74108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec79210700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec74108640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45452/0 (socket says 172.21.15.57:45452) 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec79210700 1 -- 172.21.15.57:0/901823218 learned_addr learned my addr 172.21.15.57:0/901823218 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec79210700 1 -- 172.21.15.57:0/901823218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec74108b80 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec79210700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec74108640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec70000c00 tx=0x7fec70011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec70003aa0 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec74106d70 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec7001b8f0 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.885+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec741072b0 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.886+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec70023870 con 0x7fec74108dc0 2024-04-21T08:35:47.756 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.887+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fec70023b20 con 0x7fec74108dc0 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.887+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec740640a0 con 0x7fec74108dc0 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.887+0000 7fec667fc700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec60039720 0x7fec6003bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.887+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fec7004df30 con 0x7fec74108dc0 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.887+0000 7fec78a0f700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec60039720 0x7fec6003bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.887+0000 7fec78a0f700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec60039720 0x7fec6003bbe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec68006fd0 tx=0x7fec68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:46.892+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec7001f030 con 0x7fec74108dc0 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.104+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fec74107c00 con 0x7fec60039720 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.160+0000 7fec667fc700 1 -- 172.21.15.57:0/901823218 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fec74107c00 con 0x7fec60039720 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.162+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec60039720 msgr2=0x7fec6003bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.162+0000 7fec7b474700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec60039720 0x7fec6003bbe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec68006fd0 tx=0x7fec68006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.162+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 msgr2=0x7fec74108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.162+0000 7fec7b474700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec74108640 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec70000c00 tx=0x7fec70011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.163+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 shutdown_connections 2024-04-21T08:35:47.757 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.163+0000 7fec7b474700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec60039720 0x7fec6003bbe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:47.758 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.163+0000 7fec7b474700 1 --2- 172.21.15.57:0/901823218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec74108dc0 0x7fec74108640 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:47.758 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.163+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 >> 172.21.15.57:0/901823218 conn(0x7fec74102890 msgr2=0x7fec74103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:47.758 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.163+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 shutdown_connections 2024-04-21T08:35:47.758 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:47.163+0000 7fec7b474700 1 -- 172.21.15.57:0/901823218 wait complete. 2024-04-21T08:35:47.758 INFO:teuthology.orchestra.run.smithi057.stdout:Creating initial admin user... 2024-04-21T08:35:49.234 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ExEUvr2XPTMz3sw/h1R4q.P/91Scs0y7NXu.pA/Xf/bJjHXogiC0m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713688548, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.179+0000 7f4c81bee700 1 Processor -- start 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.179+0000 7f4c81bee700 1 -- start start 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.179+0000 7f4c81bee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c109d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.179+0000 7f4c81bee700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c10a320 con 0x7f4c7c109930 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.179+0000 7f4c7b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c109d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c109d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45454/0 (socket says 172.21.15.57:45454) 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7b7fe700 1 -- 172.21.15.57:0/3997011049 learned_addr learned my addr 172.21.15.57:0/3997011049 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7b7fe700 1 -- 172.21.15.57:0/3997011049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c10a4a0 con 0x7f4c7c109930 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7b7fe700 1 --2- 172.21.15.57:0/3997011049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c109d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c640098a0 tx=0x7f4c64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d6bb38b1e1a820 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7a7fc700 1 -- 172.21.15.57:0/3997011049 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c64004160 con 0x7f4c7c109930 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7a7fc700 1 -- 172.21.15.57:0/3997011049 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c64004300 con 0x7f4c7c109930 2024-04-21T08:35:49.235 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.180+0000 7f4c7a7fc700 1 -- 172.21.15.57:0/3997011049 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c64011a20 con 0x7f4c7c109930 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 -- 172.21.15.57:0/3997011049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 msgr2=0x7f4c7c109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 --2- 172.21.15.57:0/3997011049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c109d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c640098a0 tx=0x7f4c64009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 -- 172.21.15.57:0/3997011049 shutdown_connections 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 --2- 172.21.15.57:0/3997011049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c109d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 -- 172.21.15.57:0/3997011049 >> 172.21.15.57:0/3997011049 conn(0x7f4c7c101fb0 msgr2=0x7f4c7c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 -- 172.21.15.57:0/3997011049 shutdown_connections 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 -- 172.21.15.57:0/3997011049 wait complete. 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.181+0000 7f4c81bee700 1 Processor -- start 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c81bee700 1 -- start start 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c81bee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c108f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c81bee700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c1071f0 con 0x7f4c7c109930 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c7b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c7b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45462/0 (socket says 172.21.15.57:45462) 2024-04-21T08:35:49.236 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c7b7fe700 1 -- 172.21.15.57:0/445636124 learned_addr learned my addr 172.21.15.57:0/445636124 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c7b7fe700 1 -- 172.21.15.57:0/445636124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c109490 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c7b7fe700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c108f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009520 tx=0x7f4c64011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c640039e0 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c6401b8f0 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c107680 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c64019770 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.182+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c107bc0 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.183+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f4c64019a70 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.183+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c10c900 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.184+0000 7f4c78ff9700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4c680396c0 0x7f4c6803bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.184+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4c6404e110 con 0x7f4c7c109930 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.184+0000 7f4c7affd700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4c680396c0 0x7f4c6803bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.184+0000 7f4c7affd700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4c680396c0 0x7f4c6803bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006fd0 tx=0x7f4c6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:49.237 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.189+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c6401f030 con 0x7f4c7c109930 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.430+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f4c7c0640a0 con 0x7f4c680396c0 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.681+0000 7f4c78ff9700 1 -- 172.21.15.57:0/445636124 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f4c7c0640a0 con 0x7f4c680396c0 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4c680396c0 msgr2=0x7f4c6803bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4c680396c0 0x7f4c6803bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c006fd0 tx=0x7f4c6c006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 msgr2=0x7f4c7c108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c108f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c64009520 tx=0x7f4c64011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 shutdown_connections 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4c680396c0 0x7f4c6803bb80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 --2- 172.21.15.57:0/445636124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c7c109930 0x7f4c7c108f50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.683+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 >> 172.21.15.57:0/445636124 conn(0x7f4c7c101fb0 msgr2=0x7f4c7c102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.684+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 shutdown_connections 2024-04-21T08:35:49.238 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:48.684+0000 7f4c81bee700 1 -- 172.21.15.57:0/445636124 wait complete. 2024-04-21T08:35:49.239 INFO:teuthology.orchestra.run.smithi057.stdout:Fetching dashboard port number... 2024-04-21T08:35:49.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:49.480 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:50 smithi057 ceph-mon[27968]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:35:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/489967738' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-21T08:35:50.844 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stdout 8443 2024-04-21T08:35:50.844 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.703+0000 7f5d76a52700 1 Processor -- start 2024-04-21T08:35:50.844 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.703+0000 7f5d76a52700 1 -- start start 2024-04-21T08:35:50.844 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.704+0000 7f5d76a52700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 0x7f5d70106a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:50.844 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.704+0000 7f5d76a52700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70106ff0 con 0x7f5d70106600 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.704+0000 7f5d6ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 0x7f5d70106a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.704+0000 7f5d6ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 0x7f5d70106a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45468/0 (socket says 172.21.15.57:45468) 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.704+0000 7f5d6ffff700 1 -- 172.21.15.57:0/4100915517 learned_addr learned my addr 172.21.15.57:0/4100915517 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.704+0000 7f5d6ffff700 1 -- 172.21.15.57:0/4100915517 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d70107370 con 0x7f5d70106600 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.705+0000 7f5d6ffff700 1 --2- 172.21.15.57:0/4100915517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 0x7f5d70106a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009b00 tx=0x7f5d60009e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74be038f7ac135f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.705+0000 7f5d6effd700 1 -- 172.21.15.57:0/4100915517 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d60004160 con 0x7f5d70106600 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.705+0000 7f5d6effd700 1 -- 172.21.15.57:0/4100915517 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d60004300 con 0x7f5d70106600 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.705+0000 7f5d6effd700 1 -- 172.21.15.57:0/4100915517 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d6000ba80 con 0x7f5d70106600 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 -- 172.21.15.57:0/4100915517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 msgr2=0x7f5d70106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 --2- 172.21.15.57:0/4100915517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 0x7f5d70106a20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009b00 tx=0x7f5d60009e10 comp rx=0 tx=0).stop 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 -- 172.21.15.57:0/4100915517 shutdown_connections 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 --2- 172.21.15.57:0/4100915517 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d70106600 0x7f5d70106a20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:50.845 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 -- 172.21.15.57:0/4100915517 >> 172.21.15.57:0/4100915517 conn(0x7f5d70101fa0 msgr2=0x7f5d701043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 -- 172.21.15.57:0/4100915517 shutdown_connections 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.706+0000 7f5d76a52700 1 -- 172.21.15.57:0/4100915517 wait complete. 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d76a52700 1 Processor -- start 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d76a52700 1 -- start start 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d76a52700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 0x7f5d7019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d76a52700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5d70108730 con 0x7f5d7019e680 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d6ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 0x7f5d7019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d6ffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 0x7f5d7019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45470/0 (socket says 172.21.15.57:45470) 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.707+0000 7f5d6ffff700 1 -- 172.21.15.57:0/489967738 learned_addr learned my addr 172.21.15.57:0/489967738 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.708+0000 7f5d6ffff700 1 -- 172.21.15.57:0/489967738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7019efe0 con 0x7f5d7019e680 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.708+0000 7f5d6ffff700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 0x7f5d7019eaa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000b810 tx=0x7f5d6000b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.708+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d600037d0 con 0x7f5d7019e680 2024-04-21T08:35:50.846 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.708+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7019f270 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.708+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d60003970 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.708+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d701a1d00 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.709+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d6001b440 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.710+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f5d6001b5e0 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.710+0000 7f5d6d7fa700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5d580396d0 0x7f5d5803bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.710+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d6004df80 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.710+0000 7f5d6f7fe700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5d580396d0 0x7f5d5803bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.710+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d700640a0 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.711+0000 7f5d6f7fe700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5d580396d0 0x7f5d5803bb90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006fd0 tx=0x7f5d64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.716+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d60003b20 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.939+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f5d70197a90 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.939+0000 7f5d6d7fa700 1 -- 172.21.15.57:0/489967738 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f5d70197a90 con 0x7f5d7019e680 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5d580396d0 msgr2=0x7f5d5803bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:50.847 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5d580396d0 0x7f5d5803bb90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006fd0 tx=0x7f5d64006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 msgr2=0x7f5d7019eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 0x7f5d7019eaa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000b810 tx=0x7f5d6000b9a0 comp rx=0 tx=0).stop 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 shutdown_connections 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5d580396d0 0x7f5d5803bb90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 --2- 172.21.15.57:0/489967738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7019e680 0x7f5d7019eaa0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 >> 172.21.15.57:0/489967738 conn(0x7f5d70101fa0 msgr2=0x7f5d70194b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 shutdown_connections 2024-04-21T08:35:50.848 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:49.941+0000 7f5d76a52700 1 -- 172.21.15.57:0/489967738 wait complete. 2024-04-21T08:35:50.852 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-21T08:35:50.852 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout disabled 2024-04-21T08:35:50.860 INFO:teuthology.orchestra.run.smithi057.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-21T08:35:50.860 INFO:teuthology.orchestra.run.smithi057.stdout:systemctl: stdout inactive 2024-04-21T08:35:50.860 INFO:teuthology.orchestra.run.smithi057.stdout:firewalld.service is not enabled 2024-04-21T08:35:50.860 INFO:teuthology.orchestra.run.smithi057.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-21T08:35:50.862 INFO:teuthology.orchestra.run.smithi057.stdout:Ceph Dashboard is now available at: 2024-04-21T08:35:50.862 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:50.862 INFO:teuthology.orchestra.run.smithi057.stdout: URL: https://smithi057:8443/ 2024-04-21T08:35:50.862 INFO:teuthology.orchestra.run.smithi057.stdout: User: admin 2024-04-21T08:35:50.863 INFO:teuthology.orchestra.run.smithi057.stdout: Password: 731uo8p7gt 2024-04-21T08:35:50.863 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:50.863 INFO:teuthology.orchestra.run.smithi057.stdout:Saving cluster configuration to /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config directory 2024-04-21T08:35:50.863 INFO:teuthology.orchestra.run.smithi057.stdout:Enabling autotune for osd_memory_target 2024-04-21T08:35:52.302 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:52 smithi057 ceph-mon[27968]: mgrmap e13: x(active, since 6s) 2024-04-21T08:35:52.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f8651d8c700 1 Processor -- start 2024-04-21T08:35:52.310 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f8651d8c700 1 -- start start 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f8651d8c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c107040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f8651d8c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f864c07be60 con 0x7f864c106c20 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c107040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c107040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45478/0 (socket says 172.21.15.57:45478) 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f864b7fe700 1 -- 172.21.15.57:0/224502114 learned_addr learned my addr 172.21.15.57:0/224502114 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f864b7fe700 1 -- 172.21.15.57:0/224502114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c107580 con 0x7f864c106c20 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.291+0000 7f864b7fe700 1 --2- 172.21.15.57:0/224502114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c107040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f863c0098a0 tx=0x7f863c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104606690985e7b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f864a7fc700 1 -- 172.21.15.57:0/224502114 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f863c004160 con 0x7f864c106c20 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f864a7fc700 1 -- 172.21.15.57:0/224502114 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f863c004300 con 0x7f864c106c20 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f864a7fc700 1 -- 172.21.15.57:0/224502114 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f863c011a20 con 0x7f864c106c20 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 -- 172.21.15.57:0/224502114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 msgr2=0x7f864c107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:52.311 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 --2- 172.21.15.57:0/224502114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c107040 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f863c0098a0 tx=0x7f863c009bb0 comp rx=0 tx=0).stop 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 -- 172.21.15.57:0/224502114 shutdown_connections 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 --2- 172.21.15.57:0/224502114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c107040 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 -- 172.21.15.57:0/224502114 >> 172.21.15.57:0/224502114 conn(0x7f864c102860 msgr2=0x7f864c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 -- 172.21.15.57:0/224502114 shutdown_connections 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.292+0000 7f8651d8c700 1 -- 172.21.15.57:0/224502114 wait complete. 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f8651d8c700 1 Processor -- start 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f8651d8c700 1 -- start start 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f8651d8c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c19e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f8651d8c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1085f0 con 0x7f864c106c20 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c19e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f864b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c19e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45484/0 (socket says 172.21.15.57:45484) 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.293+0000 7f864b7fe700 1 -- 172.21.15.57:0/916612081 learned_addr learned my addr 172.21.15.57:0/916612081 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f864b7fe700 1 -- 172.21.15.57:0/916612081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c19e8d0 con 0x7f864c106c20 2024-04-21T08:35:52.312 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f864b7fe700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c19e390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f863c000c00 tx=0x7f863c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f863c003aa0 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f863c01b8f0 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c19eb60 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f863c023870 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.294+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c1a1850 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.295+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f863c01b440 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.295+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864c0640a0 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.295+0000 7f8648ff9700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f86340397c0 0x7f863403bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.295+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f863c04e020 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.295+0000 7f864affd700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f86340397c0 0x7f863403bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.296+0000 7f864affd700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f86340397c0 0x7f863403bc80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8640006fd0 tx=0x7f8640006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.300+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f863c01f030 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.510+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f864c02e260 con 0x7f864c106c20 2024-04-21T08:35:52.313 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.510+0000 7f8648ff9700 1 -- 172.21.15.57:0/916612081 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f863c0292a0 con 0x7f864c106c20 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f86340397c0 msgr2=0x7f863403bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f86340397c0 0x7f863403bc80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8640006fd0 tx=0x7f8640006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 msgr2=0x7f864c19e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c19e390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f863c000c00 tx=0x7f863c011aa0 comp rx=0 tx=0).stop 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 shutdown_connections 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f86340397c0 0x7f863403bc80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 --2- 172.21.15.57:0/916612081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f864c106c20 0x7f864c19e390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 >> 172.21.15.57:0/916612081 conn(0x7f864c102860 msgr2=0x7f864c194fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 shutdown_connections 2024-04-21T08:35:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:51.512+0000 7f8651d8c700 1 -- 172.21.15.57:0/916612081 wait complete. 2024-04-21T08:35:53.659 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.731+0000 7f469d4b8700 1 Processor -- start 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.731+0000 7f469d4b8700 1 -- start start 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.731+0000 7f469d4b8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f4698107660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.731+0000 7f469d4b8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4698107c30 con 0x7f4698107240 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4696ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f4698107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4696ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f4698107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40700/0 (socket says 172.21.15.57:40700) 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4696ffd700 1 -- 172.21.15.57:0/3354028212 learned_addr learned my addr 172.21.15.57:0/3354028212 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4696ffd700 1 -- 172.21.15.57:0/3354028212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4698108440 con 0x7f4698107240 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4696ffd700 1 --2- 172.21.15.57:0/3354028212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f4698107660 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4690009e80 tx=0x7f469000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff76ca31a20259 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4695ffb700 1 -- 172.21.15.57:0/3354028212 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4690017070 con 0x7f4698107240 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4695ffb700 1 -- 172.21.15.57:0/3354028212 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f469000b8c0 con 0x7f4698107240 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.732+0000 7f4695ffb700 1 -- 172.21.15.57:0/3354028212 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4690004490 con 0x7f4698107240 2024-04-21T08:35:53.660 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 -- 172.21.15.57:0/3354028212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 msgr2=0x7f4698107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 --2- 172.21.15.57:0/3354028212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f4698107660 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4690009e80 tx=0x7f469000b330 comp rx=0 tx=0).stop 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 -- 172.21.15.57:0/3354028212 shutdown_connections 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 --2- 172.21.15.57:0/3354028212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f4698107660 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 -- 172.21.15.57:0/3354028212 >> 172.21.15.57:0/3354028212 conn(0x7f4698102820 msgr2=0x7f4698104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 -- 172.21.15.57:0/3354028212 shutdown_connections 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.733+0000 7f469d4b8700 1 -- 172.21.15.57:0/3354028212 wait complete. 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f469d4b8700 1 Processor -- start 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f469d4b8700 1 -- start start 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f469d4b8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f469819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f469d4b8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f469810b7f0 con 0x7f4698107240 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f4696ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f469819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.661 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f4696ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f469819c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40716/0 (socket says 172.21.15.57:40716) 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f4696ffd700 1 -- 172.21.15.57:0/3103997299 learned_addr learned my addr 172.21.15.57:0/3103997299 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.734+0000 7f4696ffd700 1 -- 172.21.15.57:0/3103997299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469819c7a0 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.735+0000 7f4696ffd700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f469819c260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f469000b7b0 tx=0x7f469000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.735+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4690020020 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.735+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4690003b50 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.735+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469819ca30 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.735+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4690017040 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.735+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469819f720 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.736+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4690023020 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.736+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46980640a0 con 0x7f4698107240 2024-04-21T08:35:53.662 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.736+0000 7f468ffff700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46840397b0 0x7f468403bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.736+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f469000f050 con 0x7f4698107240 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.736+0000 7f46967fc700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46840397b0 0x7f468403bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.736+0000 7f46967fc700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46840397b0 0x7f468403bc70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4680006fd0 tx=0x7f4680006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:52.741+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f469002a020 con 0x7f4698107240 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.066+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f469802e2c0 con 0x7f4698107240 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.073+0000 7f468ffff700 1 -- 172.21.15.57:0/3103997299 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f4690007570 con 0x7f4698107240 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46840397b0 msgr2=0x7f468403bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46840397b0 0x7f468403bc70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4680006fd0 tx=0x7f4680006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 msgr2=0x7f469819c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f469819c260 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f469000b7b0 tx=0x7f469000bd30 comp rx=0 tx=0).stop 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 shutdown_connections 2024-04-21T08:35:53.663 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46840397b0 0x7f468403bc70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 --2- 172.21.15.57:0/3103997299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4698107240 0x7f469819c260 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 >> 172.21.15.57:0/3103997299 conn(0x7f4698102820 msgr2=0x7f4698103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 shutdown_connections 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr 2024-04-21T08:35:53.075+0000 7f469d4b8700 1 -- 172.21.15.57:0/3103997299 wait complete. 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout:Or, if you are only running a single cluster on this host: 2024-04-21T08:35:53.664 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: ceph telemetry on 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout:For more information see: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:35:53.665 INFO:teuthology.orchestra.run.smithi057.stdout:Bootstrap complete. 2024-04-21T08:35:53.702 INFO:tasks.cephadm:Fetching config... 2024-04-21T08:35:53.703 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:35:53.703 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-21T08:35:53.721 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-21T08:35:53.721 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:35:53.721 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-21T08:35:53.777 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-21T08:35:53.777 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:35:53.777 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-21T08:35:53.843 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-21T08:35:53.843 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:35:53.843 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-21T08:35:53.899 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-21T08:35:53.899 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:35:53.972 INFO:teuthology.orchestra.run.smithi057.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:53.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:35:54.026 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:54.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-21T08:35:54.077 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC86vDR/i97vEmLJzCOpWnOb1q8x3kABmkyRjQjUV98ni3JlTjh4lpfF0OBzPb3NrOFAVKz0VRpzx0xIiQzs07tekwHwzyVPaMyggFFKKpQH9Dj0149fFpzO09LjtIEMpmc/AqvKzz3aI8JLP2PBWAcmP0BDabBSwm4dx3LDOS8t7zmY9UHqcYu9IIE2mmZVyTf++tOqz6yx4Qmbg6wVEYadKTmk7OlCUsm0xPJJ2FSxqOYhzvEe+/w0IP48WTWUd89uUiaPXpepZcaU1Ol1q+B+gXzeGnrtnn04g7RlyQDTuE7e8PL57WMGsyWdUaA+JJPJqBOXjLtJQXBBMQBMYVnUMwCS4nZTklZHRKCfT61ruobB3WC45RLhyMGsmdeAkL8OvZtbTXqxZhavMH8MmXBImTd/Kun6mCrCX9EcPqG0gJMInhSetC07ZwJwrGp37G1DMpFdE69D8K12DpCvQibsDHd6BModg6f7Z1uyEW3OnTgkqgSkTOIoBFbbeQtTDk= ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:35:54.089 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-21T08:35:54.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:54 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3103997299' entity='client.admin' 2024-04-21T08:35:54.340 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 -- 172.21.15.57:0/3534613030 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 msgr2=0x7f391c102900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 --2- 172.21.15.57:0/3534613030 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c102900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f390c009910 tx=0x7f390c009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 -- 172.21.15.57:0/3534613030 shutdown_connections 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 --2- 172.21.15.57:0/3534613030 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c102900 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 -- 172.21.15.57:0/3534613030 >> 172.21.15.57:0/3534613030 conn(0x7f391c0fda40 msgr2=0x7f391c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 -- 172.21.15.57:0/3534613030 shutdown_connections 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 -- 172.21.15.57:0/3534613030 wait complete. 2024-04-21T08:35:56.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 Processor -- start 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.687+0000 7f39226e9700 1 -- start start 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f39226e9700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c0766b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f39226e9700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f391c108420 con 0x7f391c1024e0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f391bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c0766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f391bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c0766b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40730/0 (socket says 172.21.15.57:40730) 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f391bfff700 1 -- 172.21.15.57:0/2268288414 learned_addr learned my addr 172.21.15.57:0/2268288414 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f391bfff700 1 -- 172.21.15.57:0/2268288414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c076bf0 con 0x7f391c1024e0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.688+0000 7f391bfff700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c0766b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f390c005950 tx=0x7f390c004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.689+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f390c026070 con 0x7f391c1024e0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.689+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f390c00bc00 con 0x7f391c1024e0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.689+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c074e30 con 0x7f391c1024e0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.689+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f390c003a10 con 0x7f391c1024e0 2024-04-21T08:35:56.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.689+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c075310 con 0x7f391c1024e0 2024-04-21T08:35:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.689+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f390c029070 con 0x7f391c1024e0 2024-04-21T08:35:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.690+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f391c108ba0 con 0x7f391c1024e0 2024-04-21T08:35:56.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.690+0000 7f39197fa700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3904039650 0x7f390403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:56.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.690+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f390c01e070 con 0x7f391c1024e0 2024-04-21T08:35:56.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.690+0000 7f391b7fe700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3904039650 0x7f390403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:56.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.690+0000 7f391b7fe700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3904039650 0x7f390403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3910006fd0 tx=0x7f3910006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:56.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.695+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f390c013070 con 0x7f391c1024e0 2024-04-21T08:35:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.904+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f391c190e80 con 0x7f391c1024e0 2024-04-21T08:35:56.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.907+0000 7f39197fa700 1 -- 172.21.15.57:0/2268288414 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f390c030020 con 0x7f391c1024e0 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.909+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3904039650 msgr2=0x7f390403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.909+0000 7f39226e9700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3904039650 0x7f390403bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3910006fd0 tx=0x7f3910006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.909+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 msgr2=0x7f391c0766b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.909+0000 7f39226e9700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c0766b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f390c005950 tx=0x7f390c004fa0 comp rx=0 tx=0).stop 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.909+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 shutdown_connections 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.910+0000 7f39226e9700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3904039650 0x7f390403bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.910+0000 7f39226e9700 1 --2- 172.21.15.57:0/2268288414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f391c1024e0 0x7f391c0766b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.910+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 >> 172.21.15.57:0/2268288414 conn(0x7f391c0fda40 msgr2=0x7f391c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:56.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.910+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 shutdown_connections 2024-04-21T08:35:56.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:56.910+0000 7f39226e9700 1 -- 172.21.15.57:0/2268288414 wait complete. 2024-04-21T08:35:57.560 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-21T08:35:57.561 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-21T08:35:58.137 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:35:58.260 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:57 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2268288414' entity='client.admin' 2024-04-21T08:35:59.308 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:59.309 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:59.309 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi057", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:35:59.309 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:59.309 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:35:59.309 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:35:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:35:59.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 -- 172.21.15.57:0/1135920996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 msgr2=0x7f7034103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:59.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 --2- 172.21.15.57:0/1135920996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034103fd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7024009910 tx=0x7f7024009c20 comp rx=0 tx=0).stop 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 -- 172.21.15.57:0/1135920996 shutdown_connections 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 --2- 172.21.15.57:0/1135920996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034103fd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 -- 172.21.15.57:0/1135920996 >> 172.21.15.57:0/1135920996 conn(0x7f70340fda90 msgr2=0x7f70340ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 -- 172.21.15.57:0/1135920996 shutdown_connections 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.620+0000 7f7038ab7700 1 -- 172.21.15.57:0/1135920996 wait complete. 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f7038ab7700 1 Processor -- start 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f7038ab7700 1 -- start start 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f7038ab7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034076700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f7038ab7700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7034108520 con 0x7f7034103bb0 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f703259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034076700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f703259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034076700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40744/0 (socket says 172.21.15.57:40744) 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.621+0000 7f703259c700 1 -- 172.21.15.57:0/2496438130 learned_addr learned my addr 172.21.15.57:0/2496438130 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:35:59.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f703259c700 1 -- 172.21.15.57:0/2496438130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7034076c40 con 0x7f7034103bb0 2024-04-21T08:35:59.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f703259c700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034076700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7024005800 tx=0x7f7024009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:59.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7024027070 con 0x7f7034103bb0 2024-04-21T08:35:59.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f702400be10 con 0x7f7034103bb0 2024-04-21T08:35:59.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7034074e30 con 0x7f7034103bb0 2024-04-21T08:35:59.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f702400f460 con 0x7f7034103bb0 2024-04-21T08:35:59.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.622+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7034075370 con 0x7f7034103bb0 2024-04-21T08:35:59.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.623+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f702402a070 con 0x7f7034103bb0 2024-04-21T08:35:59.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.623+0000 7f70237fe700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f701c0396b0 0x7f701c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:35:59.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.623+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f702401e030 con 0x7f7034103bb0 2024-04-21T08:35:59.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.623+0000 7f7031d9b700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f701c0396b0 0x7f701c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:35:59.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.623+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7034108ca0 con 0x7f7034103bb0 2024-04-21T08:35:59.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.624+0000 7f7031d9b700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f701c0396b0 0x7f701c03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7028006fd0 tx=0x7f7028006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:35:59.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.628+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7024031070 con 0x7f7034103bb0 2024-04-21T08:35:59.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.840+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f7034190df0 con 0x7f701c0396b0 2024-04-21T08:35:59.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.848+0000 7f70237fe700 1 -- 172.21.15.57:0/2496438130 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7034190df0 con 0x7f701c0396b0 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f701c0396b0 msgr2=0x7f701c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f701c0396b0 0x7f701c03bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7028006fd0 tx=0x7f7028006e40 comp rx=0 tx=0).stop 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 msgr2=0x7f7034076700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034076700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7024005800 tx=0x7f7024009f60 comp rx=0 tx=0).stop 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 shutdown_connections 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f701c0396b0 0x7f701c03bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 --2- 172.21.15.57:0/2496438130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7034103bb0 0x7f7034076700 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:35:59.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.850+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 >> 172.21.15.57:0/2496438130 conn(0x7f70340fda90 msgr2=0x7f70340fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:35:59.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.851+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 shutdown_connections 2024-04-21T08:35:59.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:35:59.851+0000 7f7038ab7700 1 -- 172.21.15.57:0/2496438130 wait complete. 2024-04-21T08:36:00.594 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2024-04-21T08:36:00.594 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:36:00.594 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:00.613 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:36:00.613 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:00.670 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2024-04-21T08:36:00.670 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch host add smithi129 2024-04-21T08:36:00.918 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: Updating smithi057:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:01.143 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:02.103 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:01 smithi057 ceph-mon[27968]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.450+0000 7fb545407700 1 -- 172.21.15.57:0/3864518925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 msgr2=0x7fb540103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.450+0000 7fb545407700 1 --2- 172.21.15.57:0/3864518925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540103a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb530009910 tx=0x7fb530009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.450+0000 7fb545407700 1 -- 172.21.15.57:0/3864518925 shutdown_connections 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.450+0000 7fb545407700 1 --2- 172.21.15.57:0/3864518925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540103a70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.450+0000 7fb545407700 1 -- 172.21.15.57:0/3864518925 >> 172.21.15.57:0/3864518925 conn(0x7fb5400fce40 msgr2=0x7fb5400ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.451+0000 7fb545407700 1 -- 172.21.15.57:0/3864518925 shutdown_connections 2024-04-21T08:36:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.451+0000 7fb545407700 1 -- 172.21.15.57:0/3864518925 wait complete. 2024-04-21T08:36:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.451+0000 7fb545407700 1 Processor -- start 2024-04-21T08:36:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.451+0000 7fb545407700 1 -- start start 2024-04-21T08:36:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb545407700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb545407700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb54010c9a0 con 0x7fb540103690 2024-04-21T08:36:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb53effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb53effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40776/0 (socket says 172.21.15.57:40776) 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb53effd700 1 -- 172.21.15.57:0/829282134 learned_addr learned my addr 172.21.15.57:0/829282134 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb53effd700 1 -- 172.21.15.57:0/829282134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb540198430 con 0x7fb540103690 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb53effd700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540197ef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb5300056d0 tx=0x7fb5300057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb530026070 con 0x7fb540103690 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5401986c0 con 0x7fb540103690 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb530004370 con 0x7fb540103690 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5300039b0 con 0x7fb540103690 2024-04-21T08:36:02.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.452+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb540198b20 con 0x7fb540103690 2024-04-21T08:36:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.453+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb530029070 con 0x7fb540103690 2024-04-21T08:36:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.453+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54006b110 con 0x7fb540103690 2024-04-21T08:36:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.453+0000 7fb527fff700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb528039650 0x7fb52803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.453+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb53001e070 con 0x7fb540103690 2024-04-21T08:36:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.454+0000 7fb53e7fc700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb528039650 0x7fb52803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:02.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.454+0000 7fb53e7fc700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb528039650 0x7fb52803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb534006fd0 tx=0x7fb534006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:02.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.459+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb530013070 con 0x7fb540103690 2024-04-21T08:36:02.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:02.671+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}) -- 0x7fb5401992c0 con 0x7fb528039650 2024-04-21T08:36:03.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:03 smithi057 ceph-mon[27968]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:03.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:03 smithi057 ceph-mon[27968]: Deploying cephadm binary to smithi129 2024-04-21T08:36:05.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:05 smithi057 ceph-mon[27968]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:06.881 INFO:teuthology.orchestra.run.smithi057.stdout:Added host 'smithi129' with addr '172.21.15.129' 2024-04-21T08:36:06.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.882+0000 7fb527fff700 1 -- 172.21.15.57:0/829282134 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb5401992c0 con 0x7fb528039650 2024-04-21T08:36:06.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb528039650 msgr2=0x7fb52803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb528039650 0x7fb52803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb534006fd0 tx=0x7fb534006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 msgr2=0x7fb540197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540197ef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb5300056d0 tx=0x7fb5300057b0 comp rx=0 tx=0).stop 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 shutdown_connections 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb528039650 0x7fb52803bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 --2- 172.21.15.57:0/829282134 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb540103690 0x7fb540197ef0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.884+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 >> 172.21.15.57:0/829282134 conn(0x7fb5400fce40 msgr2=0x7fb540104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.885+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 shutdown_connections 2024-04-21T08:36:06.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:06.885+0000 7fb545407700 1 -- 172.21.15.57:0/829282134 wait complete. 2024-04-21T08:36:07.495 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:36:07.742 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:08.145 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:07 smithi057 ceph-mon[27968]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:08.145 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:08.145 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:07 smithi057 ceph-mon[27968]: Added host smithi129 2024-04-21T08:36:08.145 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:08.145 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:09.052 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:08 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 -- 172.21.15.57:0/3727670021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 msgr2=0x7f0a84103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 --2- 172.21.15.57:0/3727670021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84103a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009910 tx=0x7f0a78009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 -- 172.21.15.57:0/3727670021 shutdown_connections 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 --2- 172.21.15.57:0/3727670021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84103a90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 -- 172.21.15.57:0/3727670021 >> 172.21.15.57:0/3727670021 conn(0x7f0a840fcea0 msgr2=0x7f0a840ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 -- 172.21.15.57:0/3727670021 shutdown_connections 2024-04-21T08:36:09.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.291+0000 7f0a8b74a700 1 -- 172.21.15.57:0/3727670021 wait complete. 2024-04-21T08:36:09.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a8b74a700 1 Processor -- start 2024-04-21T08:36:09.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a8b74a700 1 -- start start 2024-04-21T08:36:09.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a8b74a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a8b74a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8410c9e0 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a894e6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a894e6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41326/0 (socket says 172.21.15.57:41326) 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.292+0000 7f0a894e6700 1 -- 172.21.15.57:0/2645807212 learned_addr learned my addr 172.21.15.57:0/2645807212 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a894e6700 1 -- 172.21.15.57:0/2645807212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a84103200 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a894e6700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84101310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a780056d0 tx=0x7f0a780057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a78026070 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a78004370 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a780039b0 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a84101850 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.293+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a84101d10 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.294+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0a78029070 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.294+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8406b130 con 0x7f0a841036b0 2024-04-21T08:36:09.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.294+0000 7f0a767fc700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a70039650 0x7f0a7003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.294+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0a7801e070 con 0x7f0a841036b0 2024-04-21T08:36:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.294+0000 7f0a88ce5700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a70039650 0x7f0a7003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:09.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.295+0000 7f0a88ce5700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a70039650 0x7f0a7003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a80006fd0 tx=0x7f0a80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:09.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.299+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a78013070 con 0x7f0a841036b0 2024-04-21T08:36:09.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.517+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a84101f40 con 0x7f0a70039650 2024-04-21T08:36:09.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.519+0000 7f0a767fc700 1 -- 172.21.15.57:0/2645807212 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f0a84101f40 con 0x7f0a70039650 2024-04-21T08:36:09.518 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:36:09.518 INFO:teuthology.orchestra.run.smithi057.stdout:[{"addr": "172.21.15.57", "hostname": "smithi057", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}] 2024-04-21T08:36:09.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a70039650 msgr2=0x7f0a7003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:09.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a70039650 0x7f0a7003bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0a80006fd0 tx=0x7f0a80006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 msgr2=0x7f0a84101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84101310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a780056d0 tx=0x7f0a780057b0 comp rx=0 tx=0).stop 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 shutdown_connections 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a70039650 0x7f0a7003bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 --2- 172.21.15.57:0/2645807212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a841036b0 0x7f0a84101310 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 >> 172.21.15.57:0/2645807212 conn(0x7f0a840fcea0 msgr2=0x7f0a84106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 shutdown_connections 2024-04-21T08:36:09.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:09.521+0000 7f0a8b74a700 1 -- 172.21.15.57:0/2645807212 wait complete. 2024-04-21T08:36:10.130 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:09 smithi057 ceph-mon[27968]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:10.130 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:10.166 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2024-04-21T08:36:10.166 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:36:10.166 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:10.185 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:36:10.186 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:10.243 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2024-04-21T08:36:10.243 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch host add smithi136 2024-04-21T08:36:10.492 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:11.294 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:10 smithi057 ceph-mon[27968]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 -- 172.21.15.57:0/1014355871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 msgr2=0x7f28b4075250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 --2- 172.21.15.57:0/1014355871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b4075250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009910 tx=0x7f28a4009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 -- 172.21.15.57:0/1014355871 shutdown_connections 2024-04-21T08:36:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 --2- 172.21.15.57:0/1014355871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b4075250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:12.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 -- 172.21.15.57:0/1014355871 >> 172.21.15.57:0/1014355871 conn(0x7f28b40fdf80 msgr2=0x7f28b41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 -- 172.21.15.57:0/1014355871 shutdown_connections 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.033+0000 7f28ba883700 1 -- 172.21.15.57:0/1014355871 wait complete. 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28ba883700 1 Processor -- start 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28ba883700 1 -- start start 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28ba883700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28ba883700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f28b410ebb0 con 0x7f28b4074e70 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28b3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28b3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41342/0 (socket says 172.21.15.57:41342) 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28b3fff700 1 -- 172.21.15.57:0/2611700728 learned_addr learned my addr 172.21.15.57:0/2611700728 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28b3fff700 1 -- 172.21.15.57:0/2611700728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b419a6f0 con 0x7f28b4074e70 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28b3fff700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b419a1b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f28a40056d0 tx=0x7f28a40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.034+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28a4026070 con 0x7f28b4074e70 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.035+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a4004370 con 0x7f28b4074e70 2024-04-21T08:36:12.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.035+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b419a980 con 0x7f28b4074e70 2024-04-21T08:36:12.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.035+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28a40039b0 con 0x7f28b4074e70 2024-04-21T08:36:12.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.035+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b419ade0 con 0x7f28b4074e70 2024-04-21T08:36:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.035+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f28a4029070 con 0x7f28b4074e70 2024-04-21T08:36:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.035+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b4076520 con 0x7f28b4074e70 2024-04-21T08:36:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.036+0000 7f28b1ffb700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f28a00396b0 0x7f28a003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.036+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f28a401e070 con 0x7f28b4074e70 2024-04-21T08:36:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.036+0000 7f28abfff700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f28a00396b0 0x7f28a003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:12.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.036+0000 7f28abfff700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f28a00396b0 0x7f28a003bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f289c006fd0 tx=0x7f289c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:12.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.041+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28a4013070 con 0x7f28b4074e70 2024-04-21T08:36:12.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:11 smithi057 ceph-mon[27968]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:12.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:12.252+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}) -- 0x7f28b419b510 con 0x7f28a00396b0 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:13.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:13.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:14.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:13 smithi057 ceph-mon[27968]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:14.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:13 smithi057 ceph-mon[27968]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:36:14.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:13 smithi057 ceph-mon[27968]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:14.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:13 smithi057 ceph-mon[27968]: Deploying cephadm binary to smithi136 2024-04-21T08:36:14.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:13 smithi057 ceph-mon[27968]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:14.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:13 smithi057 ceph-mon[27968]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:16.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:15 smithi057 ceph-mon[27968]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:16.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:15 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:16.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:15 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:16.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:15 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:16.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.439+0000 7f28b1ffb700 1 -- 172.21.15.57:0/2611700728 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f28b419b510 con 0x7f28a00396b0 2024-04-21T08:36:16.439 INFO:teuthology.orchestra.run.smithi057.stdout:Added host 'smithi136' with addr '172.21.15.136' 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f28a00396b0 msgr2=0x7f28a003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f28a00396b0 0x7f28a003bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f289c006fd0 tx=0x7f289c006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 msgr2=0x7f28b419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b419a1b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f28a40056d0 tx=0x7f28a40057b0 comp rx=0 tx=0).stop 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 shutdown_connections 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f28a00396b0 0x7f28a003bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 --2- 172.21.15.57:0/2611700728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f28b4074e70 0x7f28b419a1b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 >> 172.21.15.57:0/2611700728 conn(0x7f28b40fdf80 msgr2=0x7f28b41066f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 shutdown_connections 2024-04-21T08:36:16.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:16.441+0000 7f28ba883700 1 -- 172.21.15.57:0/2611700728 wait complete. 2024-04-21T08:36:17.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-21T08:36:17.320 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:17.713 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:17.713 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:17 smithi057 ceph-mon[27968]: Added host smithi136 2024-04-21T08:36:17.713 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:17.713 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:17 smithi057 ceph-mon[27968]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:17.714 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:18.811 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:18.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 -- 172.21.15.57:0/3287938016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 msgr2=0x7f64401019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:18.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 --2- 172.21.15.57:0/3287938016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f64401019d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f643c009910 tx=0x7f643c009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:18.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 -- 172.21.15.57:0/3287938016 shutdown_connections 2024-04-21T08:36:18.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 --2- 172.21.15.57:0/3287938016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f64401019d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:18.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 -- 172.21.15.57:0/3287938016 >> 172.21.15.57:0/3287938016 conn(0x7f6440076ea0 msgr2=0x7f64400772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 -- 172.21.15.57:0/3287938016 shutdown_connections 2024-04-21T08:36:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.828+0000 7f6447daf700 1 -- 172.21.15.57:0/3287938016 wait complete. 2024-04-21T08:36:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6447daf700 1 Processor -- start 2024-04-21T08:36:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6447daf700 1 -- start start 2024-04-21T08:36:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6447daf700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f6440198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:18.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6447daf700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f644010b820 con 0x7f64401015f0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6445b4b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f6440198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6445b4b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f6440198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:32808/0 (socket says 172.21.15.57:32808) 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.829+0000 7f6445b4b700 1 -- 172.21.15.57:0/1439162683 learned_addr learned my addr 172.21.15.57:0/1439162683 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6445b4b700 1 -- 172.21.15.57:0/1439162683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6440199290 con 0x7f64401015f0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6445b4b700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f6440198d50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f643c0056d0 tx=0x7f643c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f643c026070 con 0x7f64401015f0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f643c00bba0 con 0x7f64401015f0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6440199580 con 0x7f64401015f0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f643c003920 con 0x7f64401015f0 2024-04-21T08:36:18.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.830+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6440199a40 con 0x7f64401015f0 2024-04-21T08:36:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.831+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f643c029070 con 0x7f64401015f0 2024-04-21T08:36:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.831+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6440102a80 con 0x7f64401015f0 2024-04-21T08:36:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.831+0000 7f6436ffd700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f642c039650 0x7f642c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.831+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f643c01e070 con 0x7f64401015f0 2024-04-21T08:36:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.831+0000 7f644534a700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f642c039650 0x7f642c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:18.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.831+0000 7f644534a700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f642c039650 0x7f642c03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6430006fd0 tx=0x7f6430006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:18.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:18.836+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f643c013070 con 0x7f64401015f0 2024-04-21T08:36:19.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.048+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f644006a420 con 0x7f642c039650 2024-04-21T08:36:19.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.049+0000 7f6436ffd700 1 -- 172.21.15.57:0/1439162683 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f644006a420 con 0x7f642c039650 2024-04-21T08:36:19.049 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:36:19.049 INFO:teuthology.orchestra.run.smithi057.stdout:[{"addr": "172.21.15.57", "hostname": "smithi057", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.051+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f642c039650 msgr2=0x7f642c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.051+0000 7f6447daf700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f642c039650 0x7f642c03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6430006fd0 tx=0x7f6430006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.051+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 msgr2=0x7f6440198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.051+0000 7f6447daf700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f6440198d50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f643c0056d0 tx=0x7f643c0057b0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.052+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 shutdown_connections 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.052+0000 7f6447daf700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f642c039650 0x7f642c03bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.052+0000 7f6447daf700 1 --2- 172.21.15.57:0/1439162683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64401015f0 0x7f6440198d50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:19.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.052+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 >> 172.21.15.57:0/1439162683 conn(0x7f6440076ea0 msgr2=0x7f6440102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:19.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.052+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 shutdown_connections 2024-04-21T08:36:19.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:19.052+0000 7f6447daf700 1 -- 172.21.15.57:0/1439162683 wait complete. 2024-04-21T08:36:19.644 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-21T08:36:19.644 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-21T08:36:19.885 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:20.318 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:20 smithi057 ceph-mon[27968]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:20.318 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:20 smithi057 ceph-mon[27968]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:20.319 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:21.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 -- 172.21.15.57:0/210446157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 msgr2=0x7f46281088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:21.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 --2- 172.21.15.57:0/210446157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f46281088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4624009910 tx=0x7f4624009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:21.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 -- 172.21.15.57:0/210446157 shutdown_connections 2024-04-21T08:36:21.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 --2- 172.21.15.57:0/210446157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f46281088c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:21.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 -- 172.21.15.57:0/210446157 >> 172.21.15.57:0/210446157 conn(0x7f46280fdf80 msgr2=0x7f46281003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:21.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 -- 172.21.15.57:0/210446157 shutdown_connections 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 -- 172.21.15.57:0/210446157 wait complete. 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.427+0000 7f4630554700 1 Processor -- start 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f4630554700 1 -- start start 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f4630554700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f4628197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f4630554700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f462810c940 con 0x7f46281084e0 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f462e2f0700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f4628197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f462e2f0700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f4628197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:32830/0 (socket says 172.21.15.57:32830) 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f462e2f0700 1 -- 172.21.15.57:0/1618087931 learned_addr learned my addr 172.21.15.57:0/1618087931 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f462e2f0700 1 -- 172.21.15.57:0/1618087931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46281983f0 con 0x7f46281084e0 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f462e2f0700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f4628197eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f46240056d0 tx=0x7f46240057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.428+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4624026070 con 0x7f46281084e0 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.429+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f462400bba0 con 0x7f46281084e0 2024-04-21T08:36:21.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.429+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46281986e0 con 0x7f46281084e0 2024-04-21T08:36:21.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.429+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4624003920 con 0x7f46281084e0 2024-04-21T08:36:21.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.429+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462819c780 con 0x7f46281084e0 2024-04-21T08:36:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.429+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4624029070 con 0x7f46281084e0 2024-04-21T08:36:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.430+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4628103b90 con 0x7f46281084e0 2024-04-21T08:36:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.430+0000 7f461f7fe700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46140396b0 0x7f461403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.430+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f462401e070 con 0x7f46281084e0 2024-04-21T08:36:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.430+0000 7f462daef700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46140396b0 0x7f461403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:21.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.430+0000 7f462daef700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46140396b0 0x7f461403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4618006fd0 tx=0x7f4618006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:21.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.435+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4624013070 con 0x7f46281084e0 2024-04-21T08:36:21.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:21.646+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f4628199210 con 0x7f46281084e0 2024-04-21T08:36:22.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.108+0000 7f461f7fe700 1 -- 172.21.15.57:0/1618087931 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f4624003b60 con 0x7f46281084e0 2024-04-21T08:36:22.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46140396b0 msgr2=0x7f461403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:22.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46140396b0 0x7f461403bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4618006fd0 tx=0x7f4618006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:22.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 msgr2=0x7f4628197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:22.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f4628197eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f46240056d0 tx=0x7f46240057b0 comp rx=0 tx=0).stop 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 shutdown_connections 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f46140396b0 0x7f461403bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 --2- 172.21.15.57:0/1618087931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46281084e0 0x7f4628197eb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 >> 172.21.15.57:0/1618087931 conn(0x7f46280fdf80 msgr2=0x7f462810a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 shutdown_connections 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:22.111+0000 7f4630554700 1 -- 172.21.15.57:0/1618087931 wait complete. 2024-04-21T08:36:22.111 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2024-04-21T08:36:22.420 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:22 smithi057 ceph-mon[27968]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:22.421 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:22 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1618087931' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:36:22.745 INFO:tasks.cephadm:Adding mon.a on smithi057 2024-04-21T08:36:22.745 INFO:tasks.cephadm:Adding mon.b on smithi129 2024-04-21T08:36:22.745 INFO:tasks.cephadm:Adding mon.c on smithi136 2024-04-21T08:36:22.746 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mon '3;smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c' 2024-04-21T08:36:22.976 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1618087931' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:24.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 -- 172.21.15.136:0/2970524877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 msgr2=0x7fd31c1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:24.464 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 --2- 172.21.15.136:0/2970524877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c1088c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009910 tx=0x7fd30c009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 -- 172.21.15.136:0/2970524877 shutdown_connections 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 --2- 172.21.15.136:0/2970524877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c1088c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 -- 172.21.15.136:0/2970524877 >> 172.21.15.136:0/2970524877 conn(0x7fd31c0fdf80 msgr2=0x7fd31c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 -- 172.21.15.136:0/2970524877 shutdown_connections 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 -- 172.21.15.136:0/2970524877 wait complete. 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 Processor -- start 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.463+0000 7fd322a14700 1 -- start start 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.464+0000 7fd322a14700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.464+0000 7fd322a14700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10c940 con 0x7fd31c1084e0 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.464+0000 7fd31bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:24.465 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.464+0000 7fd31bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.136:45060/0 (socket says 172.21.15.136:45060) 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.464+0000 7fd31bfff700 1 -- 172.21.15.136:0/401172176 learned_addr learned my addr 172.21.15.136:0/401172176 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.464+0000 7fd31bfff700 1 -- 172.21.15.136:0/401172176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c1983f0 con 0x7fd31c1084e0 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.465+0000 7fd31bfff700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0056d0 tx=0x7fd30c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.465+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd30c026070 con 0x7fd31c1084e0 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.465+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c1986e0 con 0x7fd31c1084e0 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.465+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd30c00bba0 con 0x7fd31c1084e0 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.465+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd30c003920 con 0x7fd31c1084e0 2024-04-21T08:36:24.466 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.465+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c19c780 con 0x7fd31c1084e0 2024-04-21T08:36:24.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.466+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd30c029070 con 0x7fd31c1084e0 2024-04-21T08:36:24.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.466+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c103b90 con 0x7fd31c1084e0 2024-04-21T08:36:24.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.466+0000 7fd3197fa700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd304039650 0x7fd30403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:24.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.466+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd30c01e070 con 0x7fd31c1084e0 2024-04-21T08:36:24.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.466+0000 7fd31b7fe700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd304039650 0x7fd30403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:24.467 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.467+0000 7fd31b7fe700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd304039650 0x7fd30403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd310006fd0 tx=0x7fd310006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:24.471 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.472+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd30c013070 con 0x7fd31c1084e0 2024-04-21T08:36:24.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:24.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.681+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}) -- 0x7fd31c199210 con 0x7fd304039650 2024-04-21T08:36:24.687 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.688+0000 7fd3197fa700 1 -- 172.21.15.136:0/401172176 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd31c199210 con 0x7fd304039650 2024-04-21T08:36:24.687 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mon update... 2024-04-21T08:36:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd304039650 msgr2=0x7fd30403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:24.689 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd304039650 0x7fd30403bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd310006fd0 tx=0x7fd310006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 msgr2=0x7fd31c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd30c0056d0 tx=0x7fd30c0057b0 comp rx=0 tx=0).stop 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 shutdown_connections 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd304039650 0x7fd30403bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 --2- 172.21.15.136:0/401172176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd31c1084e0 0x7fd31c197eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 >> 172.21.15.136:0/401172176 conn(0x7fd31c0fdf80 msgr2=0x7fd31c10a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 shutdown_connections 2024-04-21T08:36:24.690 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:24.690+0000 7fd322a14700 1 -- 172.21.15.136:0/401172176 wait complete. 2024-04-21T08:36:25.248 DEBUG:teuthology.orchestra.run.smithi129:mon.b> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.b.service 2024-04-21T08:36:25.250 DEBUG:teuthology.orchestra.run.smithi136:mon.c> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.c.service 2024-04-21T08:36:25.252 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:36:25.253 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: Saving service mon spec with placement smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c;count:3 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:26.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:26.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:26.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:25 smithi057 ceph-mon[27968]: Deploying daemon mon.c on smithi136 2024-04-21T08:36:26.368 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:28.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:27 smithi057 ceph-mon[27968]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.098 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 -- 172.21.15.136:0/1408183452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 msgr2=0x7f137c1019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 --2- 172.21.15.136:0/1408183452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c1019a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f137400b0a0 tx=0x7f137400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 -- 172.21.15.136:0/1408183452 shutdown_connections 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 --2- 172.21.15.136:0/1408183452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c1019a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 -- 172.21.15.136:0/1408183452 >> 172.21.15.136:0/1408183452 conn(0x7f137c0fce90 msgr2=0x7f137c0ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 -- 172.21.15.136:0/1408183452 shutdown_connections 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 -- 172.21.15.136:0/1408183452 wait complete. 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.097+0000 7f1380cff700 1 Processor -- start 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f1380cff700 1 -- start start 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f1380cff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:29.099 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f1380cff700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f137c10c940 con 0x7f137c1015c0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f137a59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f137a59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.136:45074/0 (socket says 172.21.15.136:45074) 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f137a59c700 1 -- 172.21.15.136:0/1340232261 learned_addr learned my addr 172.21.15.136:0/1340232261 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.098+0000 7f137a59c700 1 -- 172.21.15.136:0/1340232261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c198330 con 0x7f137c1015c0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.099+0000 7f137a59c700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c197df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1374006130 tx=0x7f13740097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.099+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1374021020 con 0x7f137c1015c0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.099+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137c1985c0 con 0x7f137c1015c0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.099+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13740079b0 con 0x7f137c1015c0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.099+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1374017040 con 0x7f137c1015c0 2024-04-21T08:36:29.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.099+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137c19c6c0 con 0x7f137c1015c0 2024-04-21T08:36:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.100+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137c102aa0 con 0x7f137c1015c0 2024-04-21T08:36:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.100+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1374024020 con 0x7f137c1015c0 2024-04-21T08:36:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.100+0000 7f136b7fe700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13640396b0 0x7f136403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.100+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f137400f050 con 0x7f137c1015c0 2024-04-21T08:36:29.102 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.100+0000 7f1379d9b700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13640396b0 0x7f136403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:29.103 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.101+0000 7f1379d9b700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13640396b0 0x7f136403bb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1370006fd0 tx=0x7f1370006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:29.105 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.105+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f137402b020 con 0x7f137c1015c0 2024-04-21T08:36:29.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.413+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f137c199150 con 0x7f137c1015c0 2024-04-21T08:36:29.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.414+0000 7f136b7fe700 1 -- 172.21.15.136:0/1340232261 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f1374011dc0 con 0x7f137c1015c0 2024-04-21T08:36:29.414 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:36:29.414 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:34:44.528184Z","created":"2024-04-21T08:34:44.528184Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:3300","nonce":0},{"type":"v1","addr":"172.21.15.57:6789","nonce":0}]},"addr":"172.21.15.57:6789/0","public_addr":"172.21.15.57:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13640396b0 msgr2=0x7f136403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13640396b0 0x7f136403bb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1370006fd0 tx=0x7f1370006e40 comp rx=0 tx=0).stop 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 msgr2=0x7f137c197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c197df0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1374006130 tx=0x7f13740097d0 comp rx=0 tx=0).stop 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 shutdown_connections 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13640396b0 0x7f136403bb70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 --2- 172.21.15.136:0/1340232261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f137c1015c0 0x7f137c197df0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:29.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 >> 172.21.15.136:0/1340232261 conn(0x7f137c0fce90 msgr2=0x7f137c102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:29.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 shutdown_connections 2024-04-21T08:36:29.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:29.416+0000 7f1380cff700 1 -- 172.21.15.136:0/1340232261 wait complete. 2024-04-21T08:36:29.417 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(???) e0 preinit fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).mds e1 new map 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout: e1 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout: legacy client fscid: -1 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout: No filesystems configured 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T08:36:29.526 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T08:36:29.527 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T08:36:29.527 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T08:36:29.527 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T08:36:29.527 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T08:36:29.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Deploying cephadm binary to smithi136 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Added host smithi136 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1618087931' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:36:29.888 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1618087931' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:29.889 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Saving service mon spec with placement smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c;count:3 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:29.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: Deploying daemon mon.c on smithi136 2024-04-21T08:36:29.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:29.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 ceph-mon[31791]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:36:29.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 bash[31759]: 830ca3f1e5998ad44ae7ef3c82db1011086b4f880c4aced7335d49f29987c05d 2024-04-21T08:36:29.891 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:29 smithi136 systemd[1]: Started Ceph mon.c for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:31.090 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:36:31.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:36:31.342 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: mon.a calling monitor election 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: mon.c calling monitor election 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: monmap epoch 2 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: last_changed 2024-04-21T08:36:29.533274+0000 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: min_mon_release 19 (squid) 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: election_strategy: 1 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-21T08:36:34.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: fsmap 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: mgrmap e13: x(active, since 50s) 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: overall HEALTH_OK 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:34.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: mon.a calling monitor election 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: mon.c calling monitor election 2024-04-21T08:36:35.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: monmap epoch 2 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: last_changed 2024-04-21T08:36:29.533274+0000 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: min_mon_release 19 (squid) 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: election_strategy: 1 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-21T08:36:35.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: fsmap 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: mgrmap e13: x(active, since 50s) 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: overall HEALTH_OK 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:35.091 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:35.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.788+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/660468977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c44108700 msgr2=0x7f9c30005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:35.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.788+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/660468977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c44108700 0x7f9c30005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9c40005fa0 tx=0x7f9c4000ffc0 comp rx=0 tx=0).stop 2024-04-21T08:36:35.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.788+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/660468977 shutdown_connections 2024-04-21T08:36:35.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.788+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/660468977 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c44108700 0x7f9c30005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:35.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.788+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/660468977 >> 172.21.15.136:0/660468977 conn(0x7f9c440767a0 msgr2=0x7f9c44076bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:35.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.789+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/660468977 shutdown_connections 2024-04-21T08:36:35.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.789+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/660468977 wait complete. 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.789+0000 7f9c4c1e9700 1 Processor -- start 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.789+0000 7f9c4c1e9700 1 -- start start 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c4c1e9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c4c1e9700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c441985e0 0x7f9c4419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c4c1e9700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4410a490 con 0x7f9c441985e0 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c4c1e9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4410a310 con 0x7f9c44108700 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:35.791 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46304/0 (socket says 172.21.15.136:46304) 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 -- 172.21.15.136:0/2029499099 learned_addr learned my addr 172.21.15.136:0/2029499099 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 -- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 msgr2=0x7f9c441980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 -- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 msgr2=0x7f9c441980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49f85700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.790+0000 7f9c49784700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c441985e0 0x7f9c4419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c49784700 1 -- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 msgr2=0x7f9c441980a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c49784700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c49784700 1 -- 172.21.15.136:0/2029499099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c44198c20 con 0x7f9c441985e0 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c49784700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c441985e0 0x7f9c4419c970 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400b4d0 tx=0x7f9c3400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9c34011630 con 0x7f9c441985e0 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c34011c80 con 0x7f9c441985e0 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4419ceb0 con 0x7f9c441985e0 2024-04-21T08:36:35.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9c34010da0 con 0x7f9c441985e0 2024-04-21T08:36:35.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.791+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4419d410 con 0x7f9c441985e0 2024-04-21T08:36:35.793 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.792+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9c34018020 con 0x7f9c441985e0 2024-04-21T08:36:35.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.792+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c4406a960 con 0x7f9c441985e0 2024-04-21T08:36:35.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.792+0000 7f9c3affd700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9c28039440 0x7f9c2803b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:35.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.792+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9c3404c1f0 con 0x7f9c441985e0 2024-04-21T08:36:35.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.793+0000 7f9c49f85700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9c28039440 0x7f9c2803b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:35.794 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.793+0000 7f9c49f85700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9c28039440 0x7f9c2803b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000fb20 tx=0x7f9c4001d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:35.798 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:35.798+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c3401f020 con 0x7f9c441985e0 2024-04-21T08:36:35.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:35 smithi136 ceph-mon[31791]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:35.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:35 smithi136 ceph-mon[31791]: Deploying daemon mon.b on smithi129 2024-04-21T08:36:35.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:36.084 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.085+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9c4419d720 con 0x7f9c441985e0 2024-04-21T08:36:36.085 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.085+0000 7f9c3affd700 1 -- 172.21.15.136:0/2029499099 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f9c34018570 con 0x7f9c441985e0 2024-04-21T08:36:36.085 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:36:36.085 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:36:29.533274Z","created":"2024-04-21T08:34:44.528184Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:3300","nonce":0},{"type":"v1","addr":"172.21.15.57:6789","nonce":0}]},"addr":"172.21.15.57:6789/0","public_addr":"172.21.15.57:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.087+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9c28039440 msgr2=0x7f9c2803b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.087+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9c28039440 0x7f9c2803b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c4000fb20 tx=0x7f9c4001d040 comp rx=0 tx=0).stop 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.087+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c441985e0 msgr2=0x7f9c4419c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.087+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c441985e0 0x7f9c4419c970 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9c3400b4d0 tx=0x7f9c3400b890 comp rx=0 tx=0).stop 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 shutdown_connections 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9c28039440 0x7f9c2803b900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:36.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c44108700 0x7f9c441980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:36.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 --2- 172.21.15.136:0/2029499099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c441985e0 0x7f9c4419c970 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:36.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 >> 172.21.15.136:0/2029499099 conn(0x7f9c440767a0 msgr2=0x7f9c44100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:36.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 shutdown_connections 2024-04-21T08:36:36.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:36.088+0000 7f9c4c1e9700 1 -- 172.21.15.136:0/2029499099 wait complete. 2024-04-21T08:36:36.088 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 2 2024-04-21T08:36:36.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:35 smithi057 ceph-mon[27968]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:36.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:35 smithi057 ceph-mon[27968]: Deploying daemon mon.b on smithi129 2024-04-21T08:36:36.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:36.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:36 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/2029499099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:36.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:36 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/2029499099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:37.667 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-21T08:36:37.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-21T08:36:37.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:37 smithi136 ceph-mon[31791]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:37.914 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:38.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:37 smithi057 ceph-mon[27968]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 podman[31253]: 2024-04-21 08:36:37.981789764 +0000 UTC m=+0.017870757 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:36:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 podman[31253]: 2024-04-21 08:36:38.259002473 +0000 UTC m=+0.295083456 container init a066933c3cdf93e877a3f064fa2419d87d792e95b84c82b2d80250a56954c483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:36:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 podman[31253]: 2024-04-21 08:36:38.262060895 +0000 UTC m=+0.298141881 container start a066933c3cdf93e877a3f064fa2419d87d792e95b84c82b2d80250a56954c483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: set uid:gid to 167:167 (ceph:ceph) 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev), process ceph-mon, pid 2 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pidfile_write: ignore empty --pid-file 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: load: jerasure load: lrc 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: RocksDB version: 7.9.2 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Git sha 0 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Compile date 2024-04-19 14:19:22 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: DB SUMMARY 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: DB Session ID: 1RWPCH0O9SWHE3K4DRII 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: CURRENT file: CURRENT 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: IDENTITY file: IDENTITY 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-21T08:36:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.error_if_exists: 0 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.create_if_missing: 0 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.paranoid_checks: 1 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.env: 0x55eef5ba1cc0 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.fs: PosixFileSystem 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.info_log: 0x55eef78adbe0 2024-04-21T08:36:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_file_opening_threads: 16 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.statistics: (nil) 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.use_fsync: 0 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_log_file_size: 0 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.keep_log_file_num: 1000 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.recycle_log_file_num: 0 2024-04-21T08:36:38.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.allow_fallocate: 1 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.allow_mmap_reads: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.allow_mmap_writes: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.use_direct_reads: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.create_missing_column_families: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.db_log_dir: 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.wal_dir: 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-21T08:36:38.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.advise_random_on_open: 1 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.db_write_buffer_size: 0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.write_buffer_manager: 0x55eef78410e0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.rate_limiter: (nil) 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.wal_recovery_mode: 2 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.enable_thread_tracking: 0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.enable_pipelined_write: 0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.unordered_write: 0 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-21T08:36:38.579 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.row_cache: None 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.wal_filter: None 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.allow_ingest_behind: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.two_write_queues: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.manual_wal_flush: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.wal_compression: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.atomic_flush: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.log_readahead_size: 0 2024-04-21T08:36:38.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.best_efforts_recovery: 0 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.allow_data_in_errors: 0 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.db_host_id: __hostname__ 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_background_jobs: 2 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_background_compactions: -1 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_subcompactions: 1 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_total_wal_size: 0 2024-04-21T08:36:38.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_open_files: -1 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bytes_per_sync: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_readahead_size: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_background_flushes: -1 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Compression algorithms supported: 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kZSTD supported: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kXpressCompression supported: 0 2024-04-21T08:36:38.582 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kLZ4HCCompression supported: 1 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kZlibCompression supported: 1 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kSnappyCompression supported: 1 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kLZ4Compression supported: 1 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: kBZip2Compression supported: 0 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.merge_operator: 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_filter: None 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_filter_factory: None 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.sst_partitioner_factory: None 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-21T08:36:38.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eef78addc0) 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_type: 0 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_index_type: 0 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_shortening: 1 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: checksum: 4 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: no_block_cache: 0 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache: 0x55eef78731f0 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-04-21T08:36:38.584 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_options: 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: capacity : 536870912 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_shard_bits : 4 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: strict_capacity_limit : 0 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_compressed: (nil) 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: persistent_cache: (nil) 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size: 4096 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size_deviation: 10 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_restart_interval: 16 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_block_restart_interval: 1 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: metadata_block_size: 4096 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: partition_filters: 0 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: use_delta_encoding: 1 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: filter_policy: bloomfilter 2024-04-21T08:36:38.585 INFO:journalctl@ceph.mon.b.smithi129.stdout: whole_key_filtering: 1 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: verify_compression: 0 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: format_version: 5 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_index_compression: 1 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_align: 0 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_auto_readahead_size: 262144 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: prepopulate_block_cache: 0 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: initial_auto_readahead_size: 8192 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.write_buffer_size: 33554432 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_write_buffer_number: 2 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression: NoCompression 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression: Disabled 2024-04-21T08:36:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.prefix_extractor: nullptr 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.num_levels: 7 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:36:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.level: 32767 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.strategy: 0 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.enabled: false 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.target_file_size_base: 67108864 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-21T08:36:38.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.arena_block_size: 1048576 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-21T08:36:38.589 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.disable_auto_compactions: 0 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-21T08:36:38.590 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.inplace_update_support: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.bloom_locality: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.max_successive_merges: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.paranoid_file_checks: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.force_consistency_checks: 1 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.report_bg_io_stats: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.ttl: 2592000 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-21T08:36:38.591 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.enable_blob_files: false 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.min_blob_size: 0 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.blob_file_size: 268435456 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.blob_file_starting_level: 0 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7bbf035a-352c-42fd-9c3b-88db7f9e1f41 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688598292143, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-21T08:36:38.592 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688598292564, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713688598, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7bbf035a-352c-42fd-9c3b-88db7f9e1f41", "db_session_id": "1RWPCH0O9SWHE3K4DRII", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713688598292644, "job": 1, "event": "recovery_finished"} 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eef7988000 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: DB pointer 0x55eef7972000 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** DB Stats ** 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-21T08:36:38.593 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-21T08:36:38.594 INFO:journalctl@ceph.mon.b.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache BinnedLRUCache@0x55eef78731f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-21T08:36:38.595 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: using public_addr v2:172.21.15.129:0/0 -> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: starting mon.b rank -1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(???) e0 preinit fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).mds e1 new map 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: e1 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: legacy client fscid: -1 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout: No filesystems configured 2024-04-21T08:36:38.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.597 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Deploying cephadm binary to smithi136 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:38.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Added host smithi136 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.599 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1618087931' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1618087931' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:36:38.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Saving service mon spec with placement smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c;count:3 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Deploying daemon mon.c on smithi136 2024-04-21T08:36:38.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.a calling monitor election 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.c calling monitor election 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: monmap epoch 2 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:38.602 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: last_changed 2024-04-21T08:36:29.533274+0000 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: min_mon_release 19 (squid) 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: election_strategy: 1 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: fsmap 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mgrmap e13: x(active, since 50s) 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: overall HEALTH_OK 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:38.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: Deploying daemon mon.b on smithi129 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/2029499099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 ceph-mon[31272]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 bash[31253]: a066933c3cdf93e877a3f064fa2419d87d792e95b84c82b2d80250a56954c483 2024-04-21T08:36:38.604 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:38 smithi129 systemd[1]: Started Ceph mon.b for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 -- 172.21.15.136:0/1618868561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce8c0035c0 msgr2=0x7fce8c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 --2- 172.21.15.136:0/1618868561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce8c0035c0 0x7fce8c005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcea006ac70 tx=0x7fce98009e80 comp rx=0 tx=0).stop 2024-04-21T08:36:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 -- 172.21.15.136:0/1618868561 shutdown_connections 2024-04-21T08:36:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 --2- 172.21.15.136:0/1618868561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce8c0035c0 0x7fce8c005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 -- 172.21.15.136:0/1618868561 >> 172.21.15.136:0/1618868561 conn(0x7fcea00fcf70 msgr2=0x7fcea00ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:43.708 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 -- 172.21.15.136:0/1618868561 shutdown_connections 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: mon.a calling monitor election 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: mon.c calling monitor election 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: mon.b calling monitor election 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.709 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: monmap epoch 3 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: last_changed 2024-04-21T08:36:38.321692+0000 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: min_mon_release 19 (squid) 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: election_strategy: 1 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: fsmap 2024-04-21T08:36:43.710 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:43.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: mgrmap e13: x(active, since 58s) 2024-04-21T08:36:43.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: overall HEALTH_OK 2024-04-21T08:36:43.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.711 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.708+0000 7fcea795c700 1 -- 172.21.15.136:0/1618868561 wait complete. 2024-04-21T08:36:43.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 Processor -- start 2024-04-21T08:36:43.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 -- start start 2024-04-21T08:36:43.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 0x7fcea0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:43.714 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea01a4370 0x7fcea01a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcea010a430 con 0x7fcea01a4370 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcea010a2b0 con 0x7fce8c0035c0 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.709+0000 7fcea795c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fcea010a5b0 con 0x7fcea0101960 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:46320/0 (socket says 172.21.15.136:46320) 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 -- 172.21.15.136:0/1958181409 learned_addr learned my addr 172.21.15.136:0/1958181409 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea56f8700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 0x7fcea0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 msgr2=0x7fcea0101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 msgr2=0x7fcea0101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea01a4370 0x7fcea01a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:36:43.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea4ef7700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea56f8700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 msgr2=0x7fcea0101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea56f8700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 msgr2=0x7fcea0101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea56f8700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 0x7fcea0101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea56f8700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 0x7fcea0101420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 msgr2=0x7fcea0101de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 msgr2=0x7fcea0101420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 0x7fcea0101420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 -- 172.21.15.136:0/1958181409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea01a6dd0 con 0x7fcea01a4370 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.710+0000 7fcea5ef9700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea01a4370 0x7fcea01a6760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fce9c006fd0 tx=0x7fce9c00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.711+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c015020 con 0x7fcea01a4370 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.711+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce9c011ce0 con 0x7fcea01a4370 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.711+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea01a70c0 con 0x7fcea01a4370 2024-04-21T08:36:43.716 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.711+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9c010d40 con 0x7fcea01a4370 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.711+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea01a7600 con 0x7fcea01a4370 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.712+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fce9c018020 con 0x7fcea01a4370 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.712+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcea00640a0 con 0x7fcea01a4370 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.712+0000 7fce967fc700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fce84039ae0 0x7fce8403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.712+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fce9c04c190 con 0x7fcea01a4370 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.712+0000 7fcea56f8700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fce84039ae0 0x7fce8403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.713+0000 7fcea56f8700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fce84039ae0 0x7fce8403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fce98009160 tx=0x7fce9800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:43.717 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:43.718+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce9c01f020 con 0x7fcea01a4370 2024-04-21T08:36:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: mon.a calling monitor election 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: mon.c calling monitor election 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: mon.b calling monitor election 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: monmap epoch 3 2024-04-21T08:36:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: last_changed 2024-04-21T08:36:38.321692+0000 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: min_mon_release 19 (squid) 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: election_strategy: 1 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: fsmap 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: mgrmap e13: x(active, since 58s) 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: overall HEALTH_OK 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.826 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: mon.a calling monitor election 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: mon.c calling monitor election 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: mon.b calling monitor election 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: monmap epoch 3 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: last_changed 2024-04-21T08:36:38.321692+0000 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: created 2024-04-21T08:34:44.528184+0000 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: min_mon_release 19 (squid) 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: election_strategy: 1 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: 0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2024-04-21T08:36:43.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: 2: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: fsmap 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: osdmap e4: 0 total, 0 up, 0 in 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: mgrmap e13: x(active, since 58s) 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: overall HEALTH_OK 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:43.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:44.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.007+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fcea002df80 con 0x7fcea01a4370 2024-04-21T08:36:44.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.007+0000 7fce967fc700 1 -- 172.21.15.136:0/1958181409 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7fce9c010380 con 0x7fcea01a4370 2024-04-21T08:36:44.007 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-21T08:36:44.007 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":3,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","modified":"2024-04-21T08:36:38.321692Z","created":"2024-04-21T08:34:44.528184Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:3300","nonce":0},{"type":"v1","addr":"172.21.15.57:6789","nonce":0}]},"addr":"172.21.15.57:6789/0","public_addr":"172.21.15.57:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-21T08:36:44.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.010+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fce84039ae0 msgr2=0x7fce8403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:44.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.010+0000 7fcea795c700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fce84039ae0 0x7fce8403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fce98009160 tx=0x7fce9800b040 comp rx=0 tx=0).stop 2024-04-21T08:36:44.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.010+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea01a4370 msgr2=0x7fcea01a6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:44.010 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.010+0000 7fcea795c700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea01a4370 0x7fcea01a6760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fce9c006fd0 tx=0x7fce9c00b7e0 comp rx=0 tx=0).stop 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 shutdown_connections 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fce84039ae0 0x7fce8403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce8c0035c0 0x7fcea0101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcea01a4370 0x7fcea01a6760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 --2- 172.21.15.136:0/1958181409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcea0101960 0x7fcea0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 >> 172.21.15.136:0/1958181409 conn(0x7fcea00fcf70 msgr2=0x7fcea0107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 shutdown_connections 2024-04-21T08:36:44.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:44.011+0000 7fcea795c700 1 -- 172.21.15.136:0/1958181409 wait complete. 2024-04-21T08:36:44.012 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 3 2024-04-21T08:36:44.621 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:44.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:44 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/1958181409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:44.622 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:44.623 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-21T08:36:44.623 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-21T08:36:44.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:44.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:44 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/1958181409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:44.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:44.650 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:44.650 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:44 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/1958181409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-21T08:36:44.650 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:36:44.880 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:45.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:36:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:45 smithi057 ceph-mon[27968]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:36:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:36:45.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:45 smithi136 ceph-mon[31791]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:36:46.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:46.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:46.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:46.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:46.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:36:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:36:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:45 smithi129 ceph-mon[31272]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:36:46.421 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 -- 172.21.15.57:0/1406785240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 msgr2=0x7fe59c1039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/1406785240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c1039d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe590009910 tx=0x7fe590009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 -- 172.21.15.57:0/1406785240 shutdown_connections 2024-04-21T08:36:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/1406785240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c1039d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 -- 172.21.15.57:0/1406785240 >> 172.21.15.57:0/1406785240 conn(0x7fe59c0fce40 msgr2=0x7fe59c0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 -- 172.21.15.57:0/1406785240 shutdown_connections 2024-04-21T08:36:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.421+0000 7fe5a3d16700 1 -- 172.21.15.57:0/1406785240 wait complete. 2024-04-21T08:36:46.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 Processor -- start 2024-04-21T08:36:46.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 -- start start 2024-04-21T08:36:46.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe59c198740 0x7fe59c19ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe59c19cf70 0x7fe59c19d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c10c990 con 0x7fe59c1035f0 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c10c810 con 0x7fe59c19cf70 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a3d16700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c10cb10 con 0x7fe59c198740 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a1ab2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a1ab2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58556/0 (socket says 172.21.15.57:58556) 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a1ab2700 1 -- 172.21.15.57:0/3037074938 learned_addr learned my addr 172.21.15.57:0/3037074938 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a12b1700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe59c198740 0x7fe59c19ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.422+0000 7fe5a22b3700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe59c19cf70 0x7fe59c19d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a1ab2700 1 -- 172.21.15.57:0/3037074938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe59c198740 msgr2=0x7fe59c19ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a1ab2700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe59c198740 0x7fe59c19ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a1ab2700 1 -- 172.21.15.57:0/3037074938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe59c19cf70 msgr2=0x7fe59c19d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a1ab2700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe59c19cf70 0x7fe59c19d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a1ab2700 1 -- 172.21.15.57:0/3037074938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59c1a4ce0 con 0x7fe59c1035f0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a22b3700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe59c19cf70 0x7fe59c19d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a1ab2700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c198200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe590005b40 tx=0x7fe590004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe590026070 con 0x7fe59c1035f0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe59000b750 con 0x7fe59c1035f0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5900038e0 con 0x7fe59c1035f0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59c1a4eb0 con 0x7fe59c1035f0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.423+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe59c1a52b0 con 0x7fe59c1035f0 2024-04-21T08:36:46.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.424+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59c0640a0 con 0x7fe59c1035f0 2024-04-21T08:36:46.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.425+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe590029070 con 0x7fe59c1035f0 2024-04-21T08:36:46.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.425+0000 7fe58effd700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe588039ae0 0x7fe58803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:46.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.425+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe59004d9e0 con 0x7fe59c1035f0 2024-04-21T08:36:46.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.425+0000 7fe5a12b1700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe588039ae0 0x7fe58803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:46.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.426+0000 7fe5a12b1700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe588039ae0 0x7fe58803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe598009ad0 tx=0x7fe598009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:46.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.430+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe590030020 con 0x7fe59c1035f0 2024-04-21T08:36:46.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.645+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe59c1a56b0 con 0x7fe59c1035f0 2024-04-21T08:36:46.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.646+0000 7fe58effd700 1 -- 172.21.15.57:0/3037074938 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fe590013070 con 0x7fe59c1035f0 2024-04-21T08:36:46.646 INFO:teuthology.orchestra.run.smithi057.stdout:# minimal ceph.conf for df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:46.646 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2024-04-21T08:36:46.647 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:36:46.647 INFO:teuthology.orchestra.run.smithi057.stdout: mon_host = [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2024-04-21T08:36:46.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe588039ae0 msgr2=0x7fe58803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe588039ae0 0x7fe58803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe598009ad0 tx=0x7fe598009210 comp rx=0 tx=0).stop 2024-04-21T08:36:46.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 msgr2=0x7fe59c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:46.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c198200 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe590005b40 tx=0x7fe590004ed0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 shutdown_connections 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe588039ae0 0x7fe58803bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe59c19cf70 0x7fe59c19d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe59c1035f0 0x7fe59c198200 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 --2- 172.21.15.57:0/3037074938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe59c198740 0x7fe59c19ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 >> 172.21.15.57:0/3037074938 conn(0x7fe59c0fce40 msgr2=0x7fe59c1063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 shutdown_connections 2024-04-21T08:36:46.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:46.648+0000 7fe5a3d16700 1 -- 172.21.15.57:0/3037074938 wait complete. 2024-04-21T08:36:47.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:36:47.095 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:47.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:47.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: Reconfiguring daemon mon.a on smithi057 2024-04-21T08:36:47.096 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:46 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3037074938' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:36:47.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:47.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:47.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: Reconfiguring daemon mon.a on smithi057 2024-04-21T08:36:47.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:46 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3037074938' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:36:47.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: Reconfiguring mon.a (unknown last config time)... 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:47.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: Reconfiguring daemon mon.a on smithi057 2024-04-21T08:36:47.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:46 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3037074938' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:47.817 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-21T08:36:47.817 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:36:47.817 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:47.884 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:36:47.885 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:47.950 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:36:47.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:47.982 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:36:47.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:48.050 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:36:48.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-04-21T08:36:48.081 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:36:48.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:36:48.148 INFO:tasks.cephadm:Adding mgr.x on smithi057 2024-04-21T08:36:48.148 INFO:tasks.cephadm:Adding mgr.y on smithi129 2024-04-21T08:36:48.148 INFO:tasks.cephadm:Adding mgr.z on smithi136 2024-04-21T08:36:48.149 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi057=x;smithi129=y;smithi136=z' 2024-04-21T08:36:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:48 smithi129 ceph-mon[31272]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:48.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:48 smithi136 ceph-mon[31791]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:48.422 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:36:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:48 smithi057 ceph-mon[27968]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:49 smithi129 ceph-mon[31272]: Reconfiguring daemon mon.b on smithi129 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:49 smithi057 ceph-mon[27968]: Reconfiguring daemon mon.b on smithi129 2024-04-21T08:36:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: Reconfiguring mon.b (monmap changed)... 2024-04-21T08:36:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:49.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:49 smithi136 ceph-mon[31791]: Reconfiguring daemon mon.b on smithi129 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.846+0000 7f2109e7e700 1 -- 172.21.15.136:0/370930392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2104103030 msgr2=0x7f2104105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.846+0000 7f2109e7e700 1 --2- 172.21.15.136:0/370930392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2104103030 0x7f2104105420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009910 tx=0x7f20f4009c20 comp rx=0 tx=0).stop 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 -- 172.21.15.136:0/370930392 shutdown_connections 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 --2- 172.21.15.136:0/370930392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2104103030 0x7f2104105420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 -- 172.21.15.136:0/370930392 >> 172.21.15.136:0/370930392 conn(0x7f21040fca40 msgr2=0x7f21040fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 -- 172.21.15.136:0/370930392 shutdown_connections 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 -- 172.21.15.136:0/370930392 wait complete. 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 Processor -- start 2024-04-21T08:36:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.847+0000 7f2109e7e700 1 -- start start 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2109e7e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2104103030 0x7f2104197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2109e7e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21041981c0 0x7f210419d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2109e7e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 0x7f210419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2109e7e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2104108480 con 0x7f21041981c0 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2109e7e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2104108300 con 0x7f2104103030 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2109e7e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2104108600 con 0x7f2104198640 2024-04-21T08:36:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 0x7f210419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 0x7f210419f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:51394/0 (socket says 172.21.15.136:51394) 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 -- 172.21.15.136:0/46062603 learned_addr learned my addr 172.21.15.136:0/46062603 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f20faffd700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21041981c0 0x7f210419d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f21037fe700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2104103030 0x7f2104197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 -- 172.21.15.136:0/46062603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2104103030 msgr2=0x7f2104197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2104103030 0x7f2104197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 -- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21041981c0 msgr2=0x7f210419d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21041981c0 0x7f210419d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 -- 172.21.15.136:0/46062603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21041a00d0 con 0x7f2104198640 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f2103fff700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 0x7f210419f7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00ba50 tx=0x7f20fc00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc019070 con 0x7f2104198640 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.848+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20fc00cde0 con 0x7f2104198640 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.849+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20fc007350 con 0x7f2104198640 2024-04-21T08:36:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.849+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21041a03c0 con 0x7f2104198640 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.849+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21041a0930 con 0x7f2104198640 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.849+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f20fc01c070 con 0x7f2104198640 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.849+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21040640a0 con 0x7f2104198640 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.850+0000 7f21017fa700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20f0039ae0 0x7f20f003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.850+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f20fc04be20 con 0x7f2104198640 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.850+0000 7f21037fe700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20f0039ae0 0x7f20f003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.850+0000 7f21037fe700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20f0039ae0 0x7f20f003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f20f4005190 tx=0x7f20f401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:49.854 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:49.855+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20fc012c60 con 0x7f2104198640 2024-04-21T08:36:50.070 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.070+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi129=y;smithi136=z", "target": ["mon-mgr", ""]}) -- 0x7f2104062f10 con 0x7f20f0039ae0 2024-04-21T08:36:50.076 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.077+0000 7f21017fa700 1 -- 172.21.15.136:0/46062603 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2104062f10 con 0x7f20f0039ae0 2024-04-21T08:36:50.076 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mgr update... 2024-04-21T08:36:50.078 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20f0039ae0 msgr2=0x7f20f003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20f0039ae0 0x7f20f003bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f20f4005190 tx=0x7f20f401a040 comp rx=0 tx=0).stop 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 msgr2=0x7f210419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 0x7f210419f7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00ba50 tx=0x7f20fc00bd60 comp rx=0 tx=0).stop 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 shutdown_connections 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20f0039ae0 0x7f20f003bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2104103030 0x7f2104197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2104198640 0x7f210419f7d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 --2- 172.21.15.136:0/46062603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21041981c0 0x7f210419d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 >> 172.21.15.136:0/46062603 conn(0x7f21040fca40 msgr2=0x7f21040fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:50.079 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 shutdown_connections 2024-04-21T08:36:50.080 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:36:50.079+0000 7f2109e7e700 1 -- 172.21.15.136:0/46062603 wait complete. 2024-04-21T08:36:50.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.664 DEBUG:teuthology.orchestra.run.smithi129:mgr.y> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.y.service 2024-04-21T08:36:50.667 DEBUG:teuthology.orchestra.run.smithi136:mgr.z> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.z.service 2024-04-21T08:36:50.669 INFO:tasks.cephadm:Deploying OSDs... 2024-04-21T08:36:50.670 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:36:50.670 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:36:50.690 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:36:50.690 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2024-04-21T08:36:50.690 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:50.691 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.691 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.691 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-21T08:36:50.691 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-21T08:36:50.691 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:50.691 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-04-21 08:35:57.287631707 +0000 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-04-21 08:30:51.081431663 +0000 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-04-21 08:30:51.081431663 +0000 2024-04-21T08:36:50.749 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-04-21 08:30:51.081431663 +0000 2024-04-21T08:36:50.750 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:36:50.818 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-04-21T08:36:50.818 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-04-21T08:36:50.818 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000325026 s, 1.6 MB/s 2024-04-21T08:36:50.820 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:36:50.878 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-04-21 08:35:57.318631244 +0000 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-04-21 08:30:51.426426311 +0000 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-04-21 08:30:51.426426311 +0000 2024-04-21T08:36:50.937 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-04-21 08:30:51.426426311 +0000 2024-04-21T08:36:50.937 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:36:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-04-21T08:36:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-04-21T08:36:51.003 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000286295 s, 1.8 MB/s 2024-04-21T08:36:51.004 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:36:51.063 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-04-21 08:35:57.346630826 +0000 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-04-21 08:30:51.786420727 +0000 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-04-21 08:30:51.786420727 +0000 2024-04-21T08:36:51.120 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-04-21 08:30:51.786420727 +0000 2024-04-21T08:36:51.121 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:36:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-04-21T08:36:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-04-21T08:36:51.185 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000298988 s, 1.7 MB/s 2024-04-21T08:36:51.187 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:36:51.244 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2024-04-21T08:36:51.300 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-04-21 08:35:57.375630393 +0000 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-04-21 08:30:52.129415406 +0000 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-04-21 08:30:52.129415406 +0000 2024-04-21T08:36:51.301 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-04-21 08:30:52.129415406 +0000 2024-04-21T08:36:51.301 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:36:51.365 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-04-21T08:36:51.366 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-04-21T08:36:51.366 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000303138 s, 1.7 MB/s 2024-04-21T08:36:51.367 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:36:51.424 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-04-21T08:36:51.424 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:36:51.443 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:36:51.444 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-04-21T08:36:51.501 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:36:51.501 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:51.501 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-21T08:36:51.501 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:51.501 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:51.501 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-21 08:36:37.779679402 +0000 2024-04-21T08:36:51.502 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-21 08:31:06.321911997 +0000 2024-04-21T08:36:51.502 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-21 08:31:06.321911997 +0000 2024-04-21T08:36:51.502 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-21 08:31:06.321911997 +0000 2024-04-21T08:36:51.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:36:51.567 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-21T08:36:51.567 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-21T08:36:51.567 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000264599 s, 1.9 MB/s 2024-04-21T08:36:51.569 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:36:51.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:51 smithi057 ceph-mon[27968]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:36:51.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:51 smithi057 ceph-mon[27968]: Reconfiguring daemon mon.c on smithi136 2024-04-21T08:36:51.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:51 smithi057 ceph-mon[27968]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi129=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:51.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:51 smithi057 ceph-mon[27968]: Saving service mgr spec with placement smithi057=x;smithi129=y;smithi136=z;count:3 2024-04-21T08:36:51.628 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-04-21T08:36:51.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:51 smithi136 ceph-mon[31791]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:36:51.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:51 smithi136 ceph-mon[31791]: Reconfiguring daemon mon.c on smithi136 2024-04-21T08:36:51.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:51 smithi136 ceph-mon[31791]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi129=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:51.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:51 smithi136 ceph-mon[31791]: Saving service mgr spec with placement smithi057=x;smithi129=y;smithi136=z;count:3 2024-04-21T08:36:51.684 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-21 08:36:37.780679380 +0000 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-21 08:31:06.656904565 +0000 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-21 08:31:06.656904565 +0000 2024-04-21T08:36:51.685 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-21 08:31:06.656904565 +0000 2024-04-21T08:36:51.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:36:51.746 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:51 smithi129 ceph-mon[31272]: Reconfiguring mon.c (monmap changed)... 2024-04-21T08:36:51.746 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:51 smithi129 ceph-mon[31272]: Reconfiguring daemon mon.c on smithi136 2024-04-21T08:36:51.746 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:51 smithi129 ceph-mon[31272]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi057=x;smithi129=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:51.746 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:51 smithi129 ceph-mon[31272]: Saving service mgr spec with placement smithi057=x;smithi129=y;smithi136=z;count:3 2024-04-21T08:36:51.748 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-21T08:36:51.748 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-21T08:36:51.748 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000218778 s, 2.3 MB/s 2024-04-21T08:36:51.750 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:36:51.808 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-21 08:36:37.780679380 +0000 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-21 08:31:06.990897156 +0000 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-21 08:31:06.990897156 +0000 2024-04-21T08:36:51.865 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-21 08:31:06.990897156 +0000 2024-04-21T08:36:51.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:36:51.929 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-21T08:36:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-21T08:36:51.930 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000275551 s, 1.9 MB/s 2024-04-21T08:36:51.931 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:36:51.988 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-04-21 08:36:37.780679380 +0000 2024-04-21T08:36:52.044 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-04-21 08:31:07.333889547 +0000 2024-04-21T08:36:52.045 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-04-21 08:31:07.333889547 +0000 2024-04-21T08:36:52.045 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-04-21 08:31:07.333889547 +0000 2024-04-21T08:36:52.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:36:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-04-21T08:36:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-04-21T08:36:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000232973 s, 2.2 MB/s 2024-04-21T08:36:52.109 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:36:52.165 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-21T08:36:52.165 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-04-21T08:36:52.181 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-21T08:36:52.181 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-04-21T08:36:52.236 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-21T08:36:52.236 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:52.236 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-21T08:36:52.236 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:52.237 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:52.237 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-21 08:36:28.802041902 +0000 2024-04-21T08:36:52.237 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-21 08:31:02.728402214 +0000 2024-04-21T08:36:52.237 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-21 08:31:02.728402214 +0000 2024-04-21T08:36:52.237 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-21 08:31:02.728402214 +0000 2024-04-21T08:36:52.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-21T08:36:52.299 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-21T08:36:52.299 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-21T08:36:52.299 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000200819 s, 2.5 MB/s 2024-04-21T08:36:52.300 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-21T08:36:52.356 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-21 08:36:28.802041902 +0000 2024-04-21T08:36:52.412 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-21 08:31:03.070394469 +0000 2024-04-21T08:36:52.413 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-21 08:31:03.070394469 +0000 2024-04-21T08:36:52.413 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-21 08:31:03.070394469 +0000 2024-04-21T08:36:52.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-21T08:36:52.474 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-21T08:36:52.474 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-21T08:36:52.475 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000243699 s, 2.1 MB/s 2024-04-21T08:36:52.476 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-21T08:36:52.531 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-04-21T08:36:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:52 smithi129 ceph-mon[31272]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-21 08:36:28.803041879 +0000 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-21 08:31:03.392387177 +0000 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-21 08:31:03.392387177 +0000 2024-04-21T08:36:52.586 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-21 08:31:03.392387177 +0000 2024-04-21T08:36:52.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-21T08:36:52.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:52 smithi136 ceph-mon[31791]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:52.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:52.649 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-21T08:36:52.649 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-21T08:36:52.649 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000224062 s, 2.3 MB/s 2024-04-21T08:36:52.650 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-21T08:36:52.706 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-04-21T08:36:52.761 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-21T08:36:52.761 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-21T08:36:52.761 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-21T08:36:52.761 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:36:52.761 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-21T08:36:52.761 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-21 08:36:28.803041879 +0000 2024-04-21T08:36:52.762 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-21 08:31:03.713379907 +0000 2024-04-21T08:36:52.762 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-21 08:31:03.713379907 +0000 2024-04-21T08:36:52.762 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-21 08:31:03.713379907 +0000 2024-04-21T08:36:52.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-21T08:36:52.824 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-21T08:36:52.824 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-21T08:36:52.824 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000223155 s, 2.3 MB/s 2024-04-21T08:36:52.825 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-21T08:36:52.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:52 smithi057 ceph-mon[27968]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:52.880 INFO:tasks.cephadm:Deploying osd.0 on smithi057 with /dev/vg_nvme/lv_4... 2024-04-21T08:36:52.881 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:36:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:53.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:53.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:53.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:54.528 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:54 smithi057 ceph-mon[27968]: Deploying daemon mgr.z on smithi136 2024-04-21T08:36:54.529 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:54 smithi057 ceph-mon[27968]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:54.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:54 smithi136 ceph-mon[31791]: Deploying daemon mgr.z on smithi136 2024-04-21T08:36:54.571 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:54 smithi136 ceph-mon[31791]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:54 smithi129 ceph-mon[31272]: Deploying daemon mgr.z on smithi136 2024-04-21T08:36:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:54 smithi129 ceph-mon[31272]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:54.865 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:54 smithi136 ceph-mgr[33648]: -- 172.21.15.136:0/2772948436 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e2b21025a0 con 0x55e2b20d6c00 2024-04-21T08:36:55.125 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:54 smithi136 bash[33629]: b1b5b8418733e7ef9e9f14a430309fc1d9be97ce69b117b1ac2e7079d42cf883 2024-04-21T08:36:55.125 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:54 smithi136 systemd[1]: Started Ceph mgr.z for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:55.125 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:54.944+0000 7fc9de654200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:36:55.125 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:55.125+0000 7fc9de654200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:55.132 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:36:55.132 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:36:55.132 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2024-04-21T08:36:55.132 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2024-04-21T08:36:55.132 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00987495 s, 1.1 GB/s 2024-04-21T08:36:55.132 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2024-04-21T08:36:55.384 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:55.224+0000 7fc9de654200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:36:55.728 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi057:vg_nvme/lv_4 2024-04-21T08:36:55.885 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:55.527+0000 7fc9de654200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:36:55.885 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:55.610+0000 7fc9de654200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:36:55.972 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:55 smithi057 ceph-mon[27968]: Deploying daemon mgr.y on smithi129 2024-04-21T08:36:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:55 smithi129 ceph-mon[31272]: Deploying daemon mgr.y on smithi129 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:55 smithi136 ceph-mon[31791]: Deploying daemon mgr.y on smithi129 2024-04-21T08:36:56.970 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:56 smithi129 systemd[1]: Starting Ceph mgr.y for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:36:57.134 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:56.791+0000 7fc9de654200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:36:57.135 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:56.961+0000 7fc9de654200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:36:57.252 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 podman[32652]: 2024-04-21 08:36:57.034327763 +0000 UTC m=+0.017813557 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:36:57.252 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 podman[32652]: 2024-04-21 08:36:57.13568902 +0000 UTC m=+0.119174829 container create b3b453020d0251078ac6efed76621a272319b93568d0fd0204bb996649cbebd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-21T08:36:57.441 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:57.139+0000 7fc9de654200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:36:57.442 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:57.442+0000 7fc9de654200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:36:57.513 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 podman[32652]: 2024-04-21 08:36:57.35493971 +0000 UTC m=+0.338425481 container init b3b453020d0251078ac6efed76621a272319b93568d0fd0204bb996649cbebd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T08:36:57.514 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 podman[32652]: 2024-04-21 08:36:57.357542468 +0000 UTC m=+0.341028244 container start b3b453020d0251078ac6efed76621a272319b93568d0fd0204bb996649cbebd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-04-21T08:36:57.514 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mgr[32671]: -- 172.21.15.129:0/2936645185 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b1fd905a0 con 0x560b1fd65000 2024-04-21T08:36:57.514 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 bash[32652]: b3b453020d0251078ac6efed76621a272319b93568d0fd0204bb996649cbebd4 2024-04-21T08:36:57.514 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 systemd[1]: Started Ceph mgr.y for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 -- 172.21.15.57:0/1679615541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350104660 msgr2=0x7f635010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 --2- 172.21.15.57:0/1679615541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350104660 0x7f635010af10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f634800b0d0 tx=0x7f634800b3e0 comp rx=0 tx=0).stop 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 -- 172.21.15.57:0/1679615541 shutdown_connections 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 --2- 172.21.15.57:0/1679615541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350104660 0x7f635010af10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 --2- 172.21.15.57:0/1679615541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6350102ac0 0x7f6350102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 --2- 172.21.15.57:0/1679615541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6350103cc0 0x7f6350104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 -- 172.21.15.57:0/1679615541 >> 172.21.15.57:0/1679615541 conn(0x7f63500fe060 msgr2=0x7f63501004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.572+0000 7f6358722700 1 -- 172.21.15.57:0/1679615541 shutdown_connections 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.573+0000 7f6358722700 1 -- 172.21.15.57:0/1679615541 wait complete. 2024-04-21T08:36:57.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.573+0000 7f6358722700 1 Processor -- start 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.573+0000 7f6358722700 1 -- start start 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.573+0000 7f6358722700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6350102ac0 0x7f6350198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.573+0000 7f6358722700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 0x7f6350199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6358722700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6350104660 0x7f63501a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6358722700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f635010e650 con 0x7f6350103cc0 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6358722700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f635010e4d0 con 0x7f6350104660 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6358722700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f635010e7d0 con 0x7f6350102ac0 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 0x7f6350199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 0x7f6350199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44352/0 (socket says 172.21.15.57:44352) 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 -- 172.21.15.57:0/1475343198 learned_addr learned my addr 172.21.15.57:0/1475343198 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6356cbf700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6350104660 0x7f63501a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 -- 172.21.15.57:0/1475343198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6350102ac0 msgr2=0x7f6350198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:57.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6350102ac0 0x7f6350198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 -- 172.21.15.57:0/1475343198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6350104660 msgr2=0x7f63501a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6350104660 0x7f63501a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 -- 172.21.15.57:0/1475343198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63501a0950 con 0x7f6350103cc0 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6355cbd700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 0x7f6350199240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f634000b530 tx=0x7f634000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340015020 con 0x7f6350103cc0 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6340011ce0 con 0x7f6350103cc0 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.574+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63501a0c40 con 0x7f6350103cc0 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.575+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63501a1180 con 0x7f6350103cc0 2024-04-21T08:36:57.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.575+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340010e40 con 0x7f6350103cc0 2024-04-21T08:36:57.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.575+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6340018020 con 0x7f6350103cc0 2024-04-21T08:36:57.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.576+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6350069730 con 0x7f6350103cc0 2024-04-21T08:36:57.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.576+0000 7f63477fe700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f633c039890 0x7f633c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:36:57.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.576+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f634004c370 con 0x7f6350103cc0 2024-04-21T08:36:57.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.576+0000 7f63564be700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f633c039890 0x7f633c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:36:57.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.576+0000 7f63564be700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f633c039890 0x7f633c03bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f635019a1e0 tx=0x7f634c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:36:57.580 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.581+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f634001f020 con 0x7f6350103cc0 2024-04-21T08:36:57.752 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:57.538+0000 7fc9de654200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:36:57.752 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:57.753+0000 7fc9de654200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:36:57.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:36:57.798+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6350062e80 con 0x7f633c039890 2024-04-21T08:36:57.807 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:57.514+0000 7f1b742e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:36:57.807 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:57.699+0000 7f1b742e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:57.807 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:57.808+0000 7f1b742e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:36:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:58.135 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:57.956+0000 7fc9de654200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:36:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:58.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:58.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:58.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:36:58.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:36:58.441 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:58 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:58.135+0000 7f1b742e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:36:58.441 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:58 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:58.232+0000 7f1b742e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:36:58.885 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:58.596+0000 7fc9de654200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:36:59.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:58 smithi057 ceph-mon[27968]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:59.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:59.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:59.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:58 smithi129 ceph-mon[31272]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:59.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:59.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:59.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:58 smithi136 ceph-mon[31791]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:36:59.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:59.336 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:36:59.336 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:59.035+0000 7fc9de654200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:36:59.336 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:59.132+0000 7fc9de654200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:36:59.336 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:59.336+0000 7fc9de654200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:36:59.809 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:59.468+0000 7f1b742e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:36:59.810 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:59.638+0000 7f1b742e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:36:59.810 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:36:59.809+0000 7f1b742e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:37:00.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:37:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: Reconfiguring daemon mgr.x on smithi057 2024-04-21T08:37:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:36:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: Reconfiguring daemon mgr.x on smithi057 2024-04-21T08:37:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:36:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.134 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:59.859+0000 7fc9de654200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:37:00.134 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:36:59.950+0000 7fc9de654200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: Reconfiguring mgr.x (unknown last config time)... 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: Reconfiguring daemon mgr.x on smithi057 2024-04-21T08:37:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:36:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:00.391 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:00.096+0000 7f1b742e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:37:00.391 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:00.188+0000 7f1b742e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:37:00.391 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:00.392+0000 7f1b742e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:37:00.532 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:00.191+0000 7fc9de654200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:37:00.533 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:00.324+0000 7fc9de654200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:37:00.533 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:00.418+0000 7fc9de654200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:37:00.533 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:00.533+0000 7fc9de654200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:37:00.823 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:00.582+0000 7f1b742e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:37:01.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3411313464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "773b7403-49ea-4952-8d1e-d30974987af6"}]: dispatch 2024-04-21T08:37:01.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3411313464' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "773b7403-49ea-4952-8d1e-d30974987af6"}]': finished 2024-04-21T08:37:01.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-mon[31791]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:37:01.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:01.136 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:00.835+0000 7fc9de654200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:37:01.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:01.183+0000 7f1b742e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:37:01.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3411313464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "773b7403-49ea-4952-8d1e-d30974987af6"}]: dispatch 2024-04-21T08:37:01.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3411313464' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "773b7403-49ea-4952-8d1e-d30974987af6"}]': finished 2024-04-21T08:37:01.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-mon[31272]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:37:01.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:01.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:00 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3411313464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "773b7403-49ea-4952-8d1e-d30974987af6"}]: dispatch 2024-04-21T08:37:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:00 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3411313464' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "773b7403-49ea-4952-8d1e-d30974987af6"}]': finished 2024-04-21T08:37:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:00 smithi057 ceph-mon[27968]: osdmap e5: 1 total, 0 up, 1 in 2024-04-21T08:37:01.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:01.884 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:01.431+0000 7fc9de654200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:37:01.885 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:37:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:37:01.647+0000 7fc9de654200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:37:01.896 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:01.602+0000 7f1b742e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:37:01.896 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:01.697+0000 7f1b742e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:37:01.896 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:01.898+0000 7f1b742e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:37:01.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:01.994+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f63400193b0 con 0x7f6350103cc0 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/664756944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: Standby manager daemon z started 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:37:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:37:02.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:02.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/664756944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:02.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: Standby manager daemon z started 2024-04-21T08:37:02.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:37:02.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:37:02.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:37:02.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:02 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:37:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/664756944' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: Standby manager daemon z started 2024-04-21T08:37:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:37:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:37:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:37:02.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:02 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/3583250105' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:37:02.752 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:02.422+0000 7f1b742e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:37:02.752 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:02.513+0000 7f1b742e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:37:03.008 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:02.753+0000 7f1b742e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:37:03.008 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:02.885+0000 7f1b742e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:37:03.008 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:02.980+0000 7f1b742e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:37:03.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:03 smithi129 ceph-mon[31272]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-21T08:37:03.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:37:03.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:03 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:03.097+0000 7f1b742e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:37:03.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:03 smithi057 ceph-mon[27968]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-21T08:37:03.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:37:03.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:03 smithi136 ceph-mon[31791]: mgrmap e14: x(active, since 77s), standbys: z 2024-04-21T08:37:03.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:37:03.823 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:03 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:03.405+0000 7f1b742e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:37:04.283 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:04 smithi057 ceph-mon[27968]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:04.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:04 smithi129 ceph-mon[31272]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:04.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:03 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:03.995+0000 7f1b742e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:37:04.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:37:04 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:37:04.211+0000 7f1b742e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:37:04.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:04 smithi136 ceph-mon[31791]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:05.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:05.023+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f634000f2e0 con 0x7f6350103cc0 2024-04-21T08:37:05.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:05 smithi057 ceph-mon[27968]: Standby manager daemon y started 2024-04-21T08:37:05.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:05 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:37:05.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:05 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:37:05.210 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:05 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:37:05.211 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:05 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:37:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:05 smithi129 ceph-mon[31272]: Standby manager daemon y started 2024-04-21T08:37:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:05 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:37:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:05 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:37:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:05 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:37:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:05 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:37:05.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:05 smithi136 ceph-mon[31791]: Standby manager daemon y started 2024-04-21T08:37:05.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:05 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:37:05.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:05 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:37:05.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:05 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:37:05.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:05 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2176382946' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:37:06.063 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:06 smithi057 ceph-mon[27968]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:06.063 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:06 smithi057 ceph-mon[27968]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T08:37:06.063 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:37:06.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:06 smithi129 ceph-mon[31272]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:06.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:06 smithi129 ceph-mon[31272]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T08:37:06.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:37:06.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:06 smithi136 ceph-mon[31791]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:06.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:06 smithi136 ceph-mon[31791]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-21T08:37:06.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:37:07.161 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.161 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.162 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:07.162 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:07.162 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:07.162 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:07.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:07.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:07.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:08.444 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:08 smithi057 ceph-mon[27968]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:08.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:08 smithi129 ceph-mon[31272]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:08 smithi136 ceph-mon[31791]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:09.521 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:37:09.521 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:09.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:37:09.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-21T08:37:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:10.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:10 smithi057 ceph-mon[27968]: Deploying daemon osd.0 on smithi057 2024-04-21T08:37:10.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:10 smithi057 ceph-mon[27968]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:10 smithi129 ceph-mon[31272]: Deploying daemon osd.0 on smithi057 2024-04-21T08:37:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:10 smithi129 ceph-mon[31272]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:10.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:10 smithi136 ceph-mon[31791]: Deploying daemon osd.0 on smithi057 2024-04-21T08:37:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:10 smithi136 ceph-mon[31791]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:12 smithi129 ceph-mon[31272]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:12 smithi057 ceph-mon[27968]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:12.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:12 smithi136 ceph-mon[31791]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:14 smithi129 ceph-mon[31272]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:14.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:14 smithi057 ceph-mon[27968]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:14.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:14.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:14 smithi136 ceph-mon[31791]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:14.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:14.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:14.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:16.466 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:16 smithi057 ceph-mon[27968]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:16 smithi129 ceph-mon[31272]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:16.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:16 smithi136 ceph-mon[31791]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:17.468 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 0 on host 'smithi057' 2024-04-21T08:37:17.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.469+0000 7f63477fe700 1 -- 172.21.15.57:0/1475343198 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6350062e80 con 0x7f633c039890 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f633c039890 msgr2=0x7f633c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f633c039890 0x7f633c03bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f635019a1e0 tx=0x7f634c006d20 comp rx=0 tx=0).stop 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 msgr2=0x7f6350199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 0x7f6350199240 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f634000b530 tx=0x7f634000b840 comp rx=0 tx=0).stop 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 shutdown_connections 2024-04-21T08:37:17.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f633c039890 0x7f633c03bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6350104660 0x7f63501a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6350102ac0 0x7f6350198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 --2- 172.21.15.57:0/1475343198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6350103cc0 0x7f6350199240 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 >> 172.21.15.57:0/1475343198 conn(0x7f63500fe060 msgr2=0x7f6350107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 shutdown_connections 2024-04-21T08:37:17.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:37:17.471+0000 7f6358722700 1 -- 172.21.15.57:0/1475343198 wait complete. 2024-04-21T08:37:18.097 DEBUG:teuthology.orchestra.run.smithi057:osd.0> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.0.service 2024-04-21T08:37:18.100 INFO:tasks.cephadm:Deploying osd.1 on smithi129 with /dev/vg_nvme/lv_4... 2024-04-21T08:37:18.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:37:18.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:18 smithi057 ceph-mon[27968]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:18.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:18 smithi057 ceph-mon[27968]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:37:18.343 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:37:18.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:18 smithi129 ceph-mon[31272]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:18.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.465 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:18 smithi129 ceph-mon[31272]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:37:18.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:18 smithi136 ceph-mon[31791]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:18 smithi136 ceph-mon[31791]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-21T08:37:19.705 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:19 smithi129 ceph-mon[31272]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:37:19.705 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:19 smithi129 ceph-mon[31272]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:37:19.705 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:19 smithi129 ceph-mon[31272]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:37:19.705 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:19 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:19.706 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:19 smithi129 ceph-mon[31272]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:19.786 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-mon[27968]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:37:19.786 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-mon[27968]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:37:19.786 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-mon[27968]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:37:19.786 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:19.786 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-mon[27968]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:19.786 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0[42757]: 2024-04-21T08:37:19.510+0000 7fd19cf5f700 -1 osd.0 0 waiting for initial osdmap 2024-04-21T08:37:19.787 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:37:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0[42757]: 2024-04-21T08:37:19.515+0000 7fd198d52700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:37:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:19 smithi136 ceph-mon[31791]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-21T08:37:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:19 smithi136 ceph-mon[31791]: osdmap e6: 1 total, 0 up, 1 in 2024-04-21T08:37:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:19 smithi136 ceph-mon[31791]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:37:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:19 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:19 smithi136 ceph-mon[31791]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-21T08:37:20.329 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:37:20.329 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:37:20.329 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-04-21T08:37:20.329 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-04-21T08:37:20.329 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113694 s, 922 MB/s 2024-04-21T08:37:20.329 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-04-21T08:37:20.637 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:20 smithi129 ceph-mon[31272]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:37:20.638 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:20 smithi129 ceph-mon[31272]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:37:20.638 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:20.638 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:20.688 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:20 smithi057 ceph-mon[27968]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:37:20.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:20 smithi057 ceph-mon[27968]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:37:20.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:20.689 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:20 smithi136 ceph-mon[31791]: from='osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:37:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:20 smithi136 ceph-mon[31791]: osdmap e7: 1 total, 0 up, 1 in 2024-04-21T08:37:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:20.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-04-21T08:37:21.204 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:37:21.567 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:37:21.567 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:37:21.567 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] boot 2024-04-21T08:37:21.567 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi057 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi057 to 17093M 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.568 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] boot 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi057 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi057 to 17093M 2024-04-21T08:37:21.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:21.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:21.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: osd.0 [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] boot 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: osdmap e8: 1 total, 1 up, 1 in 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi057 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi057 to 17093M 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:21.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:21.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:21.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:22.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:22 smithi129 ceph-mon[31272]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:37:22.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:22 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:22.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:22 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:22.559 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:22 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/619673427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec102aa0 msgr2=0x7f4bec102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/619673427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec102aa0 0x7f4bec102ec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc0099c0 tx=0x7f4bdc009cd0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/619673427 shutdown_connections 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/619673427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec102aa0 0x7f4bec102ec0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/619673427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bec104640 0x7f4bec10aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/619673427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec103ca0 0x7f4bec104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/619673427 >> 172.21.15.129:0/619673427 conn(0x7f4bec0fe040 msgr2=0x7f4bec100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/619673427 shutdown_connections 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/619673427 wait complete. 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 Processor -- start 2024-04-21T08:37:22.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.789+0000 7f4bf0bd4700 1 -- start start 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bec102aa0 0x7f4bec076790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 0x7f4bec074e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec104640 0x7f4bec075370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10e510 con 0x7f4bec104640 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10e390 con 0x7f4bec103ca0 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10e690 con 0x7f4bec102aa0 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 0x7f4bec074e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 0x7f4bec074e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60456/0 (socket says 172.21.15.129:60456) 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 -- 172.21.15.129:0/4193659237 learned_addr learned my addr 172.21.15.129:0/4193659237 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bead9d700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec104640 0x7f4bec075370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bea59c700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bec102aa0 0x7f4bec076790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 -- 172.21.15.129:0/4193659237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bec102aa0 msgr2=0x7f4bec076790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bec102aa0 0x7f4bec076790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 -- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec104640 msgr2=0x7f4bec075370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec104640 0x7f4bec075370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 -- 172.21.15.129:0/4193659237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec075c40 con 0x7f4bec103ca0 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bead9d700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec104640 0x7f4bec075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4be9d9b700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 0x7f4bec074e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4be000d950 tx=0x7f4be000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0016070 con 0x7f4bec103ca0 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1a2d70 con 0x7f4bec103ca0 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4be0010460 con 0x7f4bec103ca0 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.790+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be000f620 con 0x7f4bec103ca0 2024-04-21T08:37:22.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.791+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1a31c0 con 0x7f4bec103ca0 2024-04-21T08:37:22.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.792+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bec069730 con 0x7f4bec103ca0 2024-04-21T08:37:22.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.793+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f4be0019070 con 0x7f4bec103ca0 2024-04-21T08:37:22.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.793+0000 7f4bdb7fe700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4bd40a3bb0 0x7f4bd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:22.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.793+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f4be00ce930 con 0x7f4bec103ca0 2024-04-21T08:37:22.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.794+0000 7f4bea59c700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4bd40a3bb0 0x7f4bd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:22.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.794+0000 7f4bea59c700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4bd40a3bb0 0x7f4bd40a6070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00b580 tx=0x7f4bdc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:22.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:22.798+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4be009cd80 con 0x7f4bec103ca0 2024-04-21T08:37:22.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:22 smithi057 ceph-mon[27968]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:37:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:22 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:22 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:22 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:22 smithi136 ceph-mon[31791]: osdmap e9: 1 total, 1 up, 1 in 2024-04-21T08:37:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:22 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:22 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:22 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:23.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:23.002+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4bec062e80 con 0x7f4bd40a3bb0 2024-04-21T08:37:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:23 smithi129 ceph-mon[31272]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:23 smithi129 ceph-mon[31272]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:23.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:23 smithi057 ceph-mon[27968]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:23.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:23 smithi057 ceph-mon[27968]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:23.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:23.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:23.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:23.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:23 smithi136 ceph-mon[31791]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:23.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:23 smithi136 ceph-mon[31791]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:23.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:23.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:23.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:25.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:25 smithi136 ceph-mon[31791]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:25.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:25 smithi129 ceph-mon[31272]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:26.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:25 smithi057 ceph-mon[27968]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:26.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:26 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/236080498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]: dispatch 2024-04-21T08:37:26.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:26 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]: dispatch 2024-04-21T08:37:26.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:26 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]': finished 2024-04-21T08:37:26.883 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:26 smithi136 ceph-mon[31791]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:37:26.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:26.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:26 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/1162602893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:27.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:26 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/236080498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]: dispatch 2024-04-21T08:37:27.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:26 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]: dispatch 2024-04-21T08:37:27.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:26 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]': finished 2024-04-21T08:37:27.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:26 smithi129 ceph-mon[31272]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:37:27.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:27.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:26 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/1162602893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:26 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/236080498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]: dispatch 2024-04-21T08:37:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:26 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]: dispatch 2024-04-21T08:37:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:26 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee1d7ace-7312-4242-aa99-11fcad20dd62"}]': finished 2024-04-21T08:37:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:26 smithi057 ceph-mon[27968]: osdmap e10: 2 total, 1 up, 2 in 2024-04-21T08:37:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:26 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/1162602893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:27.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:27 smithi136 ceph-mon[31791]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:28.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:27 smithi129 ceph-mon[31272]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:28.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:27 smithi057 ceph-mon[27968]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:29 smithi129 ceph-mon[31272]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:30.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:29 smithi057 ceph-mon[27968]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:30.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:29 smithi136 ceph-mon[31791]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:31.773 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:31 smithi129 ceph-mon[31272]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:32.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:31 smithi057 ceph-mon[27968]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:32.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:31 smithi136 ceph-mon[31791]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:33.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:33 smithi129 ceph-mon[31272]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:33.800 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:37:33.800 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:33.800 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:33 smithi129 ceph-mon[31272]: Deploying daemon osd.1 on smithi129 2024-04-21T08:37:34.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:33 smithi057 ceph-mon[27968]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:34.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:37:34.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:34.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:33 smithi057 ceph-mon[27968]: Deploying daemon osd.1 on smithi129 2024-04-21T08:37:34.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:33 smithi136 ceph-mon[31791]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:34.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-21T08:37:34.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:34.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:33 smithi136 ceph-mon[31791]: Deploying daemon osd.1 on smithi129 2024-04-21T08:37:35.685 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:35 smithi129 ceph-mon[31272]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:36.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:35 smithi057 ceph-mon[27968]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:36.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:35 smithi136 ceph-mon[31791]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:37.802 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:37 smithi129 ceph-mon[31272]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:38.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:37 smithi057 ceph-mon[27968]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:38.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:37 smithi136 ceph-mon[31791]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:39 smithi129 ceph-mon[31272]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:39.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:39.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:39.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:39.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:39 smithi057 ceph-mon[27968]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:39.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:39.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:39 smithi136 ceph-mon[31791]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:41.714 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:41 smithi129 ceph-mon[31272]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:41.768 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 1 on host 'smithi129' 2024-04-21T08:37:41.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.769+0000 7f4bdb7fe700 1 -- 172.21.15.129:0/4193659237 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4bec062e80 con 0x7f4bd40a3bb0 2024-04-21T08:37:41.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4bd40a3bb0 msgr2=0x7f4bd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:41.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4bd40a3bb0 0x7f4bd40a6070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00b580 tx=0x7f4bdc005fb0 comp rx=0 tx=0).stop 2024-04-21T08:37:41.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 msgr2=0x7f4bec074e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:41.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 0x7f4bec074e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4be000d950 tx=0x7f4be000dd10 comp rx=0 tx=0).stop 2024-04-21T08:37:41.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 shutdown_connections 2024-04-21T08:37:41.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f4bd40a3bb0 0x7f4bd40a6070 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:41.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bec104640 0x7f4bec075370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:41.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bec102aa0 0x7f4bec076790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:41.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 --2- 172.21.15.129:0/4193659237 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bec103ca0 0x7f4bec074e30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:41.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.771+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 >> 172.21.15.129:0/4193659237 conn(0x7f4bec0fe040 msgr2=0x7f4bec107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:41.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.772+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 shutdown_connections 2024-04-21T08:37:41.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:37:41.772+0000 7f4bf0bd4700 1 -- 172.21.15.129:0/4193659237 wait complete. 2024-04-21T08:37:41.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:41 smithi136 ceph-mon[31791]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:42.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:41 smithi057 ceph-mon[27968]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:42.398 DEBUG:teuthology.orchestra.run.smithi129:osd.1> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.1.service 2024-04-21T08:37:42.400 INFO:tasks.cephadm:Deploying osd.2 on smithi136 with /dev/vg_nvme/lv_4... 2024-04-21T08:37:42.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-21T08:37:42.647 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:37:43.056 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.056 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.056 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.057 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.057 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:42 smithi136 ceph-mon[31791]: from='osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:43.057 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:42 smithi136 ceph-mon[31791]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:43.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:42 smithi129 ceph-mon[31272]: from='osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:43.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:42 smithi129 ceph-mon[31272]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:43.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:43.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:42 smithi057 ceph-mon[27968]: from='osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:43.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:42 smithi057 ceph-mon[27968]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-21T08:37:43.936 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:43 smithi136 ceph-mon[31791]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:43.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:43 smithi136 ceph-mon[31791]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:37:43.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:43 smithi136 ceph-mon[31791]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:37:43.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:43.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:43 smithi136 ceph-mon[31791]: from='osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:37:43.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:43 smithi136 ceph-mon[31791]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:37:44.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:43 smithi129 ceph-mon[31272]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:44.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:43 smithi129 ceph-mon[31272]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:37:44.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:43 smithi129 ceph-mon[31272]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:37:44.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:44.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:43 smithi129 ceph-mon[31272]: from='osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:37:44.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:43 smithi129 ceph-mon[31272]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:37:44.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:43 smithi057 ceph-mon[27968]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:44.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:43 smithi057 ceph-mon[27968]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-21T08:37:44.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:43 smithi057 ceph-mon[27968]: osdmap e11: 2 total, 1 up, 2 in 2024-04-21T08:37:44.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:44.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:43 smithi057 ceph-mon[27968]: from='osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:37:44.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:43 smithi057 ceph-mon[27968]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:37:44.519 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-21T08:37:44.519 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-21T08:37:44.519 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-21T08:37:44.519 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-21T08:37:44.519 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839738 s, 1.2 GB/s 2024-04-21T08:37:44.519 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-21T08:37:44.697 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1[35872]: 2024-04-21T08:37:44.433+0000 7fcfffe21700 -1 osd.1 0 waiting for initial osdmap 2024-04-21T08:37:44.697 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1[35872]: 2024-04-21T08:37:44.436+0000 7fcff7c0c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:37:45.048 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-mon[31272]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:37:45.049 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-mon[31272]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:37:45.049 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:45.049 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:45.049 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:44 smithi129 ceph-mon[31272]: from='osd.1 ' entity='osd.1' 2024-04-21T08:37:45.067 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:44 smithi136 ceph-mon[31791]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:37:45.067 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:44 smithi136 ceph-mon[31791]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:37:45.067 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:45.067 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:45.067 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:44 smithi136 ceph-mon[31791]: from='osd.1 ' entity='osd.1' 2024-04-21T08:37:45.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-04-21T08:37:45.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:44 smithi057 ceph-mon[27968]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:37:45.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:44 smithi057 ceph-mon[27968]: osdmap e12: 2 total, 1 up, 2 in 2024-04-21T08:37:45.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:45.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:45.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:44 smithi057 ceph-mon[27968]: from='osd.1 ' entity='osd.1' 2024-04-21T08:37:45.344 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:46.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] boot 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:46.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:37:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:37:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] boot 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:46.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: osd.1 [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] boot 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: osdmap e13: 2 total, 2 up, 2 in 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:37:46.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:37:46.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 -- 172.21.15.136:0/173417735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc1035a0 msgr2=0x7f7adc105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.890 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 --2- 172.21.15.136:0/173417735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc1035a0 0x7f7adc105990 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ad8009990 tx=0x7f7ad8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 -- 172.21.15.136:0/173417735 shutdown_connections 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 --2- 172.21.15.136:0/173417735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc108800 0x7f7adc10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 --2- 172.21.15.136:0/173417735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc1035a0 0x7f7adc105990 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 --2- 172.21.15.136:0/173417735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc105ed0 0x7f7adc1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 -- 172.21.15.136:0/173417735 >> 172.21.15.136:0/173417735 conn(0x7f7adc0fcf70 msgr2=0x7f7adc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 -- 172.21.15.136:0/173417735 shutdown_connections 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 -- 172.21.15.136:0/173417735 wait complete. 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.889+0000 7f7ae3800700 1 Processor -- start 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 -- start start 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc1035a0 0x7f7adc198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc105ed0 0x7f7adc199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 0x7f7adc1a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc10e680 con 0x7f7adc105ed0 2024-04-21T08:37:46.891 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc10e500 con 0x7f7adc1035a0 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae3800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc10e800 con 0x7f7adc108800 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 0x7f7adc1a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae159c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc1035a0 0x7f7adc198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 0x7f7adc1a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39724/0 (socket says 172.21.15.136:39724) 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 -- 172.21.15.136:0/2089404247 learned_addr learned my addr 172.21.15.136:0/2089404247 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae0d9b700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc105ed0 0x7f7adc199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 -- 172.21.15.136:0/2089404247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc1035a0 msgr2=0x7f7adc198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc1035a0 0x7f7adc198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 -- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc105ed0 msgr2=0x7f7adc199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc105ed0 0x7f7adc199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 -- 172.21.15.136:0/2089404247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7adc1a0a70 con 0x7f7adc108800 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae159c700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc1035a0 0x7f7adc198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.890+0000 7f7ae1d9d700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 0x7f7adc1a03d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400ba10 tx=0x7f7ad400bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.892 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.891+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4019070 con 0x7f7adc108800 2024-04-21T08:37:46.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.891+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ad400cde0 con 0x7f7adc108800 2024-04-21T08:37:46.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.891+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7adc1a0d60 con 0x7f7adc108800 2024-04-21T08:37:46.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.891+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4012340 con 0x7f7adc108800 2024-04-21T08:37:46.893 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.891+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7adc1a12c0 con 0x7f7adc108800 2024-04-21T08:37:46.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.892+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7adc069730 con 0x7f7adc108800 2024-04-21T08:37:46.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.893+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7ad401c070 con 0x7f7adc108800 2024-04-21T08:37:46.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.893+0000 7f7ad27fc700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ac80a3bb0 0x7f7ac80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:37:46.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.893+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f7ad40ce6b0 con 0x7f7adc108800 2024-04-21T08:37:46.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.894+0000 7f7ae159c700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ac80a3bb0 0x7f7ac80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:37:46.894 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.894+0000 7f7ae159c700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ac80a3bb0 0x7f7ac80a6070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800b5c0 tx=0x7f7ad8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:37:46.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:46.897+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ad4023070 con 0x7f7adc108800 2024-04-21T08:37:47.100 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:37:47.100+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7adc062e80 con 0x7f7ac80a3bb0 2024-04-21T08:37:47.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:46 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi129 2024-04-21T08:37:47.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:46 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi129 to 17088M 2024-04-21T08:37:47.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:46 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi129 2024-04-21T08:37:47.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:46 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi129 to 17088M 2024-04-21T08:37:47.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:46 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi129 2024-04-21T08:37:47.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:46 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi129 to 17088M 2024-04-21T08:37:47.970 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:47 smithi136 ceph-mon[31791]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:47.971 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:47 smithi136 ceph-mon[31791]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:37:47.971 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:47 smithi136 ceph-mon[31791]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:47.971 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:47.971 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:47.971 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:48.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:47 smithi057 ceph-mon[27968]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:48.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:47 smithi057 ceph-mon[27968]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:37:48.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:47 smithi057 ceph-mon[27968]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:48.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:48.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:48.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:47 smithi129 ceph-mon[31272]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:47 smithi129 ceph-mon[31272]: osdmap e14: 2 total, 2 up, 2 in 2024-04-21T08:37:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:47 smithi129 ceph-mon[31272]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:37:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:37:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:37:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:49 smithi136 ceph-mon[31791]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/4213961767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]: dispatch 2024-04-21T08:37:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:49 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]: dispatch 2024-04-21T08:37:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:49 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]': finished 2024-04-21T08:37:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:49 smithi136 ceph-mon[31791]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:37:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:49 smithi129 ceph-mon[31272]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/4213961767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]: dispatch 2024-04-21T08:37:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:49 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]: dispatch 2024-04-21T08:37:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:49 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]': finished 2024-04-21T08:37:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:49 smithi129 ceph-mon[31272]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:37:50.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:50.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:49 smithi057 ceph-mon[27968]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/4213961767' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]: dispatch 2024-04-21T08:37:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:49 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]: dispatch 2024-04-21T08:37:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:49 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e30fb84-6e42-4857-9e01-fecda24f1074"}]': finished 2024-04-21T08:37:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:49 smithi057 ceph-mon[27968]: osdmap e15: 3 total, 2 up, 3 in 2024-04-21T08:37:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:37:51.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:50 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/243925960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:50 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/243925960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:51.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/243925960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:37:52.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:51 smithi136 ceph-mon[31791]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:52.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:51 smithi129 ceph-mon[31272]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:52.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:51 smithi057 ceph-mon[27968]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:54.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:53 smithi136 ceph-mon[31791]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:54.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:53 smithi129 ceph-mon[31272]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:54.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:53 smithi057 ceph-mon[27968]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:56.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:55 smithi136 ceph-mon[31791]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:55 smithi129 ceph-mon[31272]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:56.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:55 smithi057 ceph-mon[27968]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:57.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:37:57.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:37:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:57.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-21T08:37:57.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:37:58.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:57 smithi136 ceph-mon[31791]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:58.068 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:57 smithi136 ceph-mon[31791]: Deploying daemon osd.2 on smithi136 2024-04-21T08:37:58.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:57 smithi129 ceph-mon[31272]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:58.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:57 smithi129 ceph-mon[31272]: Deploying daemon osd.2 on smithi136 2024-04-21T08:37:58.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:57 smithi057 ceph-mon[27968]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:37:58.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:57 smithi057 ceph-mon[27968]: Deploying daemon osd.2 on smithi136 2024-04-21T08:38:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:37:59 smithi136 ceph-mon[31791]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:00.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:37:59 smithi129 ceph-mon[31272]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:00.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:37:59 smithi057 ceph-mon[27968]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:02.018 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:01 smithi136 ceph-mon[31791]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:01 smithi129 ceph-mon[31272]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:02.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:01 smithi057 ceph-mon[27968]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:03.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:03.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:03.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:03.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:03.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:03.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:03.359 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:03.359 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:03.359 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:04.320 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:04 smithi136 ceph-mon[31791]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:04.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:04 smithi129 ceph-mon[31272]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:04.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:04 smithi057 ceph-mon[27968]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:05.316 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.316+0000 7f7ad27fc700 1 -- 172.21.15.136:0/2089404247 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7adc062e80 con 0x7f7ac80a3bb0 2024-04-21T08:38:05.316 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 2 on host 'smithi136' 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ac80a3bb0 msgr2=0x7f7ac80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ac80a3bb0 0x7f7ac80a6070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800b5c0 tx=0x7f7ad8005fb0 comp rx=0 tx=0).stop 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 msgr2=0x7f7adc1a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 0x7f7adc1a03d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400ba10 tx=0x7f7ad400bd20 comp rx=0 tx=0).stop 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 shutdown_connections 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ac80a3bb0 0x7f7ac80a6070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7adc1035a0 0x7f7adc198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:05.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7adc108800 0x7f7adc1a03d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:05.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 --2- 172.21.15.136:0/2089404247 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7adc105ed0 0x7f7adc199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:05.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 >> 172.21.15.136:0/2089404247 conn(0x7f7adc0fcf70 msgr2=0x7f7adc104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:05.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 shutdown_connections 2024-04-21T08:38:05.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:38:05.318+0000 7f7ae3800700 1 -- 172.21.15.136:0/2089404247 wait complete. 2024-04-21T08:38:05.915 DEBUG:teuthology.orchestra.run.smithi136:osd.2> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.2.service 2024-04-21T08:38:05.917 INFO:tasks.cephadm:Deploying osd.3 on smithi057 with /dev/vg_nvme/lv_3... 2024-04-21T08:38:05.917 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:38:06.173 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:06.242 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:05 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2[36946]: 2024-04-21T08:38:05.990+0000 7eff4267c7c0 -1 osd.2 0 log_to_monitors true 2024-04-21T08:38:06.549 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:06.550 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.550 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.550 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.550 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.550 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: from='osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:38:06.550 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:06 smithi057 ceph-mon[27968]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:38:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: from='osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:38:06.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:06 smithi129 ceph-mon[31272]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: from='osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:38:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:06 smithi136 ceph-mon[31791]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-21T08:38:07.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-mon[31791]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:38:07.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-mon[31791]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:38:07.453 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:07.454 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-mon[31791]: from='osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:38:07.454 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-mon[31791]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:38:07.454 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2[36946]: 2024-04-21T08:38:07.356+0000 7eff37ce1700 -1 osd.2 0 waiting for initial osdmap 2024-04-21T08:38:07.454 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:07 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2[36946]: 2024-04-21T08:38:07.360+0000 7eff33ad4700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:38:07.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:07 smithi057 ceph-mon[27968]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:38:07.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:07 smithi057 ceph-mon[27968]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:38:07.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:07.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:07 smithi057 ceph-mon[27968]: from='osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:38:07.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:07 smithi057 ceph-mon[27968]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:38:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:07 smithi129 ceph-mon[31272]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-21T08:38:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:07 smithi129 ceph-mon[31272]: osdmap e16: 3 total, 2 up, 3 in 2024-04-21T08:38:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:07 smithi129 ceph-mon[31272]: from='osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:38:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:07 smithi129 ceph-mon[31272]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:38:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:38:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:38:08.332 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2024-04-21T08:38:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2024-04-21T08:38:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0166992 s, 628 MB/s 2024-04-21T08:38:08.332 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2024-04-21T08:38:08.535 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:08 smithi136 ceph-mon[31791]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:08.535 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:08 smithi136 ceph-mon[31791]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:38:08.536 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:08 smithi136 ceph-mon[31791]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:38:08.536 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:08 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:08.536 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:08 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:08 smithi057 ceph-mon[27968]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:08 smithi057 ceph-mon[27968]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:38:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:08 smithi057 ceph-mon[27968]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:38:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:08 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:08 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:08 smithi129 ceph-mon[31272]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:08 smithi129 ceph-mon[31272]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:38:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:08 smithi129 ceph-mon[31272]: osdmap e17: 3 total, 2 up, 3 in 2024-04-21T08:38:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:08 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:08 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:08.997 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi057:vg_nvme/lv_3 2024-04-21T08:38:09.244 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] boot 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:09.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:09 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] boot 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:09.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:09 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:38:09.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:38:09.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:09.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: osd.2 [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] boot 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: osdmap e18: 3 total, 3 up, 3 in 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:09.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:09.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:09 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:10.652 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40549]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:38:10.653 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40549]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.653 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40549]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.653 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40549]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 ceph-mon[31791]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:10.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi136 2024-04-21T08:38:10.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi136 to 17088M 2024-04-21T08:38:10.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:38:10.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 ceph-mon[31791]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:38:10.653 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:38:10.703 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 ceph-mon[27968]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:10.703 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi136 2024-04-21T08:38:10.703 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi136 to 17088M 2024-04-21T08:38:10.703 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:38:10.703 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 ceph-mon[27968]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:38:10.703 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47434]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47434]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47434]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47434]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47437]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47437]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47437]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.704 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47437]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 ceph-mon[31272]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-21T08:38:10.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi136 2024-04-21T08:38:10.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi136 to 17088M 2024-04-21T08:38:10.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-21T08:38:10.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 ceph-mon[31272]: osdmap e19: 3 total, 3 up, 3 in 2024-04-21T08:38:10.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39701]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39701]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39701]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39704]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39704]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.781 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39704]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 -- 172.21.15.57:0/1596306632 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee81087b0 msgr2=0x7efee810ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:10.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 --2- 172.21.15.57:0/1596306632 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee81087b0 0x7efee810ac20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efee000b0a0 tx=0x7efee000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:38:10.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 -- 172.21.15.57:0/1596306632 shutdown_connections 2024-04-21T08:38:10.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 --2- 172.21.15.57:0/1596306632 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee81087b0 0x7efee810ac20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:10.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 --2- 172.21.15.57:0/1596306632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efee8103550 0x7efee8105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:10.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 --2- 172.21.15.57:0/1596306632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efee8105e80 0x7efee8108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 -- 172.21.15.57:0/1596306632 >> 172.21.15.57:0/1596306632 conn(0x7efee80fcfa0 msgr2=0x7efee80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 -- 172.21.15.57:0/1596306632 shutdown_connections 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 -- 172.21.15.57:0/1596306632 wait complete. 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.948+0000 7efeed558700 1 Processor -- start 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 -- start start 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 0x7efee8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efee8105e80 0x7efee8199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efee81087b0 0x7efee81a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efee810e520 con 0x7efee8103550 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efee810e3a0 con 0x7efee8105e80 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efeed558700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efee810e6a0 con 0x7efee81087b0 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 0x7efee8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 0x7efee8198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51688/0 (socket says 172.21.15.57:51688) 2024-04-21T08:38:10.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 -- 172.21.15.57:0/3995601650 learned_addr learned my addr 172.21.15.57:0/3995601650 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:38:10.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee67fc700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efee8105e80 0x7efee8199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:10.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee77fe700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efee81087b0 0x7efee81a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 -- 172.21.15.57:0/3995601650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efee81087b0 msgr2=0x7efee81a00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efee81087b0 0x7efee81a00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 -- 172.21.15.57:0/3995601650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efee8105e80 msgr2=0x7efee8199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efee8105e80 0x7efee8199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 -- 172.21.15.57:0/3995601650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee81a0740 con 0x7efee8103550 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efee6ffd700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 0x7efee8198ae0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efed800d9b0 tx=0x7efed800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.949+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8016070 con 0x7efee8103550 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.950+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee81a0a30 con 0x7efee8103550 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.950+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efed8004830 con 0x7efee8103550 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.950+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee81a0fc0 con 0x7efee8103550 2024-04-21T08:38:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.950+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8005440 con 0x7efee8103550 2024-04-21T08:38:10.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.950+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed4005360 con 0x7efee8103550 2024-04-21T08:38:10.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.952+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7efed80055e0 con 0x7efee8103550 2024-04-21T08:38:10.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.953+0000 7efecffff700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efed00a3bb0 0x7efed00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:10.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.953+0000 7efee67fc700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efed00a3bb0 0x7efed00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:10.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.953+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7efed80cf200 con 0x7efee8103550 2024-04-21T08:38:10.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.954+0000 7efee67fc700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efed00a3bb0 0x7efed00a6070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efedc00abd0 tx=0x7efedc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:10.953 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47441]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:38:10.953 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47441]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.953 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47441]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.953 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47441]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.953 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47447]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:38:10.953 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47447]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:10.954 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47447]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:10.954 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:10 smithi057 sudo[47447]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:10.957+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efed801f020 con 0x7efee8103550 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39707]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39707]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39707]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:10 smithi129 sudo[39710]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:11.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40552]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:38:11.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40552]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:11.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40552]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:11.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40552]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:11.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40555]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-21T08:38:11.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40555]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:11.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40555]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:11.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40555]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:11.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40558]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-21T08:38:11.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40558]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-21T08:38:11.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40558]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-21T08:38:11.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:10 smithi136 sudo[40558]: pam_unix(sudo:session): session closed for user root 2024-04-21T08:38:11.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:11.174+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7efed4000bf0 con 0x7efed00a3bb0 2024-04-21T08:38:11.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:11.570+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efed8098f60 con 0x7efee8103550 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.812 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:11.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:11 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:11.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:11.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:11 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: osdmap e20: 3 total, 3 up, 3 in 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-21T08:38:11.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:11.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:11.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:11 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:12.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:12 smithi057 ceph-mon[27968]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:12.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:12 smithi057 ceph-mon[27968]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:12.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:12 smithi057 ceph-mon[27968]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:38:12.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:12 smithi057 ceph-mon[27968]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:38:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:12 smithi129 ceph-mon[31272]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:12 smithi129 ceph-mon[31272]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:12 smithi129 ceph-mon[31272]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:38:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:12 smithi129 ceph-mon[31272]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:38:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:12 smithi136 ceph-mon[31791]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:12 smithi136 ceph-mon[31791]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:12 smithi136 ceph-mon[31791]: osdmap e21: 3 total, 3 up, 3 in 2024-04-21T08:38:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:12 smithi136 ceph-mon[31791]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-21T08:38:13.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:13 smithi129 ceph-mon[31272]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:13.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:13 smithi057 ceph-mon[27968]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:13.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:13 smithi136 ceph-mon[31791]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2596996974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcf84be5-1210-4e75-86ed-4aac7a0d4dd4"}]: dispatch 2024-04-21T08:38:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2596996974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcf84be5-1210-4e75-86ed-4aac7a0d4dd4"}]': finished 2024-04-21T08:38:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:14 smithi057 ceph-mon[27968]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:38:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2596996974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcf84be5-1210-4e75-86ed-4aac7a0d4dd4"}]: dispatch 2024-04-21T08:38:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2596996974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcf84be5-1210-4e75-86ed-4aac7a0d4dd4"}]': finished 2024-04-21T08:38:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:14 smithi129 ceph-mon[31272]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:38:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2596996974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bcf84be5-1210-4e75-86ed-4aac7a0d4dd4"}]: dispatch 2024-04-21T08:38:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2596996974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bcf84be5-1210-4e75-86ed-4aac7a0d4dd4"}]': finished 2024-04-21T08:38:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:14 smithi136 ceph-mon[31791]: osdmap e22: 4 total, 3 up, 4 in 2024-04-21T08:38:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:15.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:15 smithi129 ceph-mon[31272]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:15.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2279649950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:15.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:15 smithi057 ceph-mon[27968]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:15.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2279649950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:15 smithi136 ceph-mon[31791]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2279649950' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:17.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:17 smithi136 ceph-mon[31791]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:17.963 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:17 smithi057 ceph-mon[27968]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:18.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:17 smithi129 ceph-mon[31272]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:19.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:19 smithi136 ceph-mon[31791]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:19.910 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:19 smithi057 ceph-mon[27968]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:20.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:19 smithi129 ceph-mon[31272]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:21.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:21 smithi136 ceph-mon[31791]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:38:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:21 smithi136 ceph-mon[31791]: Deploying daemon osd.3 on smithi057 2024-04-21T08:38:22.039 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:21 smithi057 ceph-mon[27968]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:22.039 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:38:22.039 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:22.040 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:21 smithi057 ceph-mon[27968]: Deploying daemon osd.3 on smithi057 2024-04-21T08:38:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:21 smithi129 ceph-mon[31272]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-21T08:38:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:21 smithi129 ceph-mon[31272]: Deploying daemon osd.3 on smithi057 2024-04-21T08:38:23.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:23 smithi136 ceph-mon[31791]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:24.048 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:23 smithi057 ceph-mon[27968]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:24.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:23 smithi129 ceph-mon[31272]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:25 smithi136 ceph-mon[31791]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:25 smithi129 ceph-mon[31272]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:25 smithi057 ceph-mon[27968]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:28.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:28 smithi057 ceph-mon[27968]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:28.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:28.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:28.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:28.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:28 smithi136 ceph-mon[31791]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:28 smithi129 ceph-mon[31272]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:30 smithi136 ceph-mon[31791]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:30 smithi129 ceph-mon[31272]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:30 smithi057 ceph-mon[27968]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:30.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.986+0000 7efecffff700 1 -- 172.21.15.57:0/3995601650 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efed4000bf0 con 0x7efed00a3bb0 2024-04-21T08:38:30.986 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 3 on host 'smithi057' 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.988+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efed00a3bb0 msgr2=0x7efed00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.988+0000 7efeed558700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efed00a3bb0 0x7efed00a6070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efedc00abd0 tx=0x7efedc00a380 comp rx=0 tx=0).stop 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.988+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 msgr2=0x7efee8198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.988+0000 7efeed558700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 0x7efee8198ae0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efed800d9b0 tx=0x7efed800dcc0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 shutdown_connections 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efed00a3bb0 0x7efed00a6070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efee81087b0 0x7efee81a00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efee8103550 0x7efee8198ae0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 --2- 172.21.15.57:0/3995601650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efee8105e80 0x7efee8199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:30.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 >> 172.21.15.57:0/3995601650 conn(0x7efee80fcfa0 msgr2=0x7efee8103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:30.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 shutdown_connections 2024-04-21T08:38:30.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:38:30.989+0000 7efeed558700 1 -- 172.21.15.57:0/3995601650 wait complete. 2024-04-21T08:38:31.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:31 smithi057 ceph-mon[27968]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:38:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:31 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:31 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:31 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:31.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:31 smithi136 ceph-mon[31791]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:38:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:31 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:31 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:31 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:31.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:31 smithi129 ceph-mon[31272]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-21T08:38:31.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:31 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:31.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:31 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:31.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:31 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:31.709 DEBUG:teuthology.orchestra.run.smithi057:osd.3> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.3.service 2024-04-21T08:38:31.712 INFO:tasks.cephadm:Deploying osd.4 on smithi129 with /dev/vg_nvme/lv_3... 2024-04-21T08:38:31.712 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:38:31.957 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:38:32.286 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-mon[27968]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:32.286 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-mon[27968]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:38:32.286 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-mon[27968]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:38:32.286 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-mon[27968]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:38:32.286 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:32.286 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3[49735]: 2024-04-21T08:38:32.108+0000 7f67a3df0700 -1 osd.3 0 waiting for initial osdmap 2024-04-21T08:38:32.286 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:38:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3[49735]: 2024-04-21T08:38:32.114+0000 7f679fbe3700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:38:32.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:32 smithi129 ceph-mon[31272]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:32.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:32 smithi129 ceph-mon[31272]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:38:32.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:32 smithi129 ceph-mon[31272]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:38:32.376 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:32 smithi129 ceph-mon[31272]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:38:32.376 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:32 smithi136 ceph-mon[31791]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:32 smithi136 ceph-mon[31791]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-21T08:38:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:32 smithi136 ceph-mon[31791]: osdmap e23: 4 total, 3 up, 4 in 2024-04-21T08:38:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:32 smithi136 ceph-mon[31791]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:38:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:33 smithi057 ceph-mon[27968]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:38:33.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:33 smithi057 ceph-mon[27968]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:38:33.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.306 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:33 smithi129 ceph-mon[31272]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:38:33.306 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:33 smithi129 ceph-mon[31272]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:38:33.306 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.306 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:33 smithi136 ceph-mon[31791]: from='osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:38:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:33 smithi136 ceph-mon[31791]: osdmap e24: 4 total, 3 up, 4 in 2024-04-21T08:38:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:33.901 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:38:33.901 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:38:33.901 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-04-21T08:38:33.901 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-04-21T08:38:33.901 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128989 s, 813 MB/s 2024-04-21T08:38:33.901 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] boot 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:38:34.202 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] boot 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:38:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: osd.3 [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] boot 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: osdmap e25: 4 total, 4 up, 4 in 2024-04-21T08:38:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:38:34.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-04-21T08:38:34.753 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:38:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:35.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:35.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: osdmap e26: 4 total, 4 up, 4 in 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:35.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:36.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:36 smithi129 ceph-mon[31272]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:36.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:36 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi057 2024-04-21T08:38:36.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:36 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi057 to 8546M 2024-04-21T08:38:36.256 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:36 smithi129 ceph-mon[31272]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:38:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:36 smithi136 ceph-mon[31791]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:36 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi057 2024-04-21T08:38:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:36 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi057 to 8546M 2024-04-21T08:38:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:36 smithi136 ceph-mon[31791]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:38:36.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 -- 172.21.15.129:0/3560394068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 msgr2=0x7f18a8104100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 --2- 172.21.15.129:0/3560394068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8104100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f18a40099c0 tx=0x7f18a4009cd0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 -- 172.21.15.129:0/3560394068 shutdown_connections 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 --2- 172.21.15.129:0/3560394068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a8104640 0x7f18a810aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 --2- 172.21.15.129:0/3560394068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 0x7f18a8102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 --2- 172.21.15.129:0/3560394068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8104100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.469+0000 7f18b0501700 1 -- 172.21.15.129:0/3560394068 >> 172.21.15.129:0/3560394068 conn(0x7f18a80fe040 msgr2=0x7f18a8100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 -- 172.21.15.129:0/3560394068 shutdown_connections 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 -- 172.21.15.129:0/3560394068 wait complete. 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 Processor -- start 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 -- start start 2024-04-21T08:38:36.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 0x7f18a8198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a8104640 0x7f18a81a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810e510 con 0x7f18a8102aa0 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810e390 con 0x7f18a8103ca0 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18b0501700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810e690 con 0x7f18a8104640 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18ada9c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18ada9c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55508/0 (socket says 172.21.15.129:55508) 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.470+0000 7f18ada9c700 1 -- 172.21.15.129:0/1723870783 learned_addr learned my addr 172.21.15.129:0/1723870783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18aea9e700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a8104640 0x7f18a81a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ae29d700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 0x7f18a8198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ada9c700 1 -- 172.21.15.129:0/1723870783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a8104640 msgr2=0x7f18a81a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ada9c700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a8104640 0x7f18a81a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ada9c700 1 -- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 msgr2=0x7f18a8198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ada9c700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 0x7f18a8198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ada9c700 1 -- 172.21.15.129:0/1723870783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a81a08a0 con 0x7f18a8103ca0 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ae29d700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 0x7f18a8198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18ada9c700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8199190 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18a4003790 tx=0x7f18a4005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4027070 con 0x7f18a8103ca0 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18a4004160 con 0x7f18a8103ca0 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a81a0b30 con 0x7f18a8103ca0 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4022d90 con 0x7f18a8103ca0 2024-04-21T08:38:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.471+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a81a1060 con 0x7f18a8103ca0 2024-04-21T08:38:36.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.472+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a8069730 con 0x7f18a8103ca0 2024-04-21T08:38:36.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.473+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f18a402a070 con 0x7f18a8103ca0 2024-04-21T08:38:36.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.474+0000 7f189b7fe700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18940a3cc0 0x7f18940a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:38:36.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.474+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f18a40cfe00 con 0x7f18a8103ca0 2024-04-21T08:38:36.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.474+0000 7f18ae29d700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18940a3cc0 0x7f18940a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:38:36.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.475+0000 7f18ae29d700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18940a3cc0 0x7f18940a6180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f189c005950 tx=0x7f189c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:38:36.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.479+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18a4013070 con 0x7f18a8103ca0 2024-04-21T08:38:36.491 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:36 smithi057 ceph-mon[27968]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:36.491 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:36 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi057 2024-04-21T08:38:36.491 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:36 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi057 to 8546M 2024-04-21T08:38:36.491 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:36 smithi057 ceph-mon[27968]: osdmap e27: 4 total, 4 up, 4 in 2024-04-21T08:38:36.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:36.684+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f18a8062e80 con 0x7f18940a3cc0 2024-04-21T08:38:37.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:37 smithi136 ceph-mon[31791]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:38:37.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:37.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:37.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:37.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:37 smithi129 ceph-mon[31272]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:38:37.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:37.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:37.426 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:37.502 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:37 smithi057 ceph-mon[27968]: osdmap e28: 4 total, 4 up, 4 in 2024-04-21T08:38:37.503 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:38:37.503 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:38:37.503 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:38.279 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:38 smithi129 ceph-mon[31272]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:38:38.280 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:38 smithi129 ceph-mon[31272]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:38.280 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:38 smithi129 ceph-mon[31272]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:38:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:38 smithi057 ceph-mon[27968]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:38:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:38 smithi057 ceph-mon[27968]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:38 smithi057 ceph-mon[27968]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:38:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:38 smithi136 ceph-mon[31791]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:38:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:38 smithi136 ceph-mon[31791]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:38:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:38 smithi136 ceph-mon[31791]: osdmap e29: 4 total, 4 up, 4 in 2024-04-21T08:38:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-21T08:38:40.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/3289231336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]: dispatch 2024-04-21T08:38:40.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]: dispatch 2024-04-21T08:38:40.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]': finished 2024-04-21T08:38:40.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:38:40.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:40.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:40 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/2952249855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/3289231336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]: dispatch 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]: dispatch 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]': finished 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:40 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/2952249855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: pgmap v106: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/3289231336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]: dispatch 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]: dispatch 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2797f86-8e4f-4d66-ba17-dcd90bb6869e"}]': finished 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: osdmap e30: 5 total, 4 up, 5 in 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:40 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/2952249855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:38:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:42 smithi129 ceph-mon[31272]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-21T08:38:42.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:42 smithi057 ceph-mon[27968]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-21T08:38:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:42 smithi136 ceph-mon[31791]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-04-21T08:38:44.415 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:44 smithi129 ceph-mon[31272]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:44.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:44 smithi057 ceph-mon[27968]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:44.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:44 smithi136 ceph-mon[31791]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:46 smithi129 ceph-mon[31272]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:46.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:46 smithi057 ceph-mon[27968]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:46.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:46 smithi136 ceph-mon[31791]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:47.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:38:47.446 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:47.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:38:47.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:47.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-21T08:38:47.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:48 smithi129 ceph-mon[31272]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:48 smithi129 ceph-mon[31272]: Deploying daemon osd.4 on smithi129 2024-04-21T08:38:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:48 smithi057 ceph-mon[27968]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:48 smithi057 ceph-mon[27968]: Deploying daemon osd.4 on smithi129 2024-04-21T08:38:48.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:48 smithi136 ceph-mon[31791]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:48 smithi136 ceph-mon[31791]: Deploying daemon osd.4 on smithi129 2024-04-21T08:38:50.507 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:50 smithi129 ceph-mon[31272]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:50 smithi057 ceph-mon[27968]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:50.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:50 smithi136 ceph-mon[31791]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:52 smithi129 ceph-mon[31272]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:52 smithi057 ceph-mon[27968]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:52.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:52 smithi136 ceph-mon[31791]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:53.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:53.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:53.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:53.687 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:53.687 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:53.687 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:53.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:53.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:53.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:54.549 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:54 smithi129 ceph-mon[31272]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:54 smithi136 ceph-mon[31791]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:54 smithi057 ceph-mon[27968]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:56.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.101+0000 7f189b7fe700 1 -- 172.21.15.129:0/1723870783 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f18a8062e80 con 0x7f18940a3cc0 2024-04-21T08:38:56.102 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 4 on host 'smithi129' 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18940a3cc0 msgr2=0x7f18940a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18940a3cc0 0x7f18940a6180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f189c005950 tx=0x7f189c0058e0 comp rx=0 tx=0).stop 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 msgr2=0x7f18a8199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8199190 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f18a4003790 tx=0x7f18a4005b00 comp rx=0 tx=0).stop 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 shutdown_connections 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18940a3cc0 0x7f18940a6180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a8104640 0x7f18a81a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:56.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18a8102aa0 0x7f18a8198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:56.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 --2- 172.21.15.129:0/1723870783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18a8103ca0 0x7f18a8199190 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:38:56.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 >> 172.21.15.129:0/1723870783 conn(0x7f18a80fe040 msgr2=0x7f18a8107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:38:56.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 shutdown_connections 2024-04-21T08:38:56.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:38:56.104+0000 7f18b0501700 1 -- 172.21.15.129:0/1723870783 wait complete. 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:56.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:38:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:38:56.763 DEBUG:teuthology.orchestra.run.smithi129:osd.4> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.4.service 2024-04-21T08:38:56.766 INFO:tasks.cephadm:Deploying osd.5 on smithi136 with /dev/vg_nvme/lv_3... 2024-04-21T08:38:56.766 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-21T08:38:57.010 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:38:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:57 smithi129 ceph-mon[31272]: from='osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:57 smithi129 ceph-mon[31272]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:57.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:57 smithi136 ceph-mon[31791]: from='osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:57.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:57 smithi136 ceph-mon[31791]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:57.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:57 smithi057 ceph-mon[27968]: from='osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:57.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:57 smithi057 ceph-mon[27968]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-21T08:38:58.421 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:58 smithi136 ceph-mon[31791]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:58.421 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:58 smithi136 ceph-mon[31791]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:38:58.421 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:58 smithi136 ceph-mon[31791]: from='osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:38:58.421 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:58 smithi136 ceph-mon[31791]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:38:58.422 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:58.422 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:58 smithi136 ceph-mon[31791]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:38:58.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-mon[31272]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:58.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-mon[31272]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:38:58.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-mon[31272]: from='osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:38:58.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-mon[31272]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:38:58.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:58.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-mon[31272]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:38:58.574 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4[42512]: 2024-04-21T08:38:58.231+0000 7f97dfe0f700 -1 osd.4 0 waiting for initial osdmap 2024-04-21T08:38:58.574 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:38:58 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4[42512]: 2024-04-21T08:38:58.239+0000 7f97dbc02700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:38:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:58 smithi057 ceph-mon[27968]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:38:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:58 smithi057 ceph-mon[27968]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-21T08:38:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:58 smithi057 ceph-mon[27968]: from='osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:38:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:58 smithi057 ceph-mon[27968]: osdmap e31: 5 total, 4 up, 5 in 2024-04-21T08:38:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:58 smithi057 ceph-mon[27968]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:38:59.116 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-21T08:38:59.116 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-21T08:38:59.116 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-21T08:38:59.116 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-21T08:38:59.116 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00814661 s, 1.3 GB/s 2024-04-21T08:38:59.116 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-21T08:38:59.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:59 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:38:59.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:59 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:38:59.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:59 smithi129 ceph-mon[31272]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:38:59.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:59 smithi129 ceph-mon[31272]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:38:59.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:59.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:38:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:59.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:59 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:38:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:59 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:38:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:59 smithi136 ceph-mon[31791]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:38:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:59 smithi136 ceph-mon[31791]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:38:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:38:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:59.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:59 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:38:59.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:59 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:38:59.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:59 smithi057 ceph-mon[27968]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:38:59.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:59 smithi057 ceph-mon[27968]: osdmap e32: 5 total, 4 up, 5 in 2024-04-21T08:38:59.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:59.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:38:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:38:59.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-04-21T08:39:00.090 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:39:00.514 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:39:00.514 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] boot 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:00.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:00 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:39:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] boot 2024-04-21T08:39:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:00.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:00 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: osd.4 [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] boot 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: osdmap e33: 5 total, 5 up, 5 in 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:00 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:01.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:01 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi129 2024-04-21T08:39:01.488 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:01 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi129 to 8544M 2024-04-21T08:39:01.489 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:01 smithi136 ceph-mon[31791]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:39:01.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:01 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi129 2024-04-21T08:39:01.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:01 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi129 to 8544M 2024-04-21T08:39:01.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:01 smithi129 ceph-mon[31272]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:39:01.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:01 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi129 2024-04-21T08:39:01.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:01 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi129 to 8544M 2024-04-21T08:39:01.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:01 smithi057 ceph-mon[27968]: osdmap e34: 5 total, 5 up, 5 in 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 -- 172.21.15.136:0/1484917791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0102aa0 msgr2=0x7f6bb0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 --2- 172.21.15.136:0/1484917791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0102aa0 0x7f6bb0102ec0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009990 tx=0x7f6ba4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 -- 172.21.15.136:0/1484917791 shutdown_connections 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 --2- 172.21.15.136:0/1484917791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bb0104640 0x7f6bb010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 --2- 172.21.15.136:0/1484917791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0102aa0 0x7f6bb0102ec0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 --2- 172.21.15.136:0/1484917791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 -- 172.21.15.136:0/1484917791 >> 172.21.15.136:0/1484917791 conn(0x7f6bb00fe040 msgr2=0x7f6bb0100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 -- 172.21.15.136:0/1484917791 shutdown_connections 2024-04-21T08:39:01.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.707+0000 7f6bb7087700 1 -- 172.21.15.136:0/1484917791 wait complete. 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 Processor -- start 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 -- start start 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bb0102aa0 0x7f6bb0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bb0104640 0x7f6bb01a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb010e510 con 0x7f6bb0104640 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb010e390 con 0x7f6bb0102aa0 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb7087700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb010e690 con 0x7f6bb0103ca0 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:52068/0 (socket says 172.21.15.136:52068) 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 -- 172.21.15.136:0/742253654 learned_addr learned my addr 172.21.15.136:0/742253654 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:39:01.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb5624700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bb0104640 0x7f6bb01a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6bb4e23700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bb0102aa0 0x7f6bb0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 -- 172.21.15.136:0/742253654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bb0102aa0 msgr2=0x7f6bb0198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bb0102aa0 0x7f6bb0198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 -- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bb0104640 msgr2=0x7f6bb01a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bb0104640 0x7f6bb01a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 -- 172.21.15.136:0/742253654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb01a0860 con 0x7f6bb0103ca0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.708+0000 7f6baffff700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0199150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00d950 tx=0x7f6b9c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.709+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c016070 con 0x7f6bb0103ca0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.709+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b9c00fd40 con 0x7f6bb0103ca0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.709+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb01a0b50 con 0x7f6bb0103ca0 2024-04-21T08:39:01.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.709+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00e3b0 con 0x7f6bb0103ca0 2024-04-21T08:39:01.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.709+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb01a10e0 con 0x7f6bb0103ca0 2024-04-21T08:39:01.712 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.710+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb0069730 con 0x7f6bb0103ca0 2024-04-21T08:39:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.711+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6b9c019070 con 0x7f6bb0103ca0 2024-04-21T08:39:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.711+0000 7f6badffb700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6ba00a3cc0 0x7f6ba00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.711+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f6b9c0cf140 con 0x7f6bb0103ca0 2024-04-21T08:39:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.711+0000 7f6bb4e23700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6ba00a3cc0 0x7f6ba00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:01.713 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.712+0000 7f6bb4e23700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6ba00a3cc0 0x7f6ba00a6180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4003790 tx=0x7f6ba4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:01.715 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.716+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b9c09cb90 con 0x7f6bb0103ca0 2024-04-21T08:39:01.924 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:01.925+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6bb0062e80 con 0x7f6ba00a3cc0 2024-04-21T08:39:02.533 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:02 smithi136 ceph-mon[31791]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:02.534 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:02 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:02.534 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:02 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:02.534 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:02 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:02 smithi129 ceph-mon[31272]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:02 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:02 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:02 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:02 smithi057 ceph-mon[27968]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:02 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:02 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:02 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:03.526 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:03 smithi136 ceph-mon[31791]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:03.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:03 smithi129 ceph-mon[31272]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:03.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:03 smithi057 ceph-mon[27968]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:04.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:04 smithi129 ceph-mon[31272]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:04.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:04 smithi057 ceph-mon[27968]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:04.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:04 smithi136 ceph-mon[31791]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:05 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/1038725479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]: dispatch 2024-04-21T08:39:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:05 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]: dispatch 2024-04-21T08:39:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:05 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]': finished 2024-04-21T08:39:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:05 smithi129 ceph-mon[31272]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:39:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:05 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:05 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/2900659770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:05.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:05 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/1038725479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]: dispatch 2024-04-21T08:39:05.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:05 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]: dispatch 2024-04-21T08:39:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:05 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]': finished 2024-04-21T08:39:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:05 smithi057 ceph-mon[27968]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:39:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:05 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:05 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/2900659770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:05.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:05 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/1038725479' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]: dispatch 2024-04-21T08:39:05.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:05 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]: dispatch 2024-04-21T08:39:05.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:05 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4b316af-e19a-41f1-b552-3e6aa0310bb8"}]': finished 2024-04-21T08:39:05.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:05 smithi136 ceph-mon[31791]: osdmap e35: 6 total, 5 up, 6 in 2024-04-21T08:39:05.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:05 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:05.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:05 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/2900659770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:06 smithi129 ceph-mon[31272]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:06.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:06 smithi057 ceph-mon[27968]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:06.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:06 smithi136 ceph-mon[31791]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:08.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:08 smithi129 ceph-mon[31272]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:08.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:08 smithi057 ceph-mon[27968]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:08.599 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:08 smithi136 ceph-mon[31791]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:10.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:10 smithi136 ceph-mon[31791]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:10 smithi129 ceph-mon[31272]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:10.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:10 smithi057 ceph-mon[27968]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:12 smithi129 ceph-mon[31272]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:39:12.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:12 smithi057 ceph-mon[27968]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:39:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:12.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:12 smithi136 ceph-mon[31791]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:12.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-21T08:39:12.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:13.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:13 smithi129 ceph-mon[31272]: Deploying daemon osd.5 on smithi136 2024-04-21T08:39:13.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:13 smithi057 ceph-mon[27968]: Deploying daemon osd.5 on smithi136 2024-04-21T08:39:13.610 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:13 smithi136 ceph-mon[31791]: Deploying daemon osd.5 on smithi136 2024-04-21T08:39:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:14 smithi129 ceph-mon[31272]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:14.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:14 smithi057 ceph-mon[27968]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:14.626 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:14 smithi136 ceph-mon[31791]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:16 smithi129 ceph-mon[31272]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:16.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:16 smithi057 ceph-mon[27968]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:16.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:16 smithi136 ceph-mon[31791]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:18 smithi129 ceph-mon[31272]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:18.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:18 smithi057 ceph-mon[27968]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:18.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:18 smithi136 ceph-mon[31791]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:18.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:18.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:18.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:20.332 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:20 smithi136 ceph-mon[31791]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:20 smithi129 ceph-mon[31272]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:20.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:20 smithi057 ceph-mon[27968]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:21.651 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 5 on host 'smithi136' 2024-04-21T08:39:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.651+0000 7f6badffb700 1 -- 172.21.15.136:0/742253654 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6bb0062e80 con 0x7f6ba00a3cc0 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6ba00a3cc0 msgr2=0x7f6ba00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6ba00a3cc0 0x7f6ba00a6180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4003790 tx=0x7f6ba4005fb0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 msgr2=0x7f6bb0199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0199150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00d950 tx=0x7f6b9c00dc60 comp rx=0 tx=0).stop 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 shutdown_connections 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6ba00a3cc0 0x7f6ba00a6180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bb0102aa0 0x7f6bb0198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bb0104640 0x7f6bb01a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 --2- 172.21.15.136:0/742253654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bb0103ca0 0x7f6bb0199150 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 >> 172.21.15.136:0/742253654 conn(0x7f6bb00fe040 msgr2=0x7f6bb0107110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 shutdown_connections 2024-04-21T08:39:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:39:21.654+0000 7f6bb7087700 1 -- 172.21.15.136:0/742253654 wait complete. 2024-04-21T08:39:21.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:21.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:21 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:39:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:39:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:21 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:21 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:22.342 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.5.service 2024-04-21T08:39:22.345 INFO:tasks.cephadm:Deploying osd.6 on smithi057 with /dev/vg_nvme/lv_2... 2024-04-21T08:39:22.345 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:39:22.602 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:23 smithi057 ceph-mon[27968]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:39:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:23 smithi057 ceph-mon[27968]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:39:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:23 smithi057 ceph-mon[27968]: from='osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:39:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:23 smithi057 ceph-mon[27968]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:39:23.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-mon[31791]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:39:23.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-mon[31791]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:39:23.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:23.597 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-mon[31791]: from='osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:39:23.598 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-mon[31791]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:39:23.598 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:39:23.340+0000 7f332feaa700 -1 osd.5 0 waiting for initial osdmap 2024-04-21T08:39:23.598 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:39:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:39:23.347+0000 7f332bc9d700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:39:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:23 smithi129 ceph-mon[31272]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-21T08:39:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:23 smithi129 ceph-mon[31272]: osdmap e36: 6 total, 5 up, 6 in 2024-04-21T08:39:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:23 smithi129 ceph-mon[31272]: from='osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:39:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:23 smithi129 ceph-mon[31272]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:39:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:24 smithi057 ceph-mon[27968]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:24 smithi057 ceph-mon[27968]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:39:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:24 smithi057 ceph-mon[27968]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:39:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:24.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:24 smithi136 ceph-mon[31791]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:24 smithi136 ceph-mon[31791]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:39:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:24 smithi136 ceph-mon[31791]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:39:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:24 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:24 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:24.649 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:39:24.649 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:39:24.649 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2024-04-21T08:39:24.649 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2024-04-21T08:39:24.649 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0153577 s, 683 MB/s 2024-04-21T08:39:24.650 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2024-04-21T08:39:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:24 smithi129 ceph-mon[31272]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:24 smithi129 ceph-mon[31272]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:39:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:24 smithi129 ceph-mon[31272]: osdmap e37: 6 total, 5 up, 6 in 2024-04-21T08:39:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:24 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:24 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi057:vg_nvme/lv_2 2024-04-21T08:39:25.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:39:25.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:39:25.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] boot 2024-04-21T08:39:25.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:39:25.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:25 smithi057 ceph-mon[27968]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:39:25.620 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] boot 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:25 smithi136 ceph-mon[31791]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: osd.5 [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] boot 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: osdmap e38: 6 total, 6 up, 6 in 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:39:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:25 smithi129 ceph-mon[31272]: osdmap e39: 6 total, 6 up, 6 in 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:26 smithi136 ceph-mon[31791]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.733 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:26 smithi057 ceph-mon[27968]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:39:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-04-21T08:39:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:26.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:26.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:26.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:26.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:26 smithi129 ceph-mon[31272]: osdmap e40: 6 total, 6 up, 6 in 2024-04-21T08:39:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 -- 172.21.15.57:0/309970523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 msgr2=0x7f26a4101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 --2- 172.21.15.57:0/309970523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4101dc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f26a00099c0 tx=0x7f26a0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:27.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 -- 172.21.15.57:0/309970523 shutdown_connections 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 --2- 172.21.15.57:0/309970523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 0x7f26a4109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 --2- 172.21.15.57:0/309970523 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4101dc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 --2- 172.21.15.57:0/309970523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26a4102ba0 0x7f26a4103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.283+0000 7f26abb8a700 1 -- 172.21.15.57:0/309970523 >> 172.21.15.57:0/309970523 conn(0x7f26a40fcf40 msgr2=0x7f26a40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 -- 172.21.15.57:0/309970523 shutdown_connections 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 -- 172.21.15.57:0/309970523 wait complete. 2024-04-21T08:39:27.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 Processor -- start 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 -- start start 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26a4102ba0 0x7f26a41991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.284+0000 7f26abb8a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 0x7f26a41a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26abb8a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f26a410e570 con 0x7f26a41019a0 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26abb8a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f26a410e3f0 con 0x7f26a4102ba0 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26abb8a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f26a410e6f0 con 0x7f26a4103540 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:53888/0 (socket says 172.21.15.57:53888) 2024-04-21T08:39:27.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 -- 172.21.15.57:0/2939359707 learned_addr learned my addr 172.21.15.57:0/2939359707 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9125700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26a4102ba0 0x7f26a41991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26aa127700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 0x7f26a41a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 -- 172.21.15.57:0/2939359707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 msgr2=0x7f26a41a0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 0x7f26a41a0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 -- 172.21.15.57:0/2939359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26a4102ba0 msgr2=0x7f26a41991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26a4102ba0 0x7f26a41991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 -- 172.21.15.57:0/2939359707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a41a08e0 con 0x7f26a41019a0 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26aa127700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 0x7f26a41a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:39:27.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26a9926700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4198c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f26a000fa80 tx=0x7f26a000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0027070 con 0x7f26a41019a0 2024-04-21T08:39:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a41a0b70 con 0x7f26a41019a0 2024-04-21T08:39:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26a0022920 con 0x7f26a41019a0 2024-04-21T08:39:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.285+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0009e60 con 0x7f26a41019a0 2024-04-21T08:39:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.286+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a41a10a0 con 0x7f26a41019a0 2024-04-21T08:39:27.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.286+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a4069730 con 0x7f26a41019a0 2024-04-21T08:39:27.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.287+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f26a002a070 con 0x7f26a41019a0 2024-04-21T08:39:27.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.288+0000 7f269affd700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f26900a3c40 0x7f26900a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.288+0000 7f26a9125700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f26900a3c40 0x7f26900a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.288+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f26a00d0170 con 0x7f26a41019a0 2024-04-21T08:39:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.289+0000 7f26a9125700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f26900a3c40 0x7f26900a6100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f26a419a170 tx=0x7f2694009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:27.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.292+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26a0013070 con 0x7f26a41019a0 2024-04-21T08:39:27.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:27.513+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f26a4062e80 con 0x7f26900a3c40 2024-04-21T08:39:27.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:27 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi136 2024-04-21T08:39:27.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:27 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi136 to 8544M 2024-04-21T08:39:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:27 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi136 2024-04-21T08:39:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:27 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi136 to 8544M 2024-04-21T08:39:27.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:27 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi136 2024-04-21T08:39:27.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:27 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi136 to 8544M 2024-04-21T08:39:28.405 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:28 smithi057 ceph-mon[27968]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:28.405 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:28 smithi057 ceph-mon[27968]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:39:28.405 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:28.405 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:28.406 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:28.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:28 smithi136 ceph-mon[31791]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:28 smithi136 ceph-mon[31791]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:39:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:28.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:28 smithi129 ceph-mon[31272]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:28 smithi129 ceph-mon[31272]: osdmap e41: 6 total, 6 up, 6 in 2024-04-21T08:39:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:29.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:29 smithi057 ceph-mon[27968]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:29.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:29 smithi057 ceph-mon[27968]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:39:29.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:29 smithi136 ceph-mon[31791]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:29.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:29 smithi136 ceph-mon[31791]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:39:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:29 smithi129 ceph-mon[31272]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:29 smithi129 ceph-mon[31272]: osdmap e42: 6 total, 6 up, 6 in 2024-04-21T08:39:30.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:30 smithi136 ceph-mon[31791]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:30 smithi129 ceph-mon[31272]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:30.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:30 smithi057 ceph-mon[27968]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:31.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:31 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4227254640' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09e7ec78-6803-43af-9a9d-234a5c7d4f4f"}]: dispatch 2024-04-21T08:39:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:31 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4227254640' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09e7ec78-6803-43af-9a9d-234a5c7d4f4f"}]': finished 2024-04-21T08:39:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:31 smithi136 ceph-mon[31791]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:39:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:31 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:31 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2940290670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:31 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4227254640' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09e7ec78-6803-43af-9a9d-234a5c7d4f4f"}]: dispatch 2024-04-21T08:39:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:31 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4227254640' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09e7ec78-6803-43af-9a9d-234a5c7d4f4f"}]': finished 2024-04-21T08:39:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:31 smithi129 ceph-mon[31272]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:39:31.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:31 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:31.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:31 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2940290670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:31.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:31 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4227254640' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09e7ec78-6803-43af-9a9d-234a5c7d4f4f"}]: dispatch 2024-04-21T08:39:31.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:31 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4227254640' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09e7ec78-6803-43af-9a9d-234a5c7d4f4f"}]': finished 2024-04-21T08:39:31.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:31 smithi057 ceph-mon[27968]: osdmap e43: 7 total, 6 up, 7 in 2024-04-21T08:39:31.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:31 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:31.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:31 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2940290670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:32 smithi136 ceph-mon[31791]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:32 smithi129 ceph-mon[31272]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:32.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:32 smithi057 ceph-mon[27968]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:34.510 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:34 smithi057 ceph-mon[27968]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:39:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:34 smithi136 ceph-mon[31791]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:39:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:34 smithi129 ceph-mon[31272]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:39:36.728 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:36 smithi057 ceph-mon[27968]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-21T08:39:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:36 smithi129 ceph-mon[31272]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-21T08:39:36.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:36 smithi136 ceph-mon[31791]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-21T08:39:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:38 smithi136 ceph-mon[31791]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-21T08:39:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:39:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:38.652 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:38 smithi057 ceph-mon[27968]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-21T08:39:38.652 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:39:38.652 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:38 smithi129 ceph-mon[31272]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-21T08:39:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-21T08:39:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:39.671 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:39 smithi057 ceph-mon[27968]: Deploying daemon osd.6 on smithi057 2024-04-21T08:39:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:39 smithi129 ceph-mon[31272]: Deploying daemon osd.6 on smithi057 2024-04-21T08:39:39.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:39 smithi136 ceph-mon[31791]: Deploying daemon osd.6 on smithi057 2024-04-21T08:39:40.720 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:40 smithi057 ceph-mon[27968]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:39:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:40 smithi129 ceph-mon[31272]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:39:40.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:40 smithi136 ceph-mon[31791]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:39:41.801 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:41 smithi057 ceph-mon[27968]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-21T08:39:41.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:41 smithi129 ceph-mon[31272]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-21T08:39:41.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:41 smithi136 ceph-mon[31791]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-21T08:39:43.752 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:43 smithi057 ceph-mon[27968]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-21T08:39:43.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:43 smithi136 ceph-mon[31791]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-21T08:39:44.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:43 smithi129 ceph-mon[31272]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-21T08:39:45.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:45.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:45.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:44 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:45.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:45.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:45.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:44 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:45.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:45.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:45.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:44 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:46.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:45 smithi129 ceph-mon[31272]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:46.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:45 smithi057 ceph-mon[27968]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:46.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:45 smithi136 ceph-mon[31791]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:47.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.980+0000 7f269affd700 1 -- 172.21.15.57:0/2939359707 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f26a4062e80 con 0x7f26900a3c40 2024-04-21T08:39:47.980 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 6 on host 'smithi057' 2024-04-21T08:39:47.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f26900a3c40 msgr2=0x7f26900a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f26900a3c40 0x7f26900a6100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f26a419a170 tx=0x7f2694009260 comp rx=0 tx=0).stop 2024-04-21T08:39:47.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 msgr2=0x7f26a4198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4198c60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f26a000fa80 tx=0x7f26a000fab0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 shutdown_connections 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f26900a3c40 0x7f26900a6100 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26a4103540 0x7f26a41a0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26a41019a0 0x7f26a4198c60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 --2- 172.21.15.57:0/2939359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26a4102ba0 0x7f26a41991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 >> 172.21.15.57:0/2939359707 conn(0x7f26a40fcf40 msgr2=0x7f26a4105fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 shutdown_connections 2024-04-21T08:39:47.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:39:47.983+0000 7f26abb8a700 1 -- 172.21.15.57:0/2939359707 wait complete. 2024-04-21T08:39:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:48 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:47 smithi057 ceph-mon[27968]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:48 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:48 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:48 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:39:48.702 DEBUG:teuthology.orchestra.run.smithi057:osd.6> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.6.service 2024-04-21T08:39:48.704 INFO:tasks.cephadm:Deploying osd.7 on smithi129 with /dev/vg_nvme/lv_2... 2024-04-21T08:39:48.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:39:48.951 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:39:49.272 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:49 smithi057 ceph-mon[27968]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:39:49.273 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:49 smithi057 ceph-mon[27968]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:39:49.273 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:49 smithi057 ceph-mon[27968]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:39:49.273 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:49.305 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:49 smithi129 ceph-mon[31272]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:39:49.305 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:49 smithi129 ceph-mon[31272]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:39:49.305 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:49 smithi129 ceph-mon[31272]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:39:49.305 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:49.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:49 smithi136 ceph-mon[31791]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-21T08:39:49.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:49 smithi136 ceph-mon[31791]: osdmap e44: 7 total, 6 up, 7 in 2024-04-21T08:39:49.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:49 smithi136 ceph-mon[31791]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:39:49.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:49.589 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:39:49 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:39:49.432+0000 7f3626f98700 -1 osd.6 0 waiting for initial osdmap 2024-04-21T08:39:49.589 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:39:49 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:39:49.441+0000 7f361ed83700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:39:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:50 smithi129 ceph-mon[31272]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:50 smithi129 ceph-mon[31272]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:39:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:50 smithi129 ceph-mon[31272]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:39:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:50 smithi129 ceph-mon[31272]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' 2024-04-21T08:39:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:50 smithi057 ceph-mon[27968]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:50 smithi057 ceph-mon[27968]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:39:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:50 smithi057 ceph-mon[27968]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:39:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:50 smithi057 ceph-mon[27968]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' 2024-04-21T08:39:50.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:50 smithi136 ceph-mon[31791]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:50 smithi136 ceph-mon[31791]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:39:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:50 smithi136 ceph-mon[31791]: osdmap e45: 7 total, 6 up, 7 in 2024-04-21T08:39:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:50 smithi136 ceph-mon[31791]: from='osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147]' entity='osd.6' 2024-04-21T08:39:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:39:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:39:51.072 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-04-21T08:39:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-04-21T08:39:51.072 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114378 s, 917 MB/s 2024-04-21T08:39:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-04-21T08:39:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:51 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:39:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:51 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:39:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:51 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:51 smithi129 ceph-mon[31272]: osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] boot 2024-04-21T08:39:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:51 smithi129 ceph-mon[31272]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:39:51.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:51 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:51.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:51 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:39:51.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:51 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:39:51.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:51 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:51.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:51 smithi057 ceph-mon[27968]: osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] boot 2024-04-21T08:39:51.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:51 smithi057 ceph-mon[27968]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:39:51.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:51 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:51.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:51 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:39:51.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:51 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:39:51.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:51 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:51.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:51 smithi136 ceph-mon[31791]: osd.6 [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] boot 2024-04-21T08:39:51.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:51 smithi136 ceph-mon[31791]: osdmap e46: 7 total, 7 up, 7 in 2024-04-21T08:39:51.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:51 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:39:51.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-04-21T08:39:52.003 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:39:52.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:52.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:39:52.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:52.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:52.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:39:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:52.340 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: osdmap e47: 7 total, 7 up, 7 in 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:52.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:39:53.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:53 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi057 2024-04-21T08:39:53.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:53 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi057 to 5697M 2024-04-21T08:39:53.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:53.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:53 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi057 2024-04-21T08:39:53.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:53 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi057 to 5697M 2024-04-21T08:39:53.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:53.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:53 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi057 2024-04-21T08:39:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:53 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi057 to 5697M 2024-04-21T08:39:53.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:39:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 -- 172.21.15.129:0/3057749790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 msgr2=0x7f7bbc101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 --2- 172.21.15.129:0/3057749790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc101dd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00099c0 tx=0x7f7bb0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 -- 172.21.15.129:0/3057749790 shutdown_connections 2024-04-21T08:39:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 --2- 172.21.15.129:0/3057749790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc101dd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 --2- 172.21.15.129:0/3057749790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bbc103550 0x7f7bbc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 --2- 172.21.15.129:0/3057749790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bbc102bb0 0x7f7bbc103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 -- 172.21.15.129:0/3057749790 >> 172.21.15.129:0/3057749790 conn(0x7f7bbc0fcf70 msgr2=0x7f7bbc0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 -- 172.21.15.129:0/3057749790 shutdown_connections 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.675+0000 7f7bc1680700 1 -- 172.21.15.129:0/3057749790 wait complete. 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 Processor -- start 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 -- start start 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc06bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bbc102bb0 0x7f7bbc19fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bbc103550 0x7f7bbc1a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10d440 con 0x7f7bbc103550 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10d2c0 con 0x7f7bbc1019b0 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bc1680700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10d5c0 con 0x7f7bbc102bb0 2024-04-21T08:39:53.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bbaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc06bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bbaffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc06bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52692/0 (socket says 172.21.15.129:52692) 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.676+0000 7f7bbaffd700 1 -- 172.21.15.129:0/2571571567 learned_addr learned my addr 172.21.15.129:0/2571571567 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bba7fc700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bbc102bb0 0x7f7bbc19fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bbaffd700 1 -- 172.21.15.129:0/2571571567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bbc102bb0 msgr2=0x7f7bbc19fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bbaffd700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bbc102bb0 0x7f7bbc19fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bbaffd700 1 -- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bbc103550 msgr2=0x7f7bbc1a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bbaffd700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bbc103550 0x7f7bbc1a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bbaffd700 1 -- 172.21.15.129:0/2571571567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc1a2a90 con 0x7f7bbc1019b0 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bbaffd700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc06bd90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00038c0 tx=0x7f7bb00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0027070 con 0x7f7bbc1019b0 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc1a2d20 con 0x7f7bbc1019b0 2024-04-21T08:39:53.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bb00041a0 con 0x7f7bbc1019b0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb000f460 con 0x7f7bbc1019b0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.677+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc1a31a0 con 0x7f7bbc1019b0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.678+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bbc069730 con 0x7f7bbc1019b0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.679+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7bb002a070 con 0x7f7bbc1019b0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.680+0000 7f7ba3fff700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ba40a3c40 0x7f7ba40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.680+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f7bb00d0ae0 con 0x7f7bbc1019b0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.680+0000 7f7bba7fc700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ba40a3c40 0x7f7ba40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.681+0000 7f7bba7fc700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ba40a3c40 0x7f7ba40a6100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7bac005950 tx=0x7f7bac005eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:39:53.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.684+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7bb0031070 con 0x7f7bbc1019b0 2024-04-21T08:39:53.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:39:53.892+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7bbc062e80 con 0x7f7ba40a3c40 2024-04-21T08:39:54.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:54 smithi129 ceph-mon[31272]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:54.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:54.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:54.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:54.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:54 smithi057 ceph-mon[27968]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:54.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:54.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:54.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:54.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:54 smithi136 ceph-mon[31791]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:54.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:39:54.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:39:54.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:39:55.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:55 smithi129 ceph-mon[31272]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:55.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:55 smithi057 ceph-mon[27968]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:55.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:55 smithi136 ceph-mon[31791]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:39:56.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:56 smithi129 ceph-mon[31272]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:56.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:56 smithi057 ceph-mon[27968]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:56.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:56 smithi136 ceph-mon[31791]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:57 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/2306557662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]: dispatch 2024-04-21T08:39:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:57 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]: dispatch 2024-04-21T08:39:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:57 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]': finished 2024-04-21T08:39:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:57 smithi129 ceph-mon[31272]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:39:57.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:57.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:57 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/2306557662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]: dispatch 2024-04-21T08:39:57.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:57 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]: dispatch 2024-04-21T08:39:57.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:57 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]': finished 2024-04-21T08:39:57.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:57 smithi057 ceph-mon[27968]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:39:57.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:57.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:57 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/2306557662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]: dispatch 2024-04-21T08:39:57.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:57 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]: dispatch 2024-04-21T08:39:57.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:57 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d8a8522-645a-4a69-a6c2-d38b04198b36"}]': finished 2024-04-21T08:39:57.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:57 smithi136 ceph-mon[31791]: osdmap e48: 8 total, 7 up, 8 in 2024-04-21T08:39:57.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:39:58.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:58 smithi129 ceph-mon[31272]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:58.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:39:58 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/1372314329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:58.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:58 smithi057 ceph-mon[27968]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:58.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:39:58 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/1372314329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:39:58.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:58 smithi136 ceph-mon[31791]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:39:58.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:39:58 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/1372314329' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:00.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:00 smithi129 ceph-mon[31272]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:00.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:00 smithi129 ceph-mon[31272]: overall HEALTH_OK 2024-04-21T08:40:00.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:00 smithi057 ceph-mon[27968]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:00.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:00 smithi057 ceph-mon[27968]: overall HEALTH_OK 2024-04-21T08:40:00.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:00 smithi136 ceph-mon[31791]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:00.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:00 smithi136 ceph-mon[31791]: overall HEALTH_OK 2024-04-21T08:40:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:02 smithi129 ceph-mon[31272]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:02.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:02 smithi057 ceph-mon[27968]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:02.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:02 smithi136 ceph-mon[31791]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:04.153 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:04 smithi129 ceph-mon[31272]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:04.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:04 smithi057 ceph-mon[27968]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:04.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:04 smithi136 ceph-mon[31791]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:05 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:40:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:05 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:05.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:05 smithi129 ceph-mon[31272]: Deploying daemon osd.7 on smithi129 2024-04-21T08:40:05.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:05 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:40:05.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:05 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:05.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:05 smithi057 ceph-mon[27968]: Deploying daemon osd.7 on smithi129 2024-04-21T08:40:05.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:05 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-21T08:40:05.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:05 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:05.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:05 smithi136 ceph-mon[31791]: Deploying daemon osd.7 on smithi129 2024-04-21T08:40:06.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:06 smithi057 ceph-mon[27968]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:06.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:06 smithi136 ceph-mon[31791]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:06.388 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:06 smithi129 ceph-mon[31272]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:08.265 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:08 smithi129 ceph-mon[31272]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:08.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:08 smithi057 ceph-mon[27968]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:08.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:08 smithi136 ceph-mon[31791]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:10.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:10 smithi129 ceph-mon[31272]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:10.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:10 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:10.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:10 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:10.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:10 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:10.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:10 smithi136 ceph-mon[31791]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:10.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:10 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:10.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:10 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:10.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:10 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:10.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:10 smithi057 ceph-mon[27968]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:10 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:10 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:10 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:12.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:12 smithi129 ceph-mon[31272]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:12.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:12 smithi136 ceph-mon[31791]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:12 smithi057 ceph-mon[27968]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:13.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.962+0000 7f7ba3fff700 1 -- 172.21.15.129:0/2571571567 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7bbc062e80 con 0x7f7ba40a3c40 2024-04-21T08:40:13.963 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 7 on host 'smithi129' 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ba40a3c40 msgr2=0x7f7ba40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ba40a3c40 0x7f7ba40a6100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7bac005950 tx=0x7f7bac005eb0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 msgr2=0x7f7bbc06bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc06bd90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00038c0 tx=0x7f7bb00047b0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 shutdown_connections 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ba40a3c40 0x7f7ba40a6100 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bbc1019b0 0x7f7bbc06bd90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7bbc103550 0x7f7bbc1a2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 --2- 172.21.15.129:0/2571571567 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7bbc102bb0 0x7f7bbc19fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:13.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 >> 172.21.15.129:0/2571571567 conn(0x7f7bbc0fcf70 msgr2=0x7f7bbc105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:13.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 shutdown_connections 2024-04-21T08:40:13.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:40:13.965+0000 7f7bc1680700 1 -- 172.21.15.129:0/2571571567 wait complete. 2024-04-21T08:40:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:40:14.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:40:14.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:14 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:14 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:14 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:14.721 DEBUG:teuthology.orchestra.run.smithi129:osd.7> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.7.service 2024-04-21T08:40:14.723 INFO:tasks.cephadm:Deploying osd.8 on smithi136 with /dev/vg_nvme/lv_2... 2024-04-21T08:40:14.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-21T08:40:14.967 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:40:15.273 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-mon[31272]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:40:15.273 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-mon[31272]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:40:15.273 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:15.273 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-mon[31272]: from='osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:40:15.273 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-mon[31272]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:40:15.274 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:40:15.157+0000 7f110791f700 -1 osd.7 0 waiting for initial osdmap 2024-04-21T08:40:15.274 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:40:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:40:15.165+0000 7f1103712700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:40:15.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:15 smithi136 ceph-mon[31791]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:40:15.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:15 smithi136 ceph-mon[31791]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:40:15.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:15 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:15.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:15 smithi136 ceph-mon[31791]: from='osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:40:15.350 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:15 smithi136 ceph-mon[31791]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:40:15.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:15 smithi057 ceph-mon[27968]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-21T08:40:15.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:15 smithi057 ceph-mon[27968]: osdmap e49: 8 total, 7 up, 8 in 2024-04-21T08:40:15.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:15 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:15.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:15 smithi057 ceph-mon[27968]: from='osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:40:15.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:15 smithi057 ceph-mon[27968]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:40:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:40:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:40:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] boot 2024-04-21T08:40:16.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:40:16.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:16 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] boot 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:40:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:16 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: osdmap e50: 8 total, 7 up, 8 in 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: osd.7 [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] boot 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: osdmap e51: 8 total, 8 up, 8 in 2024-04-21T08:40:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:16 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:40:17.013 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-21T08:40:17.013 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-21T08:40:17.013 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-21T08:40:17.014 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-21T08:40:17.014 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00849838 s, 1.2 GB/s 2024-04-21T08:40:17.014 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-21T08:40:17.484 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:17 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:40:17.485 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:17 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:40:17.485 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:17 smithi129 ceph-mon[31272]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:40:17.575 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:17 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:40:17.576 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:17 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:40:17.576 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:17 smithi057 ceph-mon[27968]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:40:17.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:17 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:40:17.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:17 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:40:17.627 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:17 smithi136 ceph-mon[31791]: osdmap e52: 8 total, 8 up, 8 in 2024-04-21T08:40:17.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-04-21T08:40:17.892 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: pgmap v178: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:18.364 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:18.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:18 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: pgmap v178: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:18.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:18.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:18 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: pgmap v178: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: osdmap e53: 8 total, 8 up, 8 in 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:18.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:18 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:19.344 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:19 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi129 2024-04-21T08:40:19.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:19 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi129 to 5696M 2024-04-21T08:40:19.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:19 smithi136 ceph-mon[31791]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:40:19.345 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:19 smithi136 ceph-mon[31791]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:40:19.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.567+0000 7f7660a03700 1 -- 172.21.15.136:0/770540383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c103540 msgr2=0x7f765c109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:19.572 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.567+0000 7f7660a03700 1 --2- 172.21.15.136:0/770540383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c103540 0x7f765c109df0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f765400b0a0 tx=0x7f765400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:40:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 -- 172.21.15.136:0/770540383 shutdown_connections 2024-04-21T08:40:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 --2- 172.21.15.136:0/770540383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c103540 0x7f765c109df0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 --2- 172.21.15.136:0/770540383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c1019a0 0x7f765c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 --2- 172.21.15.136:0/770540383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c102ba0 0x7f765c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:19.573 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 -- 172.21.15.136:0/770540383 >> 172.21.15.136:0/770540383 conn(0x7f765c0fcf40 msgr2=0x7f765c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 -- 172.21.15.136:0/770540383 shutdown_connections 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 -- 172.21.15.136:0/770540383 wait complete. 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 Processor -- start 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.568+0000 7f7660a03700 1 -- start start 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c1019a0 0x7f765c1059b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 0x7f765c104050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c103540 0x7f765c104590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:19.574 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f765c10d490 con 0x7f765c103540 2024-04-21T08:40:19.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f765c10d310 con 0x7f765c1019a0 2024-04-21T08:40:19.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f765c10d610 con 0x7f765c102ba0 2024-04-21T08:40:19.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 0x7f765c104050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:19.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 0x7f765c104050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:34576/0 (socket says 172.21.15.136:34576) 2024-04-21T08:40:19.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 -- 172.21.15.136:0/3665074871 learned_addr learned my addr 172.21.15.136:0/3665074871 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:40:19.575 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f765ad9d700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c103540 0x7f765c104590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f765a59c700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c1019a0 0x7f765c1059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 -- 172.21.15.136:0/3665074871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c1019a0 msgr2=0x7f765c1059b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c1019a0 0x7f765c1059b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 -- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c103540 msgr2=0x7f765c104590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c103540 0x7f765c104590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 -- 172.21.15.136:0/3665074871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765c104e60 con 0x7f765c102ba0 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f765ad9d700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c103540 0x7f765c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f765a59c700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c1019a0 0x7f765c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:19.576 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7659d9b700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 0x7f765c104050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f765000b530 tx=0x7f765000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650015020 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765c1a2d60 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7650011ce0 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650010d20 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.569+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765c1a3240 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.571+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765c1a3470 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.571+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7650018020 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.572+0000 7f764b7fe700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f76440a3c40 0x7f76440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.572+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5397+0+0 (secure 0 0 0) 0x7f76500cf190 con 0x7f765c102ba0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.572+0000 7f765a59c700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f76440a3c40 0x7f76440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:19.577 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.573+0000 7f765a59c700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f76440a3c40 0x7f76440a6100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f764c005fd0 tx=0x7f764c005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:19.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:19 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi129 2024-04-21T08:40:19.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:19 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi129 to 5696M 2024-04-21T08:40:19.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:19 smithi129 ceph-mon[31272]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:40:19.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:19 smithi129 ceph-mon[31272]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:40:19.579 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.577+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f765001f020 con 0x7f765c102ba0 2024-04-21T08:40:19.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:19 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi129 2024-04-21T08:40:19.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:19 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi129 to 5696M 2024-04-21T08:40:19.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:19 smithi057 ceph-mon[27968]: osdmap e54: 8 total, 8 up, 8 in 2024-04-21T08:40:19.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:19 smithi057 ceph-mon[27968]: osdmap e55: 8 total, 8 up, 8 in 2024-04-21T08:40:19.783 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:19.783+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f765c1a3640 con 0x7f76440a3c40 2024-04-21T08:40:20.514 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:20 smithi136 ceph-mon[31791]: pgmap v181: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:20.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:20.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:20.515 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:20 smithi129 ceph-mon[31272]: pgmap v181: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:20.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:20 smithi057 ceph-mon[27968]: pgmap v181: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:20.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:20.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:20.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:21.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:21 smithi129 ceph-mon[31272]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:21.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:21 smithi057 ceph-mon[27968]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:21.612 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:21 smithi136 ceph-mon[31791]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:22.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:22 smithi129 ceph-mon[31272]: pgmap v183: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:22.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:22 smithi057 ceph-mon[27968]: pgmap v183: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:22.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:22 smithi136 ceph-mon[31791]: pgmap v183: 1 pgs: 1 remapped; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:23.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/1797174582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]: dispatch 2024-04-21T08:40:23.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:23 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]: dispatch 2024-04-21T08:40:23.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:23 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]': finished 2024-04-21T08:40:23.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:23 smithi129 ceph-mon[31272]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:40:23.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:23.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/320872663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:23.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/1797174582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]: dispatch 2024-04-21T08:40:23.628 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:23 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]: dispatch 2024-04-21T08:40:23.628 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:23 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]': finished 2024-04-21T08:40:23.628 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:23 smithi057 ceph-mon[27968]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:40:23.628 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:23.628 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/320872663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:23.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/1797174582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]: dispatch 2024-04-21T08:40:23.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:23 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]: dispatch 2024-04-21T08:40:23.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:23 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a0b1a0a-5937-4691-ac89-25c328fabf77"}]': finished 2024-04-21T08:40:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:23 smithi136 ceph-mon[31791]: osdmap e56: 9 total, 8 up, 9 in 2024-04-21T08:40:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/320872663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:24.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:24 smithi129 ceph-mon[31272]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:24.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:24 smithi057 ceph-mon[27968]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:24 smithi136 ceph-mon[31791]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:26.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:26 smithi136 ceph-mon[31791]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:26.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:26 smithi129 ceph-mon[31272]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:26.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:26 smithi057 ceph-mon[27968]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:28.564 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:28 smithi136 ceph-mon[31791]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:28 smithi129 ceph-mon[31272]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:28.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:28 smithi057 ceph-mon[27968]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:30 smithi129 ceph-mon[31272]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:40:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:30 smithi057 ceph-mon[27968]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:40:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:30.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:30 smithi136 ceph-mon[31791]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:30.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-21T08:40:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:31.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:31 smithi129 ceph-mon[31272]: Deploying daemon osd.8 on smithi136 2024-04-21T08:40:31.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:31 smithi057 ceph-mon[27968]: Deploying daemon osd.8 on smithi136 2024-04-21T08:40:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:31 smithi136 ceph-mon[31791]: Deploying daemon osd.8 on smithi136 2024-04-21T08:40:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:32 smithi136 ceph-mon[31791]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:32 smithi129 ceph-mon[31272]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:32.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:32 smithi057 ceph-mon[27968]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:34 smithi129 ceph-mon[31272]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:34.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:34 smithi057 ceph-mon[27968]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:34.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:34 smithi136 ceph-mon[31791]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:36.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:36 smithi136 ceph-mon[31791]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:36 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:36 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:36 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:36 smithi129 ceph-mon[31272]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:36 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:36 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:36 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:36.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:36 smithi057 ceph-mon[27968]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:36.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:36 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:36.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:36 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:36.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:36 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:38.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:38 smithi057 ceph-mon[27968]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:38.613 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:38 smithi136 ceph-mon[31791]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:38 smithi129 ceph-mon[31272]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:39.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.654+0000 7f764b7fe700 1 -- 172.21.15.136:0/3665074871 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f765c1a3640 con 0x7f76440a3c40 2024-04-21T08:40:39.655 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 8 on host 'smithi136' 2024-04-21T08:40:39.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f76440a3c40 msgr2=0x7f76440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:39.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f76440a3c40 0x7f76440a6100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f764c005fd0 tx=0x7f764c005f00 comp rx=0 tx=0).stop 2024-04-21T08:40:39.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 msgr2=0x7f765c104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:39.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 0x7f765c104050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f765000b530 tx=0x7f765000b8f0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 shutdown_connections 2024-04-21T08:40:39.657 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f76440a3c40 0x7f76440a6100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f765c1019a0 0x7f765c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f765c103540 0x7f765c104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 --2- 172.21.15.136:0/3665074871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f765c102ba0 0x7f765c104050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:39.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 >> 172.21.15.136:0/3665074871 conn(0x7f765c0fcf40 msgr2=0x7f765c06b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:39.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 shutdown_connections 2024-04-21T08:40:39.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:40:39.657+0000 7f7660a03700 1 -- 172.21.15.136:0/3665074871 wait complete. 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:40.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:40.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:40.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:40.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:40:40.342 DEBUG:teuthology.orchestra.run.smithi136:osd.8> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.8.service 2024-04-21T08:40:40.344 INFO:tasks.cephadm:Deploying osd.9 on smithi057 with /dev/vg_nvme/lv_1... 2024-04-21T08:40:40.344 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:40:40.603 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:40:41.528 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-mon[31791]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:40:41.528 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-mon[31791]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:40:41.528 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:41.528 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-mon[31791]: from='osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:40:41.528 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-mon[31791]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:40:41.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:41 smithi057 ceph-mon[27968]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:40:41.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:41 smithi057 ceph-mon[27968]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:40:41.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:41 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:41.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:41 smithi057 ceph-mon[27968]: from='osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:40:41.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:41 smithi057 ceph-mon[27968]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:40:41.782 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:40:41.771+0000 7f65d81d6700 -1 osd.8 0 waiting for initial osdmap 2024-04-21T08:40:41.782 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:40:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:40:41.782+0000 7f65cffc1700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:40:41.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:41 smithi129 ceph-mon[31272]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-21T08:40:41.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:41 smithi129 ceph-mon[31272]: osdmap e57: 9 total, 8 up, 9 in 2024-04-21T08:40:41.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:41 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:41.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:41 smithi129 ceph-mon[31272]: from='osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:40:41.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:41 smithi129 ceph-mon[31272]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:40:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:42 smithi057 ceph-mon[27968]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:42 smithi057 ceph-mon[27968]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:40:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:42 smithi057 ceph-mon[27968]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:40:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:42 smithi057 ceph-mon[27968]: from='osd.8 ' entity='osd.8' 2024-04-21T08:40:42.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:42 smithi136 ceph-mon[31791]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:42.624 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:42 smithi136 ceph-mon[31791]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:40:42.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:42 smithi136 ceph-mon[31791]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:40:42.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:42.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:42.625 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:42 smithi136 ceph-mon[31791]: from='osd.8 ' entity='osd.8' 2024-04-21T08:40:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:40:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:40:42.730 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10+0 records in 2024-04-21T08:40:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:10+0 records out 2024-04-21T08:40:42.730 INFO:teuthology.orchestra.run.smithi057.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127029 s, 825 MB/s 2024-04-21T08:40:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:--> Zapping successful for: 2024-04-21T08:40:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:42 smithi129 ceph-mon[31272]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:42 smithi129 ceph-mon[31272]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:40:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:42 smithi129 ceph-mon[31272]: osdmap e58: 9 total, 8 up, 9 in 2024-04-21T08:40:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:42 smithi129 ceph-mon[31272]: from='osd.8 ' entity='osd.8' 2024-04-21T08:40:43.469 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:40:43.469 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:40:43.469 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:43.469 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] boot 2024-04-21T08:40:43.469 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:40:43.469 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:43.470 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.498 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi057:vg_nvme/lv_1 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] boot 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:43.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:43.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.762 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:40:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:40:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:40:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: osd.8 [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] boot 2024-04-21T08:40:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: osdmap e59: 9 total, 9 up, 9 in 2024-04-21T08:40:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:40:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:40:44.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:44 smithi136 ceph-mon[31791]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:44.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:44 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi136 2024-04-21T08:40:44.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:44 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi136 to 5696M 2024-04-21T08:40:44.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:44 smithi136 ceph-mon[31791]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:40:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:44 smithi129 ceph-mon[31272]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:44 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi136 2024-04-21T08:40:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:44 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi136 to 5696M 2024-04-21T08:40:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:44 smithi129 ceph-mon[31272]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:40:44.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:44 smithi057 ceph-mon[27968]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-21T08:40:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:44 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi136 2024-04-21T08:40:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:44 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi136 to 5696M 2024-04-21T08:40:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:44 smithi057 ceph-mon[27968]: osdmap e60: 9 total, 9 up, 9 in 2024-04-21T08:40:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.539+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1261631900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 msgr2=0x7f1c281082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.539+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1261631900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c281082d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009990 tx=0x7f1c14009ca0 comp rx=0 tx=0).stop 2024-04-21T08:40:45.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.539+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1261631900 shutdown_connections 2024-04-21T08:40:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1261631900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 0x7f1c2810ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1261631900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 0x7f1c281059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1261631900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c281082d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1261631900 >> 172.21.15.57:0/1261631900 conn(0x7f1c280fcf60 msgr2=0x7f1c280ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:40:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1261631900 shutdown_connections 2024-04-21T08:40:45.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1261631900 wait complete. 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 Processor -- start 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 -- start start 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 0x7f1c281975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c28197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.540+0000 7f1c2ea74700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 0x7f1c28194120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2ea74700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2810e550 con 0x7f1c28105ee0 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2ea74700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2810e3d0 con 0x7f1c28108810 2024-04-21T08:40:45.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2ea74700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2810e6d0 con 0x7f1c281035b0 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c28197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c28197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58880/0 (socket says 172.21.15.57:58880) 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 -- 172.21.15.57:0/1674653489 learned_addr learned my addr 172.21.15.57:0/1674653489 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2d011700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 0x7f1c28194120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2c810700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 0x7f1c281975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 -- 172.21.15.57:0/1674653489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 msgr2=0x7f1c281975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 0x7f1c281975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 -- 172.21.15.57:0/1674653489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 msgr2=0x7f1c28194120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 0x7f1c28194120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 -- 172.21.15.57:0/1674653489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c281949f0 con 0x7f1c28105ee0 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2d011700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 0x7f1c28194120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2c810700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 0x7f1c281975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:40:45.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c27fff700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c28197ae0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006010 tx=0x7f1c140047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:45.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14027070 con 0x7f1c28105ee0 2024-04-21T08:40:45.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c281a7280 con 0x7f1c28105ee0 2024-04-21T08:40:45.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14004180 con 0x7f1c28105ee0 2024-04-21T08:40:45.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.541+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400f460 con 0x7f1c28105ee0 2024-04-21T08:40:45.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.542+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c281a7620 con 0x7f1c28105ee0 2024-04-21T08:40:45.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.542+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c28069730 con 0x7f1c28105ee0 2024-04-21T08:40:45.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.544+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1c1402a070 con 0x7f1c28105ee0 2024-04-21T08:40:45.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.545+0000 7f1c25ffb700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c180a3c40 0x7f1c180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:40:45.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.545+0000 7f1c2c810700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c180a3c40 0x7f1c180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:40:45.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.545+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f1c140d1380 con 0x7f1c28105ee0 2024-04-21T08:40:45.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.545+0000 7f1c2c810700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c180a3c40 0x7f1c180a6100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005fd0 tx=0x7f1c1c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:40:45.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.549+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c14031070 con 0x7f1c28105ee0 2024-04-21T08:40:45.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:45 smithi057 ceph-mon[27968]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:40:45.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:45 smithi057 ceph-mon[27968]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:40:45.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:45 smithi136 ceph-mon[31791]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:40:45.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:45 smithi136 ceph-mon[31791]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:40:45.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:40:45.765+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1c28062e80 con 0x7f1c180a3c40 2024-04-21T08:40:45.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:45 smithi129 ceph-mon[31272]: osdmap e61: 9 total, 9 up, 9 in 2024-04-21T08:40:45.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:45 smithi129 ceph-mon[31272]: osdmap e62: 9 total, 9 up, 9 in 2024-04-21T08:40:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:46 smithi136 ceph-mon[31791]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:46 smithi136 ceph-mon[31791]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:40:46.669 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:46 smithi057 ceph-mon[27968]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:46.669 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:46.669 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:46.669 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:46.669 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:46 smithi057 ceph-mon[27968]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:40:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:46 smithi129 ceph-mon[31272]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:40:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:40:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:46 smithi129 ceph-mon[31272]: osdmap e63: 9 total, 9 up, 9 in 2024-04-21T08:40:47.533 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:47 smithi057 ceph-mon[27968]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:47 smithi136 ceph-mon[31791]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:47.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:47 smithi129 ceph-mon[31272]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi057:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:40:48.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:48 smithi136 ceph-mon[31791]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:40:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:48 smithi129 ceph-mon[31272]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:40:48.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:48 smithi057 ceph-mon[27968]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-21T08:40:49.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2417206602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5fefc0-93b7-4cb0-9b64-ccd879d2f230"}]: dispatch 2024-04-21T08:40:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2417206602' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5fefc0-93b7-4cb0-9b64-ccd879d2f230"}]': finished 2024-04-21T08:40:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:49 smithi136 ceph-mon[31791]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:40:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2417206602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5fefc0-93b7-4cb0-9b64-ccd879d2f230"}]: dispatch 2024-04-21T08:40:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2417206602' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5fefc0-93b7-4cb0-9b64-ccd879d2f230"}]': finished 2024-04-21T08:40:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:49 smithi129 ceph-mon[31272]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:40:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2417206602' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5fefc0-93b7-4cb0-9b64-ccd879d2f230"}]: dispatch 2024-04-21T08:40:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2417206602' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5fefc0-93b7-4cb0-9b64-ccd879d2f230"}]': finished 2024-04-21T08:40:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:49 smithi057 ceph-mon[27968]: osdmap e64: 10 total, 9 up, 10 in 2024-04-21T08:40:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:40:50.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:50 smithi136 ceph-mon[31791]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-21T08:40:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:50 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1579623055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:50 smithi129 ceph-mon[31272]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-21T08:40:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:50 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1579623055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:50.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:50 smithi057 ceph-mon[27968]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-04-21T08:40:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1579623055' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:40:52.784 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:52 smithi057 ceph-mon[27968]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:40:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:52 smithi129 ceph-mon[31272]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:40:52.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:52 smithi136 ceph-mon[31791]: pgmap v207: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-21T08:40:54.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:54 smithi057 ceph-mon[27968]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-21T08:40:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:54 smithi129 ceph-mon[31272]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-21T08:40:54.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:54 smithi136 ceph-mon[31791]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-21T08:40:56.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:56 smithi057 ceph-mon[27968]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:56.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:56 smithi136 ceph-mon[31791]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:56 smithi129 ceph-mon[31272]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:57.449 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:40:57.450 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:57 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:57.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:40:57.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:57 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:57.876 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-21T08:40:57.876 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:57 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:40:58.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:58 smithi057 ceph-mon[27968]: Deploying daemon osd.9 on smithi057 2024-04-21T08:40:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:40:58 smithi057 ceph-mon[27968]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:58.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:58 smithi129 ceph-mon[31272]: Deploying daemon osd.9 on smithi057 2024-04-21T08:40:58.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:40:58 smithi129 ceph-mon[31272]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:40:58.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:58 smithi136 ceph-mon[31791]: Deploying daemon osd.9 on smithi057 2024-04-21T08:40:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:40:58 smithi136 ceph-mon[31791]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:00.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:00 smithi057 ceph-mon[27968]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:00 smithi129 ceph-mon[31272]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:00 smithi136 ceph-mon[31791]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:02 smithi129 ceph-mon[31272]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:02.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:02 smithi057 ceph-mon[27968]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:02.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:02 smithi136 ceph-mon[31791]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:04.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:03 smithi129 ceph-mon[31272]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:04.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:04.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:04.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:04.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:03 smithi057 ceph-mon[27968]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:04.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:04.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:04.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:04.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:03 smithi136 ceph-mon[31791]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:05.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:05 smithi057 ceph-mon[27968]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:06.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:05 smithi129 ceph-mon[31272]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:06.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:05 smithi136 ceph-mon[31791]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:06.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.776+0000 7f1c25ffb700 1 -- 172.21.15.57:0/1674653489 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1c28062e80 con 0x7f1c180a3c40 2024-04-21T08:41:06.775 INFO:teuthology.orchestra.run.smithi057.stdout:Created osd(s) 9 on host 'smithi057' 2024-04-21T08:41:06.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c180a3c40 msgr2=0x7f1c180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:06.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c180a3c40 0x7f1c180a6100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005fd0 tx=0x7f1c1c005ee0 comp rx=0 tx=0).stop 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 msgr2=0x7f1c28197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c28197ae0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c14006010 tx=0x7f1c140047b0 comp rx=0 tx=0).stop 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 shutdown_connections 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c180a3c40 0x7f1c180a6100 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c28108810 0x7f1c28194120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c281035b0 0x7f1c281975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 --2- 172.21.15.57:0/1674653489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c28105ee0 0x7f1c28197ae0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 >> 172.21.15.57:0/1674653489 conn(0x7f1c280fcf60 msgr2=0x7f1c28102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 shutdown_connections 2024-04-21T08:41:06.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:41:06.778+0000 7f1c2ea74700 1 -- 172.21.15.57:0/1674653489 wait complete. 2024-04-21T08:41:06.792 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:06.792 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:06.792 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:06.792 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:06.793 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:06.793 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:41:06.793 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:06.793 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:06.793 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:06 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:07.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:07.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:07.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:41:07.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:06 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:07.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:06 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:07.606 DEBUG:teuthology.orchestra.run.smithi057:osd.9> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.9.service 2024-04-21T08:41:07.609 INFO:tasks.cephadm:Deploying osd.10 on smithi129 with /dev/vg_nvme/lv_1... 2024-04-21T08:41:07.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:41:07.824 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:07 smithi057 ceph-mon[27968]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:07.824 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:07 smithi057 ceph-mon[27968]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:41:07.824 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:07 smithi057 ceph-mon[27968]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:41:07.825 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:07 smithi057 ceph-mon[27968]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:41:07.825 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:07 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:07.862 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:41:07.984 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:07 smithi129 ceph-mon[31272]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:07.984 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:07 smithi129 ceph-mon[31272]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:41:07.984 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:07 smithi129 ceph-mon[31272]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:41:07.984 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:07 smithi129 ceph-mon[31272]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:41:07.984 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:07 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:07 smithi136 ceph-mon[31791]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:07 smithi136 ceph-mon[31791]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-21T08:41:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:07 smithi136 ceph-mon[31791]: osdmap e65: 10 total, 9 up, 10 in 2024-04-21T08:41:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:07 smithi136 ceph-mon[31791]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]: dispatch 2024-04-21T08:41:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:07 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:08.589 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:41:08.264+0000 7f933add6700 -1 osd.9 0 waiting for initial osdmap 2024-04-21T08:41:08.589 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:41:08.278+0000 7f93323c0700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:41:08.941 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-mon[27968]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:41:08.941 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-mon[27968]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:41:08.941 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:08.941 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:08.941 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-mon[27968]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' 2024-04-21T08:41:08.941 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:08 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:08 smithi129 ceph-mon[31272]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:41:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:08 smithi129 ceph-mon[31272]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:41:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:08 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:08 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:08 smithi129 ceph-mon[31272]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' 2024-04-21T08:41:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:08 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:09.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:08 smithi136 ceph-mon[31791]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi057", "root=default"]}]': finished 2024-04-21T08:41:09.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:08 smithi136 ceph-mon[31791]: osdmap e66: 10 total, 9 up, 10 in 2024-04-21T08:41:09.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:08 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:09.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:08 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:09.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:08 smithi136 ceph-mon[31791]: from='osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547]' entity='osd.9' 2024-04-21T08:41:09.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:08 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:41:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:41:10.018 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-04-21T08:41:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-04-21T08:41:10.018 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112341 s, 933 MB/s 2024-04-21T08:41:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-04-21T08:41:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:10 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:41:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:10 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:41:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:10 smithi129 ceph-mon[31272]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:10 smithi129 ceph-mon[31272]: osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] boot 2024-04-21T08:41:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:10 smithi129 ceph-mon[31272]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:41:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:10 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:10 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:41:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:10 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:41:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:10 smithi057 ceph-mon[27968]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:10 smithi057 ceph-mon[27968]: osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] boot 2024-04-21T08:41:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:10 smithi057 ceph-mon[27968]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:41:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:10 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:10.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:10 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:41:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:10 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:41:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:10 smithi136 ceph-mon[31791]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-21T08:41:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:10 smithi136 ceph-mon[31791]: osd.9 [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] boot 2024-04-21T08:41:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:10 smithi136 ceph-mon[31791]: osdmap e67: 10 total, 10 up, 10 in 2024-04-21T08:41:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:10 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:41:10.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-04-21T08:41:11.078 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:41:11.450 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:11 smithi129 ceph-mon[31272]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:41:11.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:11 smithi057 ceph-mon[27968]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:41:11.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:11 smithi136 ceph-mon[31791]: osdmap e68: 10 total, 10 up, 10 in 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi057 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi057 to 4273M 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:12.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:12.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:12 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:12.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi057 2024-04-21T08:41:12.654 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi057 to 4273M 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:12.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:12 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi057 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:12.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi057 to 4273M 2024-04-21T08:41:12.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:12.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:12.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:12 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 -- 172.21.15.129:0/2510232498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c104660 msgr2=0x7fde9c10af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 --2- 172.21.15.129:0/2510232498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c104660 0x7fde9c10af10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fde940099c0 tx=0x7fde94009cd0 comp rx=0 tx=0).stop 2024-04-21T08:41:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 -- 172.21.15.129:0/2510232498 shutdown_connections 2024-04-21T08:41:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 --2- 172.21.15.129:0/2510232498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c104660 0x7fde9c10af10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 --2- 172.21.15.129:0/2510232498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c102ac0 0x7fde9c102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:12.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 --2- 172.21.15.129:0/2510232498 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde9c103cc0 0x7fde9c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 -- 172.21.15.129:0/2510232498 >> 172.21.15.129:0/2510232498 conn(0x7fde9c0fe060 msgr2=0x7fde9c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 -- 172.21.15.129:0/2510232498 shutdown_connections 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.873+0000 7fdea1ced700 1 -- 172.21.15.129:0/2510232498 wait complete. 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 Processor -- start 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 -- start start 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 0x7fde9c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c103cc0 0x7fde9c199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde9c104660 0x7fde9c1a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c10e5b0 con 0x7fde9c103cc0 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c10e430 con 0x7fde9c102ac0 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fdea1ced700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fde9c10e730 con 0x7fde9c104660 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 0x7fde9c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:12.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fde9b7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 0x7fde9c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37630/0 (socket says 172.21.15.129:37630) 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.874+0000 7fde9b7fe700 1 -- 172.21.15.129:0/218298744 learned_addr learned my addr 172.21.15.129:0/218298744 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9affd700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c103cc0 0x7fde9c199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9bfff700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde9c104660 0x7fde9c1a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9b7fe700 1 -- 172.21.15.129:0/218298744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde9c104660 msgr2=0x7fde9c1a00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9b7fe700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde9c104660 0x7fde9c1a00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9b7fe700 1 -- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c103cc0 msgr2=0x7fde9c199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9b7fe700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c103cc0 0x7fde9c199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9b7fe700 1 -- 172.21.15.129:0/218298744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9c1a0790 con 0x7fde9c102ac0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9affd700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c103cc0 0x7fde9c199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde9b7fe700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 0x7fde9c198b30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde8c007e00 tx=0x7fde8c00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c016020 con 0x7fde9c102ac0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9c1a0a80 con 0x7fde9c102ac0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde8c00ad40 con 0x7fde9c102ac0 2024-04-21T08:41:12.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c011670 con 0x7fde9c102ac0 2024-04-21T08:41:12.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.875+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9c1a1010 con 0x7fde9c102ac0 2024-04-21T08:41:12.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.876+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde9c069730 con 0x7fde9c102ac0 2024-04-21T08:41:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.877+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fde8c019020 con 0x7fde9c102ac0 2024-04-21T08:41:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.878+0000 7fde98ff9700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fde880a3c40 0x7fde880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.878+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fde8c0cfd10 con 0x7fde9c102ac0 2024-04-21T08:41:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.878+0000 7fde9affd700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fde880a3c40 0x7fde880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:12.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.878+0000 7fde9affd700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fde880a3c40 0x7fde880a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fde8400bcf0 tx=0x7fde8400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:12.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:12.882+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde8c020030 con 0x7fde9c102ac0 2024-04-21T08:41:13.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:13.091+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fde9c062e80 con 0x7fde880a3c40 2024-04-21T08:41:13.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:13 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:13.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:13 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:13.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:13 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:13.701 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:13 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:13.702 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:13 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:13.702 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:13 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:13.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:13 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:13.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:13 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:13.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:13 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:14.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:14 smithi136 ceph-mon[31791]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:14.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:14 smithi136 ceph-mon[31791]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:14.744 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:14 smithi129 ceph-mon[31272]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:14.744 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:14 smithi129 ceph-mon[31272]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:14.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:14 smithi057 ceph-mon[27968]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:14 smithi057 ceph-mon[27968]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:16 smithi129 ceph-mon[31272]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:16 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/1267558424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]: dispatch 2024-04-21T08:41:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:16 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]: dispatch 2024-04-21T08:41:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:16 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]': finished 2024-04-21T08:41:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:16 smithi129 ceph-mon[31272]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:41:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:16 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:16 smithi057 ceph-mon[27968]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:16 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/1267558424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]: dispatch 2024-04-21T08:41:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:16 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]: dispatch 2024-04-21T08:41:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:16 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]': finished 2024-04-21T08:41:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:16 smithi057 ceph-mon[27968]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:41:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:16 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:16 smithi136 ceph-mon[31791]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:16 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/1267558424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]: dispatch 2024-04-21T08:41:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:16 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]: dispatch 2024-04-21T08:41:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:16 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4319594b-8e0c-4120-a5d9-40c9e1536cac"}]': finished 2024-04-21T08:41:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:16 smithi136 ceph-mon[31791]: osdmap e69: 11 total, 10 up, 11 in 2024-04-21T08:41:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:16 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:17 smithi129 ceph-mon[31272]: from='client.? 172.21.15.129:0/2438811971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:17.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:17 smithi057 ceph-mon[27968]: from='client.? 172.21.15.129:0/2438811971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:17.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:17 smithi136 ceph-mon[31791]: from='client.? 172.21.15.129:0/2438811971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:18 smithi129 ceph-mon[31272]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:18 smithi057 ceph-mon[27968]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:18.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:18 smithi136 ceph-mon[31791]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:20 smithi129 ceph-mon[31272]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:20 smithi057 ceph-mon[27968]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:20.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:20 smithi136 ceph-mon[31791]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:22.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:22 smithi129 ceph-mon[31272]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:22.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:22 smithi057 ceph-mon[27968]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:22.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:22 smithi136 ceph-mon[31791]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:24 smithi129 ceph-mon[31272]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:24 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:41:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:24 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:24.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:24 smithi057 ceph-mon[27968]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:24.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:41:24.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:24 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:24.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:24 smithi136 ceph-mon[31791]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:24 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-21T08:41:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:24 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:25.436 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:25 smithi129 ceph-mon[31272]: Deploying daemon osd.10 on smithi129 2024-04-21T08:41:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:25 smithi057 ceph-mon[27968]: Deploying daemon osd.10 on smithi129 2024-04-21T08:41:25.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:25 smithi136 ceph-mon[31791]: Deploying daemon osd.10 on smithi129 2024-04-21T08:41:26.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:26 smithi129 ceph-mon[31272]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:26.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:26 smithi136 ceph-mon[31791]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:26.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:26 smithi057 ceph-mon[27968]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:28 smithi129 ceph-mon[31272]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:28.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:28 smithi057 ceph-mon[27968]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:28.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:28 smithi136 ceph-mon[31791]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:30 smithi129 ceph-mon[31272]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:30 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:30 smithi057 ceph-mon[27968]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:30 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:30.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:30 smithi136 ceph-mon[31791]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:30.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:30 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:32.569 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:32 smithi129 ceph-mon[31272]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:32.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:32 smithi057 ceph-mon[27968]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:32.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:32 smithi136 ceph-mon[31791]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:32.961 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 10 on host 'smithi129' 2024-04-21T08:41:32.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.961+0000 7fde98ff9700 1 -- 172.21.15.129:0/218298744 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fde9c062e80 con 0x7fde880a3c40 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fde880a3c40 msgr2=0x7fde880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fde880a3c40 0x7fde880a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fde8400bcf0 tx=0x7fde8400b2c0 comp rx=0 tx=0).stop 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 msgr2=0x7fde9c198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 0x7fde9c198b30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde8c007e00 tx=0x7fde8c00a340 comp rx=0 tx=0).stop 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 shutdown_connections 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fde880a3c40 0x7fde880a6100 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde9c102ac0 0x7fde9c198b30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde9c104660 0x7fde9c1a00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 --2- 172.21.15.129:0/218298744 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde9c103cc0 0x7fde9c199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 >> 172.21.15.129:0/218298744 conn(0x7fde9c0fe060 msgr2=0x7fde9c106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 shutdown_connections 2024-04-21T08:41:32.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-04-21T08:41:32.964+0000 7fdea1ced700 1 -- 172.21.15.129:0/218298744 wait complete. 2024-04-21T08:41:33.628 DEBUG:teuthology.orchestra.run.smithi129:osd.10> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.10.service 2024-04-21T08:41:33.630 INFO:tasks.cephadm:Deploying osd.11 on smithi136 with /dev/vg_nvme/lv_1... 2024-04-21T08:41:33.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-21T08:41:33.878 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: from='osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.248 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:33 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: from='osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:41:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:41:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:33 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: from='osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:41:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-21T08:41:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:33 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:35.276 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:34 smithi136 ceph-mon[31791]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:41:35.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:34 smithi136 ceph-mon[31791]: from='osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:41:35.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:34 smithi136 ceph-mon[31791]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:41:35.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:35.277 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:34 smithi136 ceph-mon[31791]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:41:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-mon[31272]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:41:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-mon[31272]: from='osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:41:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-mon[31272]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:41:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-mon[31272]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:41:35.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:41:34.963+0000 7fdff0e1a700 -1 osd.10 0 waiting for initial osdmap 2024-04-21T08:41:35.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:41:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:41:34.976+0000 7fdfecc0d700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:41:35.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:34 smithi057 ceph-mon[27968]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-21T08:41:35.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:34 smithi057 ceph-mon[27968]: from='osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:41:35.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:34 smithi057 ceph-mon[27968]: osdmap e70: 11 total, 10 up, 11 in 2024-04-21T08:41:35.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:35.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:34 smithi057 ceph-mon[27968]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-04-21T08:41:35.979 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-21T08:41:35.979 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-21T08:41:35.980 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-21T08:41:35.980 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-21T08:41:35.980 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00853872 s, 1.2 GB/s 2024-04-21T08:41:35.980 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-21T08:41:36.240 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:36.240 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:41:36.240 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:41:36.240 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:36.241 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] boot 2024-04-21T08:41:36.241 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:41:36.241 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] boot 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:41:36.309 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-04-21T08:41:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-04-21T08:41:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: osdmap e71: 11 total, 10 up, 11 in 2024-04-21T08:41:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: osd.10 [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] boot 2024-04-21T08:41:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: osdmap e72: 11 total, 11 up, 11 in 2024-04-21T08:41:36.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:41:36.729 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-04-21T08:41:36.963 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:41:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:36 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:41:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:36 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:41:37.221 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:36 smithi136 ceph-mon[31791]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:41:37.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:36 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:41:37.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:36 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:41:37.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:36 smithi129 ceph-mon[31272]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:41:37.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:36 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:41:37.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:36 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:41:37.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:36 smithi057 ceph-mon[27968]: osdmap e73: 11 total, 11 up, 11 in 2024-04-21T08:41:38.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:38.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi129 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi129 to 4272M 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:38.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:38.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:38.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:38 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi129 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi129 to 4272M 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:38.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:38 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: osdmap e74: 11 total, 11 up, 11 in 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi129 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi129 to 4272M 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:41:38.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:38 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:38.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.660+0000 7fad8739d700 1 -- 172.21.15.136:0/3945840372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad80104560 msgr2=0x7fad8010ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.660+0000 7fad8739d700 1 --2- 172.21.15.136:0/3945840372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad80104560 0x7fad8010ae10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fad7800b0a0 tx=0x7fad7800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- 172.21.15.136:0/3945840372 shutdown_connections 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 --2- 172.21.15.136:0/3945840372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad801029c0 0x7fad80102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 --2- 172.21.15.136:0/3945840372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad80104560 0x7fad8010ae10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 --2- 172.21.15.136:0/3945840372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad80103bc0 0x7fad80104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- 172.21.15.136:0/3945840372 >> 172.21.15.136:0/3945840372 conn(0x7fad800fdf80 msgr2=0x7fad801003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:38.661 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- 172.21.15.136:0/3945840372 shutdown_connections 2024-04-21T08:41:38.662 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- 172.21.15.136:0/3945840372 wait complete. 2024-04-21T08:41:38.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 Processor -- start 2024-04-21T08:41:38.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- start start 2024-04-21T08:41:38.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 0x7fad80198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad80103bc0 0x7fad80198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.663 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad80104560 0x7fad801a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fad8010e460 con 0x7fad80103bc0 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fad8010e2e0 con 0x7fad80104560 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8739d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fad8010e5e0 con 0x7fad801029c0 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 0x7fad80198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8639b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 0x7fad80198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:41614/0 (socket says 172.21.15.136:41614) 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.661+0000 7fad8639b700 1 -- 172.21.15.136:0/603764892 learned_addr learned my addr 172.21.15.136:0/603764892 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8639b700 1 -- 172.21.15.136:0/603764892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad80104560 msgr2=0x7fad801a0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8639b700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad80104560 0x7fad801a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.664 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8639b700 1 -- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad80103bc0 msgr2=0x7fad80198fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8639b700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad80103bc0 0x7fad80198fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8639b700 1 -- 172.21.15.136:0/603764892 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad801a06d0 con 0x7fad801029c0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8639b700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 0x7fad80198a70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00c920 tx=0x7fad7c00cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c004220 con 0x7fad801029c0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad801a09c0 con 0x7fad801029c0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad7c007d70 con 0x7fad801029c0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c003d40 con 0x7fad801029c0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.662+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad801a0eb0 con 0x7fad801029c0 2024-04-21T08:41:38.665 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.663+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad80069730 con 0x7fad801029c0 2024-04-21T08:41:38.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.664+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fad7c0078c0 con 0x7fad801029c0 2024-04-21T08:41:38.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.665+0000 7fad777fe700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fad6c0a3c40 0x7fad6c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:41:38.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.665+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7fad7c0cfa10 con 0x7fad801029c0 2024-04-21T08:41:38.666 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.665+0000 7fad85b9a700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fad6c0a3c40 0x7fad6c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:41:38.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.666+0000 7fad85b9a700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fad6c0a3c40 0x7fad6c0a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fad7000ba10 tx=0x7fad7000b3f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:41:38.669 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.669+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad7c014030 con 0x7fad801029c0 2024-04-21T08:41:38.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:38.877+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fad80062e80 con 0x7fad6c0a3c40 2024-04-21T08:41:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:39 smithi129 ceph-mon[31272]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:41:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:39 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:39.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:39 smithi057 ceph-mon[27968]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:41:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:39 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:39.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:39 smithi136 ceph-mon[31791]: osdmap e75: 11 total, 11 up, 11 in 2024-04-21T08:41:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-21T08:41:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-21T08:41:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:39 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:40 smithi129 ceph-mon[31272]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:40 smithi129 ceph-mon[31272]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:40 smithi129 ceph-mon[31272]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:41:40.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:40 smithi057 ceph-mon[27968]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:40.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:40 smithi057 ceph-mon[27968]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:40 smithi057 ceph-mon[27968]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:41:40.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:40 smithi136 ceph-mon[31791]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:40 smithi136 ceph-mon[31791]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:41:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:40 smithi136 ceph-mon[31791]: osdmap e76: 11 total, 11 up, 11 in 2024-04-21T08:41:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:42 smithi129 ceph-mon[31272]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:42 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/4019346629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]: dispatch 2024-04-21T08:41:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:42 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]: dispatch 2024-04-21T08:41:42.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:42 smithi129 ceph-mon[31272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]': finished 2024-04-21T08:41:42.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:42 smithi129 ceph-mon[31272]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:41:42.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:42 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:42.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:42 smithi057 ceph-mon[27968]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:42 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/4019346629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]: dispatch 2024-04-21T08:41:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:42 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]: dispatch 2024-04-21T08:41:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:42 smithi057 ceph-mon[27968]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]': finished 2024-04-21T08:41:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:42 smithi057 ceph-mon[27968]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:41:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:42 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:42 smithi136 ceph-mon[31791]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:42 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/4019346629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]: dispatch 2024-04-21T08:41:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:42 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]: dispatch 2024-04-21T08:41:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:42 smithi136 ceph-mon[31791]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b88b4078-cb27-4bfd-a17a-bf0a05dcc291"}]': finished 2024-04-21T08:41:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:42 smithi136 ceph-mon[31791]: osdmap e77: 12 total, 11 up, 12 in 2024-04-21T08:41:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:42 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:41:43.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:43 smithi129 ceph-mon[31272]: from='client.? 172.21.15.136:0/3856289570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:43.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:43 smithi057 ceph-mon[27968]: from='client.? 172.21.15.136:0/3856289570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:43.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:43 smithi136 ceph-mon[31791]: from='client.? 172.21.15.136:0/3856289570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-21T08:41:44.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:44 smithi129 ceph-mon[31272]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-21T08:41:44.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:44 smithi057 ceph-mon[27968]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-21T08:41:44.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:44 smithi136 ceph-mon[31791]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-04-21T08:41:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:46 smithi129 ceph-mon[31272]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-21T08:41:46.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:46 smithi057 ceph-mon[27968]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-21T08:41:46.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:46 smithi136 ceph-mon[31791]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-21T08:41:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:48 smithi129 ceph-mon[31272]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:41:48.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:48 smithi057 ceph-mon[27968]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:41:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:48 smithi136 ceph-mon[31791]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-21T08:41:50.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:50 smithi136 ceph-mon[31791]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:41:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:41:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:50 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:50.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:50 smithi129 ceph-mon[31272]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:41:50.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:41:50.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:50 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:50.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:50 smithi057 ceph-mon[27968]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-21T08:41:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-21T08:41:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:50 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:41:51.394 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:51 smithi136 ceph-mon[31791]: Deploying daemon osd.11 on smithi136 2024-04-21T08:41:51.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:51 smithi129 ceph-mon[31272]: Deploying daemon osd.11 on smithi136 2024-04-21T08:41:51.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:51 smithi057 ceph-mon[27968]: Deploying daemon osd.11 on smithi136 2024-04-21T08:41:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:52 smithi129 ceph-mon[31272]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:41:52.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:52 smithi057 ceph-mon[27968]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:41:52.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:52 smithi136 ceph-mon[31791]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-21T08:41:54.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:54 smithi136 ceph-mon[31791]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:54.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:54 smithi129 ceph-mon[31272]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:54 smithi057 ceph-mon[27968]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:56.426 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:56 smithi136 ceph-mon[31791]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:56.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:56.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:56.427 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:56 smithi057 ceph-mon[27968]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:56 smithi129 ceph-mon[31272]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:41:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:41:58.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:58 smithi136 ceph-mon[31791]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:58.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:58 smithi057 ceph-mon[27968]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:58.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:58 smithi129 ceph-mon[31272]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:41:59.499 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 11 on host 'smithi136' 2024-04-21T08:41:59.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.498+0000 7fad777fe700 1 -- 172.21.15.136:0/603764892 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fad80062e80 con 0x7fad6c0a3c40 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fad6c0a3c40 msgr2=0x7fad6c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fad6c0a3c40 0x7fad6c0a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fad7000ba10 tx=0x7fad7000b3f0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 msgr2=0x7fad80198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 0x7fad80198a70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fad7c00c920 tx=0x7fad7c00cce0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 shutdown_connections 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fad6c0a3c40 0x7fad6c0a6100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad80104560 0x7fad801a0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad801029c0 0x7fad80198a70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 --2- 172.21.15.136:0/603764892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad80103bc0 0x7fad80198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 >> 172.21.15.136:0/603764892 conn(0x7fad800fdf80 msgr2=0x7fad80106ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 shutdown_connections 2024-04-21T08:41:59.501 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:41:59.501+0000 7fad8739d700 1 -- 172.21.15.136:0/603764892 wait complete. 2024-04-21T08:42:00.027 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:00.027 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.027 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.027 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:00.027 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:00.028 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.028 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:42:00.028 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:42:00.028 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.028 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.028 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:41:59 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:41:59 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:00.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:00.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:41:59 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:00.196 DEBUG:teuthology.orchestra.run.smithi136:osd.11> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.11.service 2024-04-21T08:42:00.199 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-21T08:42:00.199 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T08:42:00.459 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:01.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-mon[31791]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:42:01.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-mon[31791]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:42:01.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:01.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-mon[31791]: from='osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:42:01.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-mon[31791]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:42:01.606 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:01 smithi057 ceph-mon[27968]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:42:01.606 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:01 smithi057 ceph-mon[27968]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:42:01.606 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:01 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:01.606 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:01 smithi057 ceph-mon[27968]: from='osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:42:01.607 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:01 smithi057 ceph-mon[27968]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:42:01.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:01 smithi129 ceph-mon[31272]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-21T08:42:01.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:01 smithi129 ceph-mon[31272]: osdmap e78: 12 total, 11 up, 12 in 2024-04-21T08:42:01.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:01 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:01.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:01 smithi129 ceph-mon[31272]: from='osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:42:01.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:01 smithi129 ceph-mon[31272]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-21T08:42:02.135 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:42:01.833+0000 7f83ec0cc700 -1 osd.11 0 waiting for initial osdmap 2024-04-21T08:42:02.136 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:42:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:42:01.846+0000 7f83e3eb7700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-21T08:42:02.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 -- 172.21.15.57:0/1053313637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c104640 msgr2=0x7f199c10aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1053313637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c104640 0x7f199c10aef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f199400b0d0 tx=0x7f199400b3e0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 -- 172.21.15.57:0/1053313637 shutdown_connections 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1053313637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199c102aa0 0x7f199c102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1053313637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c104640 0x7f199c10aef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1053313637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c103ca0 0x7f199c104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 -- 172.21.15.57:0/1053313637 >> 172.21.15.57:0/1053313637 conn(0x7f199c0fe040 msgr2=0x7f199c100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 -- 172.21.15.57:0/1053313637 shutdown_connections 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.548+0000 7f19a3b66700 1 -- 172.21.15.57:0/1053313637 wait complete. 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 Processor -- start 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 -- start start 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c102aa0 0x7f199c198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 0x7f199c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:02.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199c104640 0x7f199c1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f199c10e590 con 0x7f199c103ca0 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f199c10e410 con 0x7f199c104640 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a3b66700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f199c10e710 con 0x7f199c102aa0 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a1101700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 0x7f199c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a1902700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c102aa0 0x7f199c198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a1902700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c102aa0 0x7f199c198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:39782/0 (socket says 172.21.15.57:39782) 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a1101700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 0x7f199c199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45686/0 (socket says 172.21.15.57:45686) 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a1902700 1 -- 172.21.15.57:0/1125657973 learned_addr learned my addr 172.21.15.57:0/1125657973 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.549+0000 7f19a2103700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199c104640 0x7f199c1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a1101700 1 -- 172.21.15.57:0/1125657973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c102aa0 msgr2=0x7f199c198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a1101700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c102aa0 0x7f199c198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a1101700 1 -- 172.21.15.57:0/1125657973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199c104640 msgr2=0x7f199c1a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a1101700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199c104640 0x7f199c1a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a1101700 1 -- 172.21.15.57:0/1125657973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c1a0830 con 0x7f199c103ca0 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a1101700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 0x7f199c199110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f198c00ea20 tx=0x7f198c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c00cbd0 con 0x7f199c103ca0 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f198c00ee60 con 0x7f199c103ca0 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c005020 con 0x7f199c103ca0 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199c1a0b20 con 0x7f199c103ca0 2024-04-21T08:42:02.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.550+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199c1a1010 con 0x7f199c103ca0 2024-04-21T08:42:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.551+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199c069730 con 0x7f199c103ca0 2024-04-21T08:42:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.552+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f198c0051c0 con 0x7f199c103ca0 2024-04-21T08:42:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.553+0000 7f1992ffd700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19880a3c40 0x7f19880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.553+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f198c0cfe40 con 0x7f199c103ca0 2024-04-21T08:42:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.553+0000 7f19a1902700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19880a3c40 0x7f19880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:02.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.554+0000 7f19a1902700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19880a3c40 0x7f19880a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f199800bd70 tx=0x7f199800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:02.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.557+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f198c09ce70 con 0x7f199c103ca0 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:02 smithi136 ceph-mon[31791]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:02 smithi136 ceph-mon[31791]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:02 smithi136 ceph-mon[31791]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:02 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:02 smithi136 ceph-mon[31791]: from='osd.11 ' entity='osd.11' 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:02 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:02.635 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:02 smithi057 ceph-mon[27968]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:02.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:02 smithi057 ceph-mon[27968]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:42:02.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:02 smithi057 ceph-mon[27968]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:42:02.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:02 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:02.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:02 smithi057 ceph-mon[27968]: from='osd.11 ' entity='osd.11' 2024-04-21T08:42:02.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:02 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.766+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f199c02e0e0 con 0x7f199c103ca0 2024-04-21T08:42:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.766+0000 7f1992ffd700 1 -- 172.21.15.57:0/1125657973 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f199c02e0e0 con 0x7f199c103ca0 2024-04-21T08:42:02.766 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.768+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19880a3c40 msgr2=0x7f19880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19880a3c40 0x7f19880a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f199800bd70 tx=0x7f199800b4a0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 msgr2=0x7f199c199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 0x7f199c199110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f198c00ea20 tx=0x7f198c00ed30 comp rx=0 tx=0).stop 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 shutdown_connections 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19880a3c40 0x7f19880a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199c104640 0x7f199c1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f199c102aa0 0x7f199c198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 --2- 172.21.15.57:0/1125657973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f199c103ca0 0x7f199c199110 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 >> 172.21.15.57:0/1125657973 conn(0x7f199c0fe040 msgr2=0x7f199c106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 shutdown_connections 2024-04-21T08:42:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:02.769+0000 7f19a3b66700 1 -- 172.21.15.57:0/1125657973 wait complete. 2024-04-21T08:42:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:02 smithi129 ceph-mon[31272]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:02 smithi129 ceph-mon[31272]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-21T08:42:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:02 smithi129 ceph-mon[31272]: osdmap e79: 12 total, 11 up, 12 in 2024-04-21T08:42:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:02 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:02 smithi129 ceph-mon[31272]: from='osd.11 ' entity='osd.11' 2024-04-21T08:42:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:02 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:03.417 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1713688895,"num_in_osds":12,"osd_in_since":1713688901,"num_remapped_pgs":0} 2024-04-21T08:42:03.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:03 smithi136 ceph-mon[31791]: purged_snaps scrub starts 2024-04-21T08:42:03.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:03 smithi136 ceph-mon[31791]: purged_snaps scrub ok 2024-04-21T08:42:03.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1125657973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:42:03.531 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:03 smithi136 ceph-mon[31791]: osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] boot 2024-04-21T08:42:03.532 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:03 smithi136 ceph-mon[31791]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:42:03.532 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:03 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:03.625 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:03 smithi057 ceph-mon[27968]: purged_snaps scrub starts 2024-04-21T08:42:03.625 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:03 smithi057 ceph-mon[27968]: purged_snaps scrub ok 2024-04-21T08:42:03.625 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1125657973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:42:03.625 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:03 smithi057 ceph-mon[27968]: osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] boot 2024-04-21T08:42:03.625 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:03 smithi057 ceph-mon[27968]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:42:03.625 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:03 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:03 smithi129 ceph-mon[31272]: purged_snaps scrub starts 2024-04-21T08:42:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:03 smithi129 ceph-mon[31272]: purged_snaps scrub ok 2024-04-21T08:42:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1125657973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:42:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:03 smithi129 ceph-mon[31272]: osd.11 [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] boot 2024-04-21T08:42:03.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:03 smithi129 ceph-mon[31272]: osdmap e80: 12 total, 12 up, 12 in 2024-04-21T08:42:03.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:03 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:42:04.418 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:04.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:04.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:04 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.679 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:04.803 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:04.803 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.803 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.803 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:04.804 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:04 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-21T08:42:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:42:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:04 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:05.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:05 smithi129 ceph-mon[31272]: Detected new or changed devices on smithi136 2024-04-21T08:42:05.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:05 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi136 to 4272M 2024-04-21T08:42:05.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:05 smithi129 ceph-mon[31272]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:42:05.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:05 smithi057 ceph-mon[27968]: Detected new or changed devices on smithi136 2024-04-21T08:42:05.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:05 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi136 to 4272M 2024-04-21T08:42:05.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:05 smithi057 ceph-mon[27968]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:42:05.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:05 smithi136 ceph-mon[31791]: Detected new or changed devices on smithi136 2024-04-21T08:42:05.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:05 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi136 to 4272M 2024-04-21T08:42:05.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:05 smithi136 ceph-mon[31791]: osdmap e81: 12 total, 12 up, 12 in 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 -- 172.21.15.57:0/1600562738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 msgr2=0x7f8b48102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1600562738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48102ee0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8b380099c0 tx=0x7f8b38009cd0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 -- 172.21.15.57:0/1600562738 shutdown_connections 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1600562738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48104660 0x7f8b4810af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1600562738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48102ee0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1600562738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 0x7f8b48104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 -- 172.21.15.57:0/1600562738 >> 172.21.15.57:0/1600562738 conn(0x7f8b480fe060 msgr2=0x7f8b481004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 -- 172.21.15.57:0/1600562738 shutdown_connections 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.533+0000 7f8b4da36700 1 -- 172.21.15.57:0/1600562738 wait complete. 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 Processor -- start 2024-04-21T08:42:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 -- start start 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 0x7f8b48199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48104660 0x7f8b481a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810e550 con 0x7f8b48102ac0 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810e3d0 con 0x7f8b48104660 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b4da36700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4810e6d0 con 0x7f8b48103cc0 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b46ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b46ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42086/0 (socket says 172.21.15.57:42086) 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b46ffd700 1 -- 172.21.15.57:0/1731349893 learned_addr learned my addr 172.21.15.57:0/1731349893 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.534+0000 7f8b477fe700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48104660 0x7f8b481a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b467fc700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 0x7f8b48199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b46ffd700 1 -- 172.21.15.57:0/1731349893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 msgr2=0x7f8b48199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b46ffd700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 0x7f8b48199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b46ffd700 1 -- 172.21.15.57:0/1731349893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48104660 msgr2=0x7f8b481a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b46ffd700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48104660 0x7f8b481a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b46ffd700 1 -- 172.21.15.57:0/1731349893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b481a0830 con 0x7f8b48102ac0 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b467fc700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 0x7f8b48199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b46ffd700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48198c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009f50 tx=0x7f8b380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b38027070 con 0x7f8b48102ac0 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b481a0ac0 con 0x7f8b48102ac0 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b380040a0 con 0x7f8b48102ac0 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b38022da0 con 0x7f8b48102ac0 2024-04-21T08:42:06.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.535+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b481a0fa0 con 0x7f8b48102ac0 2024-04-21T08:42:06.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.536+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b48069730 con 0x7f8b48102ac0 2024-04-21T08:42:06.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.537+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8b3802a070 con 0x7f8b48102ac0 2024-04-21T08:42:06.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.538+0000 7f8b4ca34700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8b300a3c40 0x7f8b300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:06.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.538+0000 7f8b467fc700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8b300a3c40 0x7f8b300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:06.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.538+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8b380d1100 con 0x7f8b48102ac0 2024-04-21T08:42:06.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.538+0000 7f8b467fc700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8b300a3c40 0x7f8b300a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00bcf0 tx=0x7f8b3c00b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:06.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.542+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b38031070 con 0x7f8b48102ac0 2024-04-21T08:42:06.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.752+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8b48063650 con 0x7f8b48102ac0 2024-04-21T08:42:06.752 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:06 smithi057 ceph-mon[27968]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:06.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.753+0000 7f8b4ca34700 1 -- 172.21.15.57:0/1731349893 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f8b38013070 con 0x7f8b48102ac0 2024-04-21T08:42:06.753 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:06.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8b300a3c40 msgr2=0x7f8b300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8b300a3c40 0x7f8b300a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c00bcf0 tx=0x7f8b3c00b2c0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 msgr2=0x7f8b48198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48198c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009f50 tx=0x7f8b380047b0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 shutdown_connections 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8b300a3c40 0x7f8b300a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48104660 0x7f8b481a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b48102ac0 0x7f8b48198c20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 --2- 172.21.15.57:0/1731349893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b48103cc0 0x7f8b48199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 >> 172.21.15.57:0/1731349893 conn(0x7f8b480fe060 msgr2=0x7f8b48106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 shutdown_connections 2024-04-21T08:42:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:06.755+0000 7f8b4da36700 1 -- 172.21.15.57:0/1731349893 wait complete. 2024-04-21T08:42:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:06 smithi129 ceph-mon[31272]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:06.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:06 smithi136 ceph-mon[31791]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:07.593 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713688922,"num_in_osds":12,"osd_in_since":1713688901,"num_remapped_pgs":0} 2024-04-21T08:42:07.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:42:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:07 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1731349893' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:42:07.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:07 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1731349893' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:42:07.847 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:07.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:07 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1731349893' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-21T08:42:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:08 smithi057 ceph-mon[27968]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:08 smithi129 ceph-mon[31272]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:08 smithi136 ceph-mon[31791]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:09.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 -- 172.21.15.57:0/620166697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64103480 msgr2=0x7f3d6410f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 --2- 172.21.15.57:0/620166697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64103480 0x7f3d6410f8d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c00b0a0 tx=0x7f3d5c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 -- 172.21.15.57:0/620166697 shutdown_connections 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 --2- 172.21.15.57:0/620166697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64103480 0x7f3d6410f8d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 --2- 172.21.15.57:0/620166697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d64108a70 0x7f3d64108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 --2- 172.21.15.57:0/620166697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d64102ac0 0x7f3d64102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 -- 172.21.15.57:0/620166697 >> 172.21.15.57:0/620166697 conn(0x7f3d64079bb0 msgr2=0x7f3d64079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 -- 172.21.15.57:0/620166697 shutdown_connections 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 -- 172.21.15.57:0/620166697 wait complete. 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 Processor -- start 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.573+0000 7f3d6c740700 1 -- start start 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6c740700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 0x7f3d64074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6c740700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d64103480 0x7f3d64075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6c740700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d64108a70 0x7f3d640757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:09.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6c740700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64112d50 con 0x7f3d64102ac0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6c740700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64112bd0 con 0x7f3d64103480 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6c740700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64112ed0 con 0x7f3d64108a70 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 0x7f3d64074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 0x7f3d64074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42106/0 (socket says 172.21.15.57:42106) 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 -- 172.21.15.57:0/1219872597 learned_addr learned my addr 172.21.15.57:0/1219872597 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6acdd700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d64108a70 0x7f3d640757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d69cdb700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d64103480 0x7f3d64075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 -- 172.21.15.57:0/1219872597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d64108a70 msgr2=0x7f3d640757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d64108a70 0x7f3d640757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 -- 172.21.15.57:0/1219872597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d64103480 msgr2=0x7f3d64075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d64103480 0x7f3d64075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 -- 172.21.15.57:0/1219872597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d641a7920 con 0x7f3d64102ac0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d6a4dc700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 0x7f3d64074d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000b530 tx=0x7f3d6000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60015020 con 0x7f3d64102ac0 2024-04-21T08:42:09.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.574+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d60011ce0 con 0x7f3d64102ac0 2024-04-21T08:42:09.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.575+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d641a7c10 con 0x7f3d64102ac0 2024-04-21T08:42:09.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.575+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60010e40 con 0x7f3d64102ac0 2024-04-21T08:42:09.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.575+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d641a8150 con 0x7f3d64102ac0 2024-04-21T08:42:09.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.575+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d64069730 con 0x7f3d64102ac0 2024-04-21T08:42:09.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.577+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3d60018020 con 0x7f3d64102ac0 2024-04-21T08:42:09.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.577+0000 7f3d5b7fe700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3d500a3cc0 0x7f3d500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:09.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.577+0000 7f3d69cdb700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3d500a3cc0 0x7f3d500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:09.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.577+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3d600cfca0 con 0x7f3d64102ac0 2024-04-21T08:42:09.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.578+0000 7f3d69cdb700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3d500a3cc0 0x7f3d500a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d54006fd0 tx=0x7f3d54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:09.580 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.581+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d6001f020 con 0x7f3d64102ac0 2024-04-21T08:42:09.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.792+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d64076270 con 0x7f3d64102ac0 2024-04-21T08:42:09.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.793+0000 7f3d5b7fe700 1 -- 172.21.15.57:0/1219872597 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7f3d6009cb20 con 0x7f3d64102ac0 2024-04-21T08:42:09.793 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:09.793 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:42:04.373173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3d500a3cc0 msgr2=0x7f3d500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3d500a3cc0 0x7f3d500a6180 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3d54006fd0 tx=0x7f3d54008040 comp rx=0 tx=0).stop 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 msgr2=0x7f3d64074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 0x7f3d64074d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000b530 tx=0x7f3d6000b840 comp rx=0 tx=0).stop 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 shutdown_connections 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3d500a3cc0 0x7f3d500a6180 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d64108a70 0x7f3d640757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3d64102ac0 0x7f3d64074d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 --2- 172.21.15.57:0/1219872597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d64103480 0x7f3d64075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 >> 172.21.15.57:0/1219872597 conn(0x7f3d64079bb0 msgr2=0x7f3d64101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 shutdown_connections 2024-04-21T08:42:09.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:09.795+0000 7f3d6c740700 1 -- 172.21.15.57:0/1219872597 wait complete. 2024-04-21T08:42:10.581 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:42:10.581 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-21T08:42:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:10 smithi129 ceph-mon[31272]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:10 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1219872597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:10.831 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:42:10.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:10 smithi057 ceph-mon[27968]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:10.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:10 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1219872597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:10.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:10 smithi136 ceph-mon[31791]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:10.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:10 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1219872597' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:12.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 -- 172.21.15.57:0/2815259017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c06bbb0 msgr2=0x7fd52c10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:12.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2815259017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c06bbb0 0x7fd52c10f930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd52400b0a0 tx=0x7fd52400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 -- 172.21.15.57:0/2815259017 shutdown_connections 2024-04-21T08:42:12.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2815259017 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c06bbb0 0x7fd52c10f930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2815259017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd52c108a60 0x7fd52c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2815259017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd52c06b1f0 0x7fd52c06b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 -- 172.21.15.57:0/2815259017 >> 172.21.15.57:0/2815259017 conn(0x7fd52c0789e0 msgr2=0x7fd52c078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 -- 172.21.15.57:0/2815259017 shutdown_connections 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 -- 172.21.15.57:0/2815259017 wait complete. 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.567+0000 7fd5345ee700 1 Processor -- start 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 -- start start 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd52c06b1f0 0x7fd52c1992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd52c06bbb0 0x7fd52c199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 0x7fd52c19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112d50 con 0x7fd52c108a60 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112bd0 con 0x7fd52c06b1f0 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd5345ee700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c112ed0 con 0x7fd52c06bbb0 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 0x7fd52c19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:12.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 0x7fd52c19db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42118/0 (socket says 172.21.15.57:42118) 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 -- 172.21.15.57:0/2724464613 learned_addr learned my addr 172.21.15.57:0/2724464613 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd531b89700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd52c06bbb0 0x7fd52c199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd53238a700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd52c06b1f0 0x7fd52c1992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 -- 172.21.15.57:0/2724464613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd52c06bbb0 msgr2=0x7fd52c199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd52c06bbb0 0x7fd52c199820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 -- 172.21.15.57:0/2724464613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd52c06b1f0 msgr2=0x7fd52c1992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd52c06b1f0 0x7fd52c1992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.568+0000 7fd532b8b700 1 -- 172.21.15.57:0/2724464613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c19e2a0 con 0x7fd52c108a60 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.569+0000 7fd532b8b700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 0x7fd52c19db60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd52400b070 tx=0x7fd52400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.569+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524021070 con 0x7fd52c108a60 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.569+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd524007b80 con 0x7fd52c108a60 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.569+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c19e530 con 0x7fd52c108a60 2024-04-21T08:42:12.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.569+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5240052a0 con 0x7fd52c108a60 2024-04-21T08:42:12.571 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:12 smithi057 ceph-mon[27968]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:12.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.569+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1a5e10 con 0x7fd52c108a60 2024-04-21T08:42:12.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.570+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd52c1041a0 con 0x7fd52c108a60 2024-04-21T08:42:12.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.572+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd524024070 con 0x7fd52c108a60 2024-04-21T08:42:12.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.572+0000 7fd5237fe700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd5180a3c40 0x7fd5180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:12.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.573+0000 7fd53238a700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd5180a3c40 0x7fd5180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:12.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.573+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd5240d1060 con 0x7fd52c108a60 2024-04-21T08:42:12.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.573+0000 7fd53238a700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd5180a3c40 0x7fd5180a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009a10 tx=0x7fd51c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:12.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.577+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd52409df70 con 0x7fd52c108a60 2024-04-21T08:42:12.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.784+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd52c02df80 con 0x7fd52c108a60 2024-04-21T08:42:12.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.784+0000 7fd5237fe700 1 -- 172.21.15.57:0/2724464613 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fd52409d870 con 0x7fd52c108a60 2024-04-21T08:42:12.784 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd5180a3c40 msgr2=0x7fd5180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd5180a3c40 0x7fd5180a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009a10 tx=0x7fd51c009190 comp rx=0 tx=0).stop 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 msgr2=0x7fd52c19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 0x7fd52c19db60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd52400b070 tx=0x7fd52400ce50 comp rx=0 tx=0).stop 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 shutdown_connections 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd5180a3c40 0x7fd5180a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd52c06b1f0 0x7fd52c1992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd52c108a60 0x7fd52c19db60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 --2- 172.21.15.57:0/2724464613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd52c06bbb0 0x7fd52c199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:12.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 >> 172.21.15.57:0/2724464613 conn(0x7fd52c0789e0 msgr2=0x7fd52c0ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 shutdown_connections 2024-04-21T08:42:12.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:12.786+0000 7fd5345ee700 1 -- 172.21.15.57:0/2724464613 wait complete. 2024-04-21T08:42:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:12 smithi129 ceph-mon[31272]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:12.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:12 smithi136 ceph-mon[31791]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:13.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2724464613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:13.627 INFO:tasks.cephadm:Adding mds.a on smithi057 2024-04-21T08:42:13.628 INFO:tasks.cephadm:Adding mds.d on smithi057 2024-04-21T08:42:13.628 INFO:tasks.cephadm:Adding mds.g on smithi057 2024-04-21T08:42:13.628 INFO:tasks.cephadm:Adding mds.j on smithi057 2024-04-21T08:42:13.628 INFO:tasks.cephadm:Adding mds.b on smithi129 2024-04-21T08:42:13.628 INFO:tasks.cephadm:Adding mds.e on smithi129 2024-04-21T08:42:13.628 INFO:tasks.cephadm:Adding mds.h on smithi129 2024-04-21T08:42:13.629 INFO:tasks.cephadm:Adding mds.k on smithi129 2024-04-21T08:42:13.629 INFO:tasks.cephadm:Adding mds.c on smithi136 2024-04-21T08:42:13.629 INFO:tasks.cephadm:Adding mds.f on smithi136 2024-04-21T08:42:13.629 INFO:tasks.cephadm:Adding mds.i on smithi136 2024-04-21T08:42:13.629 INFO:tasks.cephadm:Adding mds.l on smithi136 2024-04-21T08:42:13.629 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l' 2024-04-21T08:42:13.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2724464613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:13.882 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:42:13.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2724464613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:14.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:14 smithi136 ceph-mon[31791]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:14 smithi129 ceph-mon[31272]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:14.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:14 smithi057 ceph-mon[27968]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:15.691 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 -- 172.21.15.136:0/3333276309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 msgr2=0x7f9cd8109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 --2- 172.21.15.136:0/3333276309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd8109df0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000b0a0 tx=0x7f9cd000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 -- 172.21.15.136:0/3333276309 shutdown_connections 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 --2- 172.21.15.136:0/3333276309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd81019a0 0x7f9cd8101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 --2- 172.21.15.136:0/3333276309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd8109df0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 --2- 172.21.15.136:0/3333276309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9cd8102ba0 0x7f9cd8103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 -- 172.21.15.136:0/3333276309 >> 172.21.15.136:0/3333276309 conn(0x7f9cd80fcf40 msgr2=0x7f9cd80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 -- 172.21.15.136:0/3333276309 shutdown_connections 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.690+0000 7f9cdf092700 1 -- 172.21.15.136:0/3333276309 wait complete. 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 Processor -- start 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 -- start start 2024-04-21T08:42:15.692 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9cd81019a0 0x7f9cd8105930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8102ba0 0x7f9cd8103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd81044c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd810d420 con 0x7f9cd81019a0 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd810d2a0 con 0x7f9cd8102ba0 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdf092700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd810d5a0 con 0x7f9cd8103540 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd81044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd81044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58310/0 (socket says 172.21.15.136:58310) 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 -- 172.21.15.136:0/1032421470 learned_addr learned my addr 172.21.15.136:0/1032421470 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cd7fff700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8102ba0 0x7f9cd8103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 -- 172.21.15.136:0/1032421470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8102ba0 msgr2=0x7f9cd8103f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8102ba0 0x7f9cd8103f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.693 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 -- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9cd81019a0 msgr2=0x7f9cd8105930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9cd81019a0 0x7f9cd8105930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 -- 172.21.15.136:0/1032421470 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd8104d70 con 0x7f9cd8103540 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.691+0000 7f9cdd62f700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd81044c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00062a0 tx=0x7f9cd0004ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.692+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0021070 con 0x7f9cd8103540 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.692+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd81a2c50 con 0x7f9cd8103540 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.692+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cd0007db0 con 0x7f9cd8103540 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.692+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd001d450 con 0x7f9cd8103540 2024-04-21T08:42:15.694 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.692+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd81a3100 con 0x7f9cd8103540 2024-04-21T08:42:15.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.693+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd8069730 con 0x7f9cd8103540 2024-04-21T08:42:15.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.694+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9cd0003a00 con 0x7f9cd8103540 2024-04-21T08:42:15.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.694+0000 7f9cd5ffb700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9cc80a3c40 0x7f9cc80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:15.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.695+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9cd000f020 con 0x7f9cd8103540 2024-04-21T08:42:15.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.695+0000 7f9cdce2e700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9cc80a3c40 0x7f9cc80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:15.695 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.695+0000 7f9cdce2e700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9cc80a3c40 0x7f9cc80a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8102a00 tx=0x7f9ccc009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:15.698 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.699+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9cd002a020 con 0x7f9cd8103540 2024-04-21T08:42:15.909 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.910+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}) -- 0x7f9cd80ff1e0 con 0x7f9cc80a3c40 2024-04-21T08:42:15.918 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.919+0000 7f9cd5ffb700 1 -- 172.21.15.136:0/1032421470 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f9cd80ff1e0 con 0x7f9cc80a3c40 2024-04-21T08:42:15.918 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mds.all update... 2024-04-21T08:42:15.920 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9cc80a3c40 msgr2=0x7f9cc80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9cc80a3c40 0x7f9cc80a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8102a00 tx=0x7f9ccc009470 comp rx=0 tx=0).stop 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 msgr2=0x7f9cd81044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd81044c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00062a0 tx=0x7f9cd0004ef0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 shutdown_connections 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9cc80a3c40 0x7f9cc80a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cd8103540 0x7f9cd81044c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9cd81019a0 0x7f9cd8105930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 --2- 172.21.15.136:0/1032421470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8102ba0 0x7f9cd8103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 >> 172.21.15.136:0/1032421470 conn(0x7f9cd80fcf40 msgr2=0x7f9cd80fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 shutdown_connections 2024-04-21T08:42:15.921 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-21T08:42:15.921+0000 7f9cdf092700 1 -- 172.21.15.136:0/1032421470 wait complete. 2024-04-21T08:42:16.614 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:16 smithi136 ceph-mon[31791]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:16.615 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:16 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:16.615 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:16 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:16.653 DEBUG:teuthology.orchestra.run.smithi057:mds.a> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.a.service 2024-04-21T08:42:16.657 DEBUG:teuthology.orchestra.run.smithi057:mds.d> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.d.service 2024-04-21T08:42:16.701 DEBUG:teuthology.orchestra.run.smithi057:mds.g> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.g.service 2024-04-21T08:42:16.720 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:16 smithi057 ceph-mon[27968]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:16.721 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:16 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:16.721 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:16 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:16.722 DEBUG:teuthology.orchestra.run.smithi057:mds.j> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.j.service 2024-04-21T08:42:16.765 DEBUG:teuthology.orchestra.run.smithi129:mds.b> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.b.service 2024-04-21T08:42:16.769 DEBUG:teuthology.orchestra.run.smithi129:mds.e> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.e.service 2024-04-21T08:42:16.788 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:16 smithi129 ceph-mon[31272]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:16.789 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:16 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:16.789 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:16 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:16.790 DEBUG:teuthology.orchestra.run.smithi129:mds.h> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.h.service 2024-04-21T08:42:16.835 DEBUG:teuthology.orchestra.run.smithi129:mds.k> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.k.service 2024-04-21T08:42:16.880 DEBUG:teuthology.orchestra.run.smithi136:mds.c> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.c.service 2024-04-21T08:42:16.883 DEBUG:teuthology.orchestra.run.smithi136:mds.f> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.f.service 2024-04-21T08:42:16.927 DEBUG:teuthology.orchestra.run.smithi136:mds.i> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.i.service 2024-04-21T08:42:16.970 DEBUG:teuthology.orchestra.run.smithi136:mds.l> sudo journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.l.service 2024-04-21T08:42:17.013 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-21T08:42:17.013 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:17.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.128+0000 7f0e31d04640 1 -- 172.21.15.57:0/2130255243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c102980 msgr2=0x7f0e2c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.128+0000 7f0e31d04640 1 --2- 172.21.15.57:0/2130255243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c102980 0x7f0e2c1092a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009be0 tx=0x7f0e2402bef0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 -- 172.21.15.57:0/2130255243 shutdown_connections 2024-04-21T08:42:17.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 --2- 172.21.15.57:0/2130255243 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c102980 0x7f0e2c1092a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 --2- 172.21.15.57:0/2130255243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e2c101fc0 0x7f0e2c102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 --2- 172.21.15.57:0/2130255243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e2c100e70 0x7f0e2c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 -- 172.21.15.57:0/2130255243 >> 172.21.15.57:0/2130255243 conn(0x7f0e2c0fc3e0 msgr2=0x7f0e2c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 -- 172.21.15.57:0/2130255243 shutdown_connections 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 -- 172.21.15.57:0/2130255243 wait complete. 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 Processor -- start 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 -- start start 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.129+0000 7f0e31d04640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 0x7f0e2c198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e2c101fc0 0x7f0e2c198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e2c102980 0x7f0e2c19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10c9a0 con 0x7f0e2c100e70 2024-04-21T08:42:17.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10c820 con 0x7f0e2c101fc0 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10cb20 con 0x7f0e2c102980 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 0x7f0e2c198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 0x7f0e2c198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46792/0 (socket says 172.21.15.57:46792) 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 -- 172.21.15.57:0/3674434433 learned_addr learned my addr 172.21.15.57:0/3674434433 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 -- 172.21.15.57:0/3674434433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e2c102980 msgr2=0x7f0e2c19f6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e2c102980 0x7f0e2c19f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 -- 172.21.15.57:0/3674434433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e2c101fc0 msgr2=0x7f0e2c198650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e2c101fc0 0x7f0e2c198650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 -- 172.21.15.57:0/3674434433 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c19fd30 con 0x7f0e2c100e70 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e2b7fe640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 0x7f0e2c198110 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00ea60 tx=0x7f0e1c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c00cde0 con 0x7f0e2c100e70 2024-04-21T08:42:17.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e1c0035f0 con 0x7f0e2c100e70 2024-04-21T08:42:17.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c010690 con 0x7f0e2c100e70 2024-04-21T08:42:17.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c19ffc0 con 0x7f0e2c100e70 2024-04-21T08:42:17.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.130+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c1a0460 con 0x7f0e2c100e70 2024-04-21T08:42:17.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.131+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2c1a0720 con 0x7f0e2c100e70 2024-04-21T08:42:17.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.132+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0e1c0037a0 con 0x7f0e2c100e70 2024-04-21T08:42:17.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.133+0000 7f0e297fa640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0e180a3bb0 0x7f0e180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.133+0000 7f0e22ffd640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0e180a3bb0 0x7f0e180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.133+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0e1c0d03f0 con 0x7f0e2c100e70 2024-04-21T08:42:17.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.133+0000 7f0e22ffd640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0e180a3bb0 0x7f0e180a6070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c199620 tx=0x7f0e14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.138+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e1c09d1d0 con 0x7f0e2c100e70 2024-04-21T08:42:17.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.325+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e2c1a0f50 con 0x7f0e2c100e70 2024-04-21T08:42:17.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.326+0000 7f0e297fa640 1 -- 172.21.15.57:0/3674434433 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7f0e1c09cad0 con 0x7f0e2c100e70 2024-04-21T08:42:17.327 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:17.327 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:42:04.373173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0e180a3bb0 msgr2=0x7f0e180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0e180a3bb0 0x7f0e180a6070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c199620 tx=0x7f0e14009260 comp rx=0 tx=0).stop 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 msgr2=0x7f0e2c198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 0x7f0e2c198110 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00ea60 tx=0x7f0e1c00ef20 comp rx=0 tx=0).stop 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 shutdown_connections 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0e180a3bb0 0x7f0e180a6070 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e2c102980 0x7f0e2c19f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e2c101fc0 0x7f0e2c198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 --2- 172.21.15.57:0/3674434433 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e2c100e70 0x7f0e2c198110 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 >> 172.21.15.57:0/3674434433 conn(0x7f0e2c0fc3e0 msgr2=0x7f0e2c0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 shutdown_connections 2024-04-21T08:42:17.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.328+0000 7f0e31d04640 1 -- 172.21.15.57:0/3674434433 wait complete. 2024-04-21T08:42:17.340 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:42:17.341 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:42:17.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 -- 172.21.15.57:0/771292855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 msgr2=0x7f8d4c107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 --2- 172.21.15.57:0/771292855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c107730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009900 tx=0x7f8d4002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:17.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 -- 172.21.15.57:0/771292855 shutdown_connections 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 --2- 172.21.15.57:0/771292855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d4c107c70 0x7f8d4c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 --2- 172.21.15.57:0/771292855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c107730 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 --2- 172.21.15.57:0/771292855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 0x7f8d4c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 -- 172.21.15.57:0/771292855 >> 172.21.15.57:0/771292855 conn(0x7f8d4c0fc400 msgr2=0x7f8d4c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 -- 172.21.15.57:0/771292855 shutdown_connections 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.446+0000 7f8d52443640 1 -- 172.21.15.57:0/771292855 wait complete. 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 Processor -- start 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 -- start start 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 0x7f8d4c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d4c107c70 0x7f8d4c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c10dab0 con 0x7f8d4c105340 2024-04-21T08:42:17.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c10d930 con 0x7f8d4c102a10 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d52443640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4c10dc30 con 0x7f8d4c107c70 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d4b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d4b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46808/0 (socket says 172.21.15.57:46808) 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d4b7fe640 1 -- 172.21.15.57:0/877363359 learned_addr learned my addr 172.21.15.57:0/877363359 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d509e0640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d4c107c70 0x7f8d4c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.447+0000 7f8d4bfff640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 0x7f8d4c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4b7fe640 1 -- 172.21.15.57:0/877363359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d4c107c70 msgr2=0x7f8d4c19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4b7fe640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d4c107c70 0x7f8d4c19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4b7fe640 1 -- 172.21.15.57:0/877363359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 msgr2=0x7f8d4c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4b7fe640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 0x7f8d4c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4b7fe640 1 -- 172.21.15.57:0/877363359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4c19fe50 con 0x7f8d4c105340 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4bfff640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 0x7f8d4c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:17.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d4b7fe640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c198770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000c00 tx=0x7f8d40004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40043020 con 0x7f8d4c105340 2024-04-21T08:42:17.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d40004440 con 0x7f8d4c105340 2024-04-21T08:42:17.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d4c1a00e0 con 0x7f8d4c105340 2024-04-21T08:42:17.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40003c90 con 0x7f8d4c105340 2024-04-21T08:42:17.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.448+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d4c1a0630 con 0x7f8d4c105340 2024-04-21T08:42:17.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.449+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d4c0fdef0 con 0x7f8d4c105340 2024-04-21T08:42:17.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.450+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8d40046020 con 0x7f8d4c105340 2024-04-21T08:42:17.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.451+0000 7f8d497fa640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d340a3bb0 0x7f8d340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.451+0000 7f8d4bfff640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d340a3bb0 0x7f8d340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.451+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8d400f2250 con 0x7f8d4c105340 2024-04-21T08:42:17.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.451+0000 7f8d4bfff640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d340a3bb0 0x7f8d340a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009c20 tx=0x7f8d3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.455+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d4004d020 con 0x7f8d4c105340 2024-04-21T08:42:17.593 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:17.593 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: Saving service mds.all spec with placement smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.594 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:17 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3674434433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:17.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: Saving service mds.all spec with placement smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:17 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3674434433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:17.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.643+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d4c062b50 con 0x7f8d4c105340 2024-04-21T08:42:17.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.644+0000 7f8d497fa640 1 -- 172.21.15.57:0/877363359 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f8d400bf030 con 0x7f8d4c105340 2024-04-21T08:42:17.644 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:42:17.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d340a3bb0 msgr2=0x7f8d340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d340a3bb0 0x7f8d340a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009c20 tx=0x7f8d3c005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 msgr2=0x7f8d4c198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c198770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000c00 tx=0x7f8d40004040 comp rx=0 tx=0).stop 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 shutdown_connections 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d340a3bb0 0x7f8d340a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d4c107c70 0x7f8d4c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8d4c105340 0x7f8d4c198770 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 --2- 172.21.15.57:0/877363359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d4c102a10 0x7f8d4c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 >> 172.21.15.57:0/877363359 conn(0x7f8d4c0fc400 msgr2=0x7f8d4c1048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 shutdown_connections 2024-04-21T08:42:17.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.646+0000 7f8d52443640 1 -- 172.21.15.57:0/877363359 wait complete. 2024-04-21T08:42:17.659 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 -- 172.21.15.57:0/284140613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe481007e0 msgr2=0x7efe48100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 --2- 172.21.15.57:0/284140613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe481007e0 0x7efe48100c00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe3c009900 tx=0x7efe3c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 -- 172.21.15.57:0/284140613 shutdown_connections 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 --2- 172.21.15.57:0/284140613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe48101ae0 0x7efe48101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 --2- 172.21.15.57:0/284140613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe48101140 0x7efe481015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 --2- 172.21.15.57:0/284140613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe481007e0 0x7efe48100c00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 -- 172.21.15.57:0/284140613 >> 172.21.15.57:0/284140613 conn(0x7efe480fc3e0 msgr2=0x7efe480fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.761+0000 7efe4d831640 1 -- 172.21.15.57:0/284140613 shutdown_connections 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 -- 172.21.15.57:0/284140613 wait complete. 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 Processor -- start 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 -- start start 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe481007e0 0x7efe4819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe48101140 0x7efe4819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 0x7efe481a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810fd30 con 0x7efe48101ae0 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810fbb0 con 0x7efe481007e0 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe4d831640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efe4810feb0 con 0x7efe48101140 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.762+0000 7efe477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 0x7efe481a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 0x7efe481a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46834/0 (socket says 172.21.15.57:46834) 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 -- 172.21.15.57:0/4034060981 learned_addr learned my addr 172.21.15.57:0/4034060981 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:17.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe46ffd640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe481007e0 0x7efe4819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe467fc640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe48101140 0x7efe4819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 -- 172.21.15.57:0/4034060981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe48101140 msgr2=0x7efe4819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe48101140 0x7efe4819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 -- 172.21.15.57:0/4034060981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe481007e0 msgr2=0x7efe4819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe481007e0 0x7efe4819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 -- 172.21.15.57:0/4034060981 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe481a2020 con 0x7efe48101ae0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe477fe640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 0x7efe481a19b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efe4000eaa0 tx=0x7efe4000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40004d20 con 0x7efe48101ae0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe40005020 con 0x7efe48101ae0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe481a2310 con 0x7efe48101ae0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe400116b0 con 0x7efe48101ae0 2024-04-21T08:42:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.763+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe481a2890 con 0x7efe48101ae0 2024-04-21T08:42:17.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.764+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe4810b720 con 0x7efe48101ae0 2024-04-21T08:42:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.765+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efe400051d0 con 0x7efe48101ae0 2024-04-21T08:42:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.766+0000 7efe4c82f640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efe300a3bb0 0x7efe300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.766+0000 7efe46ffd640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efe300a3bb0 0x7efe300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.766+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efe400d0130 con 0x7efe48101ae0 2024-04-21T08:42:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.766+0000 7efe46ffd640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efe300a3bb0 0x7efe300a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000c00 tx=0x7efe3c02c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:17.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.770+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe40016030 con 0x7efe48101ae0 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: Saving service mds.all spec with placement smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:17.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:17.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:17 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3674434433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:17.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.995+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe48062b50 con 0x7efe48101ae0 2024-04-21T08:42:17.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.996+0000 7efe4c82f640 1 -- 172.21.15.57:0/4034060981 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7efe4009cfb0 con 0x7efe48101ae0 2024-04-21T08:42:17.996 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:17.997 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-21T08:42:17.997 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 1 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efe300a3bb0 msgr2=0x7efe300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efe300a3bb0 0x7efe300a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000c00 tx=0x7efe3c02c7e0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 msgr2=0x7efe481a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 0x7efe481a19b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efe4000eaa0 tx=0x7efe4000eea0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 shutdown_connections 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7efe300a3bb0 0x7efe300a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efe48101ae0 0x7efe481a19b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe48101140 0x7efe4819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 --2- 172.21.15.57:0/4034060981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe481007e0 0x7efe4819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 >> 172.21.15.57:0/4034060981 conn(0x7efe480fc3e0 msgr2=0x7efe480fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 shutdown_connections 2024-04-21T08:42:17.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:17.998+0000 7efe4d831640 1 -- 172.21.15.57:0/4034060981 wait complete. 2024-04-21T08:42:18.610 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:18 smithi136 ceph-mon[31791]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:18.610 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:18 smithi136 ceph-mon[31791]: Deploying daemon mds.l on smithi136 2024-04-21T08:42:18.610 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:18 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/877363359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:18.610 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:18 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4034060981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:18 smithi129 ceph-mon[31272]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:18 smithi129 ceph-mon[31272]: Deploying daemon mds.l on smithi136 2024-04-21T08:42:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:18 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/877363359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:18 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4034060981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:18 smithi057 ceph-mon[27968]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:18 smithi057 ceph-mon[27968]: Deploying daemon mds.l on smithi136 2024-04-21T08:42:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:18 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/877363359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:18 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4034060981' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:18.885 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:18 smithi136 systemd[1]: Starting Ceph mds.l for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:19.320 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 podman[61116]: 2024-04-21 08:42:18.907744411 +0000 UTC m=+0.018343186 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:19.320 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 podman[61116]: 2024-04-21 08:42:19.010282259 +0000 UTC m=+0.120881014 container create e954924f42a4b4136ea08c2eb0bf6134f07966b1ad3ed293301e277e83de104d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-21T08:42:19.320 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 podman[61116]: 2024-04-21 08:42:19.321708596 +0000 UTC m=+0.432307348 container init e954924f42a4b4136ea08c2eb0bf6134f07966b1ad3ed293301e277e83de104d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-21T08:42:19.635 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 podman[61116]: 2024-04-21 08:42:19.324477047 +0000 UTC m=+0.435075802 container start e954924f42a4b4136ea08c2eb0bf6134f07966b1ad3ed293301e277e83de104d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:42:19.635 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 ceph-mds[61135]: -- 172.21.15.136:0/2040763191 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55751bb0ba40 con 0x55751badf000 2024-04-21T08:42:19.635 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l[61131]: starting mds.l at 2024-04-21T08:42:19.635 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 bash[61116]: e954924f42a4b4136ea08c2eb0bf6134f07966b1ad3ed293301e277e83de104d 2024-04-21T08:42:19.635 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:42:19 smithi136 systemd[1]: Started Ceph mds.l for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:20.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:20.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 -- 172.21.15.57:0/2549455898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7541030f0 msgr2=0x7fd754103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2549455898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7541030f0 0x7fd754103570 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd744009900 tx=0x7fd74402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:20.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 -- 172.21.15.57:0/2549455898 shutdown_connections 2024-04-21T08:42:20.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2549455898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754103ab0 0x7fd75410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2549455898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7541030f0 0x7fd754103570 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2549455898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd754101f20 0x7fd754102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.125+0000 7fd75b3db640 1 -- 172.21.15.57:0/2549455898 >> 172.21.15.57:0/2549455898 conn(0x7fd7540fd500 msgr2=0x7fd7540ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 -- 172.21.15.57:0/2549455898 shutdown_connections 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 -- 172.21.15.57:0/2549455898 wait complete. 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 Processor -- start 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 -- start start 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 0x7fd7541981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7541030f0 0x7fd7541986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754103ab0 0x7fd75419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd75410daa0 con 0x7fd754101f20 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd75410d920 con 0x7fd7541030f0 2024-04-21T08:42:20.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd75b3db640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd75410dc20 con 0x7fd754103ab0 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.126+0000 7fd759177640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 0x7fd7541981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 0x7fd7541981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46852/0 (socket says 172.21.15.57:46852) 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 -- 172.21.15.57:0/2326410630 learned_addr learned my addr 172.21.15.57:0/2326410630 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd758976640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7541030f0 0x7fd7541986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759978640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754103ab0 0x7fd75419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 -- 172.21.15.57:0/2326410630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754103ab0 msgr2=0x7fd75419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754103ab0 0x7fd75419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 -- 172.21.15.57:0/2326410630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7541030f0 msgr2=0x7fd7541986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7541030f0 0x7fd7541986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 -- 172.21.15.57:0/2326410630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd75419fdc0 con 0x7fd754101f20 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd758976640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7541030f0 0x7fd7541986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd759177640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 0x7fd7541981a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd75000e9c0 tx=0x7fd75000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:20.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75000cd80 con 0x7fd754101f20 2024-04-21T08:42:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7500035f0 con 0x7fd754101f20 2024-04-21T08:42:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7541a00b0 con 0x7fd754101f20 2024-04-21T08:42:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750010690 con 0x7fd754101f20 2024-04-21T08:42:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.127+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7541a0600 con 0x7fd754101f20 2024-04-21T08:42:20.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.128+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7541093d0 con 0x7fd754101f20 2024-04-21T08:42:20.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.129+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd7500037a0 con 0x7fd754101f20 2024-04-21T08:42:20.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.130+0000 7fd74a7fc640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7400a3bb0 0x7fd7400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:20.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.130+0000 7fd758976640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7400a3bb0 0x7fd7400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:20.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.130+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd7500d0560 con 0x7fd754101f20 2024-04-21T08:42:20.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.131+0000 7fd758976640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7400a3bb0 0x7fd7400a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7541996b0 tx=0x7fd74402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:20.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.134+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd75009d340 con 0x7fd754101f20 2024-04-21T08:42:20.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.364+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd75402d5c0 con 0x7fd754101f20 2024-04-21T08:42:20.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.364+0000 7fd74a7fc640 1 -- 172.21.15.57:0/2326410630 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fd75009cc40 con 0x7fd754101f20 2024-04-21T08:42:20.365 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:20.365 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:42:20.365 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 2 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7400a3bb0 msgr2=0x7fd7400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7400a3bb0 0x7fd7400a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7541996b0 tx=0x7fd74402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 msgr2=0x7fd7541981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 0x7fd7541981a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd75000e9c0 tx=0x7fd75000ee80 comp rx=0 tx=0).stop 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 shutdown_connections 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7400a3bb0 0x7fd7400a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754103ab0 0x7fd75419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7541030f0 0x7fd7541986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 --2- 172.21.15.57:0/2326410630 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd754101f20 0x7fd7541981a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 >> 172.21.15.57:0/2326410630 conn(0x7fd7540fd500 msgr2=0x7fd7541065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 shutdown_connections 2024-04-21T08:42:20.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:20.366+0000 7fd75b3db640 1 -- 172.21.15.57:0/2326410630 wait complete. 2024-04-21T08:42:20.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:20.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.136:6832/3199760269,v1:172.21.15.136:6833/3199760269] up:boot 2024-04-21T08:42:20.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: fsmap 1 up:standby 2024-04-21T08:42:20.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:42:20.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:20 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.136:6832/3199760269,v1:172.21.15.136:6833/3199760269] up:boot 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: fsmap 1 up:standby 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:20.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:20.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:20 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.136:6832/3199760269,v1:172.21.15.136:6833/3199760269] up:boot 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: fsmap 1 up:standby 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:20.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:20 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:21.540 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:21 smithi129 ceph-mon[31272]: Deploying daemon mds.k on smithi129 2024-04-21T08:42:21.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:21 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2326410630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:21.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:21 smithi136 ceph-mon[31791]: Deploying daemon mds.k on smithi129 2024-04-21T08:42:21.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:21 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2326410630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:21.823 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:21 smithi129 systemd[1]: Starting Ceph mds.k for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:21.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:21 smithi057 ceph-mon[27968]: Deploying daemon mds.k on smithi129 2024-04-21T08:42:21.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:21 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2326410630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:22.231 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:21 smithi129 podman[60140]: 2024-04-21 08:42:21.86421589 +0000 UTC m=+0.018238171 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:22.232 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:21 smithi129 podman[60140]: 2024-04-21 08:42:21.978950485 +0000 UTC m=+0.132972791 container create 2f532aef29f7e130a1cb000eba3a657b71a5d3a2b5cf16f2d5fa7364c7062118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T08:42:22.232 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:22 smithi129 podman[60140]: 2024-04-21 08:42:22.206530063 +0000 UTC m=+0.360552328 container init 2f532aef29f7e130a1cb000eba3a657b71a5d3a2b5cf16f2d5fa7364c7062118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T08:42:22.232 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:22 smithi129 podman[60140]: 2024-04-21 08:42:22.209287299 +0000 UTC m=+0.363309575 container start 2f532aef29f7e130a1cb000eba3a657b71a5d3a2b5cf16f2d5fa7364c7062118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-21T08:42:22.378 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 -- 172.21.15.57:0/2213244786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 msgr2=0x7fbbb4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/2213244786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb4101290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009900 tx=0x7fbba402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 -- 172.21.15.57:0/2213244786 shutdown_connections 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/2213244786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb4102980 0x7fbbb41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/2213244786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb4101fc0 0x7fbbb4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/2213244786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb4101290 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 -- 172.21.15.57:0/2213244786 >> 172.21.15.57:0/2213244786 conn(0x7fbbb40fc3e0 msgr2=0x7fbbb40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 -- 172.21.15.57:0/2213244786 shutdown_connections 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 -- 172.21.15.57:0/2213244786 wait complete. 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 Processor -- start 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.493+0000 7fbbba0c1640 1 -- start start 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbba0c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbba0c1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb4101fc0 0x7fbbb4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbba0c1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb4102980 0x7fbbb419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbba0c1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410c9c0 con 0x7fbbb4100e70 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbba0c1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410c840 con 0x7fbbb4102980 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbba0c1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410cb40 con 0x7fbbb4101fc0 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb41981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46862/0 (socket says 172.21.15.57:46862) 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 -- 172.21.15.57:0/714618372 learned_addr learned my addr 172.21.15.57:0/714618372 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbaaffd640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb4101fc0 0x7fbbb4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb3fff640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb4102980 0x7fbbb419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 -- 172.21.15.57:0/714618372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb4101fc0 msgr2=0x7fbbb4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb4101fc0 0x7fbbb4198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 -- 172.21.15.57:0/714618372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb4102980 msgr2=0x7fbbb419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb4102980 0x7fbbb419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.494+0000 7fbbb37fe640 1 -- 172.21.15.57:0/714618372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb419fe00 con 0x7fbbb4100e70 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.495+0000 7fbbb37fe640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb41981e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbba4006010 tx=0x7fbba40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.495+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4043020 con 0x7fbbb4100e70 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.495+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb41a0090 con 0x7fbbb4100e70 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.495+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbba4004aa0 con 0x7fbbb4100e70 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.495+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4003c70 con 0x7fbbb4100e70 2024-04-21T08:42:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.495+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb41a05e0 con 0x7fbbb4100e70 2024-04-21T08:42:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.496+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb41a07e0 con 0x7fbbb4100e70 2024-04-21T08:42:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.497+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbba4046020 con 0x7fbbb4100e70 2024-04-21T08:42:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.498+0000 7fbbb17fa640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbba00a3bb0 0x7fbba00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.498+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbba40f2240 con 0x7fbbb4100e70 2024-04-21T08:42:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.498+0000 7fbbaaffd640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbba00a3bb0 0x7fbba00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:22.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.498+0000 7fbbaaffd640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbba00a3bb0 0x7fbba00a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbbb41996f0 tx=0x7fbb9c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:22.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.507+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbba404d020 con 0x7fbbb4100e70 2024-04-21T08:42:22.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:22 smithi129 ceph-mon[31272]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:22.573 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:22 smithi129 ceph-mds[60159]: -- 172.21.15.129:0/2136562849 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e90e9fba40 con 0x55e90e9cec00 2024-04-21T08:42:22.573 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k[60155]: starting mds.k at 2024-04-21T08:42:22.573 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:22 smithi129 bash[60140]: 2f532aef29f7e130a1cb000eba3a657b71a5d3a2b5cf16f2d5fa7364c7062118 2024-04-21T08:42:22.573 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:42:22 smithi129 systemd[1]: Started Ceph mds.k for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:22.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:22 smithi136 ceph-mon[31791]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:22.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.737+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbb41a09b0 con 0x7fbbb4100e70 2024-04-21T08:42:22.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.737+0000 7fbbb17fa640 1 -- 172.21.15.57:0/714618372 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fbba40bf0c0 con 0x7fbbb4100e70 2024-04-21T08:42:22.738 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:22.738 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:42:22.738 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 3 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbba00a3bb0 msgr2=0x7fbba00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbba00a3bb0 0x7fbba00a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbbb41996f0 tx=0x7fbb9c00b390 comp rx=0 tx=0).stop 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 msgr2=0x7fbbb41981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb41981e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbba4006010 tx=0x7fbba40042f0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 shutdown_connections 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbba00a3bb0 0x7fbba00a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb4102980 0x7fbbb419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbb4101fc0 0x7fbbb4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 --2- 172.21.15.57:0/714618372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbb4100e70 0x7fbbb41981e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:22.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.739+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 >> 172.21.15.57:0/714618372 conn(0x7fbbb40fc3e0 msgr2=0x7fbbb40fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:22.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.740+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 shutdown_connections 2024-04-21T08:42:22.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:22.740+0000 7fbbba0c1640 1 -- 172.21.15.57:0/714618372 wait complete. 2024-04-21T08:42:22.750 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:22 smithi057 ceph-mon[27968]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.129:6832/3055010748,v1:172.21.15.129:6833/3055010748] up:boot 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: fsmap 2 up:standby 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: Deploying daemon mds.j on smithi057 2024-04-21T08:42:23.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/714618372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.129:6832/3055010748,v1:172.21.15.129:6833/3055010748] up:boot 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: fsmap 2 up:standby 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:23.633 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:23.634 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: Deploying daemon mds.j on smithi057 2024-04-21T08:42:23.634 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/714618372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:23.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.129:6832/3055010748,v1:172.21.15.129:6833/3055010748] up:boot 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: fsmap 2 up:standby 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: Deploying daemon mds.j on smithi057 2024-04-21T08:42:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/714618372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:24.708 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:24 smithi057 ceph-mon[27968]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:24.752 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:24 smithi129 ceph-mon[31272]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.864+0000 7f1026d5f640 1 -- 172.21.15.57:0/1533217264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020107ca0 msgr2=0x7f102010a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.867 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.864+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1533217264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020107ca0 0x7f102010a0f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1018009ca0 tx=0x7f101802c120 comp rx=0 tx=0).stop 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- 172.21.15.57:0/1533217264 shutdown_connections 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1533217264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020107ca0 0x7f102010a0f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1533217264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1020105370 0x7f1020107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1533217264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1020102a40 0x7f1020104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- 172.21.15.57:0/1533217264 >> 172.21.15.57:0/1533217264 conn(0x7f10200fc470 msgr2=0x7f10200fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- 172.21.15.57:0/1533217264 shutdown_connections 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- 172.21.15.57:0/1533217264 wait complete. 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 Processor -- start 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- start start 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1020102a40 0x7f102019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 0x7f102019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1020107ca0 0x7f10201a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f102010db80 con 0x7f1020105370 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f102010da00 con 0x7f1020107ca0 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f1026d5f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f102010dd00 con 0x7f1020102a40 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f101ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 0x7f102019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f101ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 0x7f102019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34388/0 (socket says 172.21.15.57:34388) 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.865+0000 7f101ffff640 1 -- 172.21.15.57:0/1058476121 learned_addr learned my addr 172.21.15.57:0/1058476121 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101ffff640 1 -- 172.21.15.57:0/1058476121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1020102a40 msgr2=0x7f102019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101ffff640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1020102a40 0x7f102019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101ffff640 1 -- 172.21.15.57:0/1058476121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1020107ca0 msgr2=0x7f10201a19c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101ffff640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1020107ca0 0x7f10201a19c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101ffff640 1 -- 172.21.15.57:0/1058476121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10201a2030 con 0x7f1020105370 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101ffff640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 0x7f102019a950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f101400b470 tx=0x7f101400b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014015020 con 0x7f1020105370 2024-04-21T08:42:24.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1014011d40 con 0x7f1020105370 2024-04-21T08:42:24.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10201a2320 con 0x7f1020105370 2024-04-21T08:42:24.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101400f340 con 0x7f1020105370 2024-04-21T08:42:24.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.866+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10201a28f0 con 0x7f1020105370 2024-04-21T08:42:24.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.867+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1020109790 con 0x7f1020105370 2024-04-21T08:42:24.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.868+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1014018020 con 0x7f1020105370 2024-04-21T08:42:24.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.869+0000 7f101dffb640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10080a3bb0 0x7f10080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:24.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.869+0000 7f1024afb640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10080a3bb0 0x7f10080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:24.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.869+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f10140cfb30 con 0x7f1020105370 2024-04-21T08:42:24.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.870+0000 7f1024afb640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10080a3bb0 0x7f10080a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1010005fd0 tx=0x7f1010005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:24.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:24.874+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f101401f020 con 0x7f1020105370 2024-04-21T08:42:24.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:24 smithi136 ceph-mon[31791]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:25.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.102+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1020062b50 con 0x7f1020105370 2024-04-21T08:42:25.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.103+0000 7f101dffb640 1 -- 172.21.15.57:0/1058476121 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f101400d040 con 0x7f1020105370 2024-04-21T08:42:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:25.103 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 3 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10080a3bb0 msgr2=0x7f10080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10080a3bb0 0x7f10080a6070 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1010005fd0 tx=0x7f1010005850 comp rx=0 tx=0).stop 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 msgr2=0x7f102019a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 0x7f102019a950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f101400b470 tx=0x7f101400b930 comp rx=0 tx=0).stop 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 shutdown_connections 2024-04-21T08:42:25.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10080a3bb0 0x7f10080a6070 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:25.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1020107ca0 0x7f10201a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:25.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1020105370 0x7f102019a950 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:25.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 --2- 172.21.15.57:0/1058476121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1020102a40 0x7f102019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:25.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 >> 172.21.15.57:0/1058476121 conn(0x7f10200fc470 msgr2=0x7f1020102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:25.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 shutdown_connections 2024-04-21T08:42:25.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:25.105+0000 7f1026d5f640 1 -- 172.21.15.57:0/1058476121 wait complete. 2024-04-21T08:42:25.423 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:42:25 smithi057 systemd[1]: Started Ceph mds.j for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1058476121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:25.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:25 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1058476121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:25 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1058476121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:25 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:26 smithi057 ceph-mon[27968]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:26 smithi057 ceph-mon[27968]: Deploying daemon mds.d on smithi057 2024-04-21T08:42:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:26 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.57:6834/2211232297,v1:172.21.15.57:6835/2211232297] up:boot 2024-04-21T08:42:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:26 smithi057 ceph-mon[27968]: fsmap 3 up:standby 2024-04-21T08:42:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:26 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:42:26.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:26 smithi136 ceph-mon[31791]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:26.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:26 smithi136 ceph-mon[31791]: Deploying daemon mds.d on smithi057 2024-04-21T08:42:26.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:26 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.57:6834/2211232297,v1:172.21.15.57:6835/2211232297] up:boot 2024-04-21T08:42:26.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:26 smithi136 ceph-mon[31791]: fsmap 3 up:standby 2024-04-21T08:42:26.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:26 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:42:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:26 smithi129 ceph-mon[31272]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:26 smithi129 ceph-mon[31272]: Deploying daemon mds.d on smithi057 2024-04-21T08:42:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:26 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.57:6834/2211232297,v1:172.21.15.57:6835/2211232297] up:boot 2024-04-21T08:42:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:26 smithi129 ceph-mon[31272]: fsmap 3 up:standby 2024-04-21T08:42:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:26 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:42:27.118 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/3422566812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8100e70 msgr2=0x7fcbb8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/3422566812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8100e70 0x7fcbb8101290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009900 tx=0x7fcba802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/3422566812 shutdown_connections 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/3422566812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8102980 0x7fcbb81092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/3422566812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbb8101fc0 0x7fcbb8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/3422566812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8100e70 0x7fcbb8101290 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/3422566812 >> 172.21.15.57:0/3422566812 conn(0x7fcbb80fc3e0 msgr2=0x7fcbb80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/3422566812 shutdown_connections 2024-04-21T08:42:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.243+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/3422566812 wait complete. 2024-04-21T08:42:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 Processor -- start 2024-04-21T08:42:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 -- start start 2024-04-21T08:42:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8100e70 0x7fcbb819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbb8101fc0 0x7fcbb819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8102980 0x7fcbb81a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb810c9a0 con 0x7fcbb8102980 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb810c820 con 0x7fcbb8101fc0 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbbc9f4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb810cb20 con 0x7fcbb8100e70 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbb6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8102980 0x7fcbb81a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbb659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8100e70 0x7fcbb819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbb6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8102980 0x7fcbb81a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34436/0 (socket says 172.21.15.57:34436) 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbb6d9d640 1 -- 172.21.15.57:0/1487919332 learned_addr learned my addr 172.21.15.57:0/1487919332 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.244+0000 7fcbb5d9b640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbb8101fc0 0x7fcbb819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbb659c640 1 -- 172.21.15.57:0/1487919332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbb8101fc0 msgr2=0x7fcbb819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbb659c640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbb8101fc0 0x7fcbb819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbb659c640 1 -- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8102980 msgr2=0x7fcbb81a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbb659c640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8102980 0x7fcbb81a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbb659c640 1 -- 172.21.15.57:0/1487919332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb81a1fb0 con 0x7fcbb8100e70 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbb659c640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8100e70 0x7fcbb819a390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcba8000c00 tx=0x7fcba8005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8043020 con 0x7fcbb8100e70 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb81a2240 con 0x7fcbb8100e70 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba8004350 con 0x7fcbb8100e70 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba8003b50 con 0x7fcbb8100e70 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.245+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb81a27e0 con 0x7fcbb8100e70 2024-04-21T08:42:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.246+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb81a29e0 con 0x7fcbb8100e70 2024-04-21T08:42:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.247+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcba8046020 con 0x7fcbb8100e70 2024-04-21T08:42:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.248+0000 7fcbaf7fe640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcba40a3bb0 0x7fcba40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.248+0000 7fcbb5d9b640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcba40a3bb0 0x7fcba40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.248+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcba80f2590 con 0x7fcbb8100e70 2024-04-21T08:42:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.249+0000 7fcbb5d9b640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcba40a3bb0 0x7fcba40a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcba00095f0 tx=0x7fcba0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:27.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.252+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcba804d020 con 0x7fcbb8100e70 2024-04-21T08:42:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.489+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbb81a2bb0 con 0x7fcbb8100e70 2024-04-21T08:42:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.490+0000 7fcbaf7fe640 1 -- 172.21.15.57:0/1487919332 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fcba80bf410 con 0x7fcbb8100e70 2024-04-21T08:42:27.491 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:27.491 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:42:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 4 2024-04-21T08:42:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcba40a3bb0 msgr2=0x7fcba40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcba40a3bb0 0x7fcba40a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcba00095f0 tx=0x7fcba0005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8100e70 msgr2=0x7fcbb819a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8100e70 0x7fcbb819a390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcba8000c00 tx=0x7fcba8005dc0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 shutdown_connections 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcba40a3bb0 0x7fcba40a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbb8102980 0x7fcbb81a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbb8101fc0 0x7fcbb819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 --2- 172.21.15.57:0/1487919332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcbb8100e70 0x7fcbb819a390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 >> 172.21.15.57:0/1487919332 conn(0x7fcbb80fc3e0 msgr2=0x7fcbb80fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 shutdown_connections 2024-04-21T08:42:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:27.492+0000 7fcbbc9f4640 1 -- 172.21.15.57:0/1487919332 wait complete. 2024-04-21T08:42:27.781 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:27 smithi057 systemd[1]: Starting Ceph mds.d for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:28.051 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:27 smithi057 podman[69452]: 2024-04-21 08:42:27.685638325 +0000 UTC m=+0.018422239 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:28.052 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:27 smithi057 podman[69452]: 2024-04-21 08:42:27.806370748 +0000 UTC m=+0.139154635 container create daf41631cee289a1e40e2048697b2d14bd4d2836f184b6d4247fc888d6016fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T08:42:28.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:28 smithi057 podman[69452]: 2024-04-21 08:42:28.167486355 +0000 UTC m=+0.500270247 container init daf41631cee289a1e40e2048697b2d14bd4d2836f184b6d4247fc888d6016fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:42:28.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:28 smithi057 podman[69452]: 2024-04-21 08:42:28.170260773 +0000 UTC m=+0.503044668 container start daf41631cee289a1e40e2048697b2d14bd4d2836f184b6d4247fc888d6016fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-04-21T08:42:28.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mds[69471]: -- 172.21.15.57:0/3329975749 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560fb8c97a40 con 0x560fb8c6b000 2024-04-21T08:42:28.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d[69467]: starting mds.d at 2024-04-21T08:42:28.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:28 smithi057 bash[69452]: daf41631cee289a1e40e2048697b2d14bd4d2836f184b6d4247fc888d6016fdd 2024-04-21T08:42:28.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:42:28 smithi057 systemd[1]: Started Ceph mds.d for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1487919332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:28.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:28.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:28 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:28.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1487919332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:28 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1487919332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:28 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:29.504 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:29.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:29 smithi129 ceph-mon[31272]: Deploying daemon mds.e on smithi129 2024-04-21T08:42:29.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:29 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.57:6836/2986531851,v1:172.21.15.57:6837/2986531851] up:boot 2024-04-21T08:42:29.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:29 smithi129 ceph-mon[31272]: fsmap 4 up:standby 2024-04-21T08:42:29.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:29 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 -- 172.21.15.57:0/1318725036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac103ab0 msgr2=0x7f51ac10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 --2- 172.21.15.57:0/1318725036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac103ab0 0x7f51ac10a3d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009ca0 tx=0x7f51a402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 -- 172.21.15.57:0/1318725036 shutdown_connections 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 --2- 172.21.15.57:0/1318725036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac103ab0 0x7f51ac10a3d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 --2- 172.21.15.57:0/1318725036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac1030f0 0x7f51ac103570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 --2- 172.21.15.57:0/1318725036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51ac101f20 0x7f51ac102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 -- 172.21.15.57:0/1318725036 >> 172.21.15.57:0/1318725036 conn(0x7f51ac0fd500 msgr2=0x7f51ac0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 -- 172.21.15.57:0/1318725036 shutdown_connections 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.624+0000 7f51b413d640 1 -- 172.21.15.57:0/1318725036 wait complete. 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 Processor -- start 2024-04-21T08:42:29.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 -- start start 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 0x7f51ac1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51ac1030f0 0x7f51ac1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac103ab0 0x7f51ac19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac10da90 con 0x7f51ac101f20 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac10d910 con 0x7f51ac103ab0 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b413d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f51ac10dc10 con 0x7f51ac1030f0 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 0x7f51ac1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 0x7f51ac1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34464/0 (socket says 172.21.15.57:34464) 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 -- 172.21.15.57:0/959110929 learned_addr learned my addr 172.21.15.57:0/959110929 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b26da640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac103ab0 0x7f51ac19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b16d8640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51ac1030f0 0x7f51ac1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 -- 172.21.15.57:0/959110929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51ac1030f0 msgr2=0x7f51ac1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51ac1030f0 0x7f51ac1985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 -- 172.21.15.57:0/959110929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac103ab0 msgr2=0x7f51ac19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac103ab0 0x7f51ac19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b1ed9640 1 -- 172.21.15.57:0/959110929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ac19fcd0 con 0x7f51ac101f20 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.625+0000 7f51b26da640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac103ab0 0x7f51ac19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.626+0000 7f51b1ed9640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 0x7f51ac1980b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f51a800d950 tx=0x7f51a800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.626+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a8017070 con 0x7f51ac101f20 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.626+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a8009c30 con 0x7f51ac101f20 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.626+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ac19ffc0 con 0x7f51ac101f20 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.626+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a800f620 con 0x7f51ac101f20 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.626+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ac1a03d0 con 0x7f51ac101f20 2024-04-21T08:42:29.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.627+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51ac1092b0 con 0x7f51ac101f20 2024-04-21T08:42:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.628+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f51a801a070 con 0x7f51ac101f20 2024-04-21T08:42:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.629+0000 7f51a2ffd640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51980a3bb0 0x7f51980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.629+0000 7f51b16d8640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51980a3bb0 0x7f51980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.629+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f51a80cfad0 con 0x7f51ac101f20 2024-04-21T08:42:29.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.629+0000 7f51b16d8640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51980a3bb0 0x7f51980a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51ac1995c0 tx=0x7f519c005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:29.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.632+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51a8009de0 con 0x7f51ac101f20 2024-04-21T08:42:29.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:29 smithi057 ceph-mon[27968]: Deploying daemon mds.e on smithi129 2024-04-21T08:42:29.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:29 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.57:6836/2986531851,v1:172.21.15.57:6837/2986531851] up:boot 2024-04-21T08:42:29.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:29 smithi057 ceph-mon[27968]: fsmap 4 up:standby 2024-04-21T08:42:29.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:29 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:42:29.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.859+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51ac02d910 con 0x7f51ac101f20 2024-04-21T08:42:29.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.859+0000 7f51a2ffd640 1 -- 172.21.15.57:0/959110929 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f51ac02d910 con 0x7f51ac101f20 2024-04-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-21T08:42:29.859 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 5 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51980a3bb0 msgr2=0x7f51980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51980a3bb0 0x7f51980a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51ac1995c0 tx=0x7f519c005cc0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 msgr2=0x7f51ac1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 0x7f51ac1980b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f51a800d950 tx=0x7f51a800de10 comp rx=0 tx=0).stop 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 shutdown_connections 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51980a3bb0 0x7f51980a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51ac103ab0 0x7f51ac19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51ac1030f0 0x7f51ac1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 --2- 172.21.15.57:0/959110929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51ac101f20 0x7f51ac1980b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 >> 172.21.15.57:0/959110929 conn(0x7f51ac0fd500 msgr2=0x7f51ac106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 shutdown_connections 2024-04-21T08:42:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:29.861+0000 7f51b413d640 1 -- 172.21.15.57:0/959110929 wait complete. 2024-04-21T08:42:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:29 smithi136 ceph-mon[31791]: Deploying daemon mds.e on smithi129 2024-04-21T08:42:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:29 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.57:6836/2986531851,v1:172.21.15.57:6837/2986531851] up:boot 2024-04-21T08:42:29.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:29 smithi136 ceph-mon[31791]: fsmap 4 up:standby 2024-04-21T08:42:29.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:29 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:42:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:30 smithi129 ceph-mon[31272]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:30 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/959110929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.573 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:30 smithi129 systemd[1]: Starting Ceph mds.e for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:30.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:30 smithi057 ceph-mon[27968]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:30.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:30 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/959110929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:30 smithi136 ceph-mon[31791]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:30 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/959110929' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:31.014 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:30 smithi129 podman[60460]: 2024-04-21 08:42:30.627930585 +0000 UTC m=+0.017526456 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:31.014 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:30 smithi129 podman[60460]: 2024-04-21 08:42:30.75402969 +0000 UTC m=+0.143625548 container create 0317caf43338e5d935327b295f4223bc7fe4f5b06aa4519c56a058e3cf1cc127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True) 2024-04-21T08:42:31.014 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:31 smithi129 podman[60460]: 2024-04-21 08:42:31.015068212 +0000 UTC m=+0.404664070 container init 0317caf43338e5d935327b295f4223bc7fe4f5b06aa4519c56a058e3cf1cc127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:42:31.323 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:31 smithi129 podman[60460]: 2024-04-21 08:42:31.017793151 +0000 UTC m=+0.407389020 container start 0317caf43338e5d935327b295f4223bc7fe4f5b06aa4519c56a058e3cf1cc127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-21T08:42:31.323 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:31 smithi129 ceph-mds[60479]: -- 172.21.15.129:0/884100618 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647e209da40 con 0x5647e2070c00 2024-04-21T08:42:31.323 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:31 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e[60475]: starting mds.e at 2024-04-21T08:42:31.323 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:31 smithi129 bash[60460]: 0317caf43338e5d935327b295f4223bc7fe4f5b06aa4519c56a058e3cf1cc127 2024-04-21T08:42:31.324 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:42:31 smithi129 systemd[1]: Started Ceph mds.e for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:31.873 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 -- 172.21.15.57:0/3831252815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 msgr2=0x7fdc3c10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/3831252815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c10a3d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009ca0 tx=0x7fdc3402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 -- 172.21.15.57:0/3831252815 shutdown_connections 2024-04-21T08:42:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/3831252815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c10a3d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/3831252815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc3c103180 0x7fdc3c103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/3831252815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc3c101f90 0x7fdc3c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 -- 172.21.15.57:0/3831252815 >> 172.21.15.57:0/3831252815 conn(0x7fdc3c0fd500 msgr2=0x7fdc3c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.988+0000 7fdc42a5c640 1 -- 172.21.15.57:0/3831252815 shutdown_connections 2024-04-21T08:42:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 -- 172.21.15.57:0/3831252815 wait complete. 2024-04-21T08:42:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 Processor -- start 2024-04-21T08:42:31.989 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 -- start start 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc3c101f90 0x7fdc3c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc3c103180 0x7fdc3c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10dad0 con 0x7fdc3c103b40 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10d950 con 0x7fdc3c101f90 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.989+0000 7fdc42a5c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c10dc50 con 0x7fdc3c103180 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c1a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34480/0 (socket says 172.21.15.57:34480) 2024-04-21T08:42:31.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 -- 172.21.15.57:0/461079842 learned_addr learned my addr 172.21.15.57:0/461079842 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc3bfff640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc3c101f90 0x7fdc3c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc3b7fe640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc3c103180 0x7fdc3c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 -- 172.21.15.57:0/461079842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc3c103180 msgr2=0x7fdc3c19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc3c103180 0x7fdc3c19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 -- 172.21.15.57:0/461079842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc3c101f90 msgr2=0x7fdc3c19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc3c101f90 0x7fdc3c19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 -- 172.21.15.57:0/461079842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c1a1fb0 con 0x7fdc3c103b40 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc40ff9640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c1a1940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc3402c700 tx=0x7fdc34034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34039040 con 0x7fdc3c103b40 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc34044070 con 0x7fdc3c103b40 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c1a2240 con 0x7fdc3c103b40 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc340042a0 con 0x7fdc3c103b40 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.990+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1a2780 con 0x7fdc3c103b40 2024-04-21T08:42:31.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.991+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c1a29e0 con 0x7fdc3c103b40 2024-04-21T08:42:31.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.993+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdc34003970 con 0x7fdc3c103b40 2024-04-21T08:42:31.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.993+0000 7fdc397fa640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdc280a3c30 0x7fdc280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:31.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.993+0000 7fdc3bfff640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdc280a3c30 0x7fdc280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:31.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.993+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdc340f23a0 con 0x7fdc3c103b40 2024-04-21T08:42:31.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.994+0000 7fdc3bfff640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdc280a3c30 0x7fdc280a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005fd0 tx=0x7fdc2c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:31.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:31.998+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc340bf1c0 con 0x7fdc3c103b40 2024-04-21T08:42:32.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.226+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc3c1a2bb0 con 0x7fdc3c103b40 2024-04-21T08:42:32.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.227+0000 7fdc397fa640 1 -- 172.21.15.57:0/461079842 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7fdc340beac0 con 0x7fdc3c103b40 2024-04-21T08:42:32.226 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:32.226 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-21T08:42:32.227 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 6 2024-04-21T08:42:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdc280a3c30 msgr2=0x7fdc280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdc280a3c30 0x7fdc280a60f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005fd0 tx=0x7fdc2c005850 comp rx=0 tx=0).stop 2024-04-21T08:42:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 msgr2=0x7fdc3c1a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c1a1940 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc3402c700 tx=0x7fdc34034760 comp rx=0 tx=0).stop 2024-04-21T08:42:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 shutdown_connections 2024-04-21T08:42:32.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdc280a3c30 0x7fdc280a60f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdc3c103b40 0x7fdc3c1a1940 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdc3c103180 0x7fdc3c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 --2- 172.21.15.57:0/461079842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc3c101f90 0x7fdc3c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 >> 172.21.15.57:0/461079842 conn(0x7fdc3c0fd500 msgr2=0x7fdc3c074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 shutdown_connections 2024-04-21T08:42:32.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:32.229+0000 7fdc42a5c640 1 -- 172.21.15.57:0/461079842 wait complete. 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:32.238 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: Deploying daemon mds.f on smithi136 2024-04-21T08:42:32.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.129:6834/2158348740,v1:172.21.15.129:6835/2158348740] up:boot 2024-04-21T08:42:32.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: fsmap 5 up:standby 2024-04-21T08:42:32.239 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:32 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:42:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:32.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:32.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: Deploying daemon mds.f on smithi136 2024-04-21T08:42:32.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.129:6834/2158348740,v1:172.21.15.129:6835/2158348740] up:boot 2024-04-21T08:42:32.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: fsmap 5 up:standby 2024-04-21T08:42:32.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:32 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:42:32.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:32.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:32.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:32.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:32.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:32.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: Deploying daemon mds.f on smithi136 2024-04-21T08:42:32.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.129:6834/2158348740,v1:172.21.15.129:6835/2158348740] up:boot 2024-04-21T08:42:32.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: fsmap 5 up:standby 2024-04-21T08:42:32.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:32 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:42:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:33 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/461079842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:33.385 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 systemd[1]: Starting Ceph mds.f for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:33.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:33 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/461079842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:33.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:33 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/461079842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:33.801 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 podman[61428]: 2024-04-21 08:42:33.436994781 +0000 UTC m=+0.017876940 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:33.801 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 podman[61428]: 2024-04-21 08:42:33.557211248 +0000 UTC m=+0.138093450 container create 445152d84157217d263c824dc90638efd7e4f251e452af109d952007b47a277b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T08:42:33.801 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 podman[61428]: 2024-04-21 08:42:33.776476242 +0000 UTC m=+0.357358385 container init 445152d84157217d263c824dc90638efd7e4f251e452af109d952007b47a277b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:42:33.801 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 podman[61428]: 2024-04-21 08:42:33.779113136 +0000 UTC m=+0.359995281 container start 445152d84157217d263c824dc90638efd7e4f251e452af109d952007b47a277b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-21T08:42:33.801 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 ceph-mds[61447]: -- 172.21.15.136:0/1442688450 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561945b23a40 con 0x561945af6800 2024-04-21T08:42:34.135 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f[61443]: starting mds.f at 2024-04-21T08:42:34.135 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 bash[61428]: 445152d84157217d263c824dc90638efd7e4f251e452af109d952007b47a277b 2024-04-21T08:42:34.135 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:42:33 smithi136 systemd[1]: Started Ceph mds.f for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:34.240 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:34.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:34.325 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:34.357 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 -- 172.21.15.57:0/609339935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c107ce0 msgr2=0x7f378c10a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 --2- 172.21.15.57:0/609339935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c107ce0 0x7f378c10a0d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3784009ca0 tx=0x7f378402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 -- 172.21.15.57:0/609339935 shutdown_connections 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 --2- 172.21.15.57:0/609339935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c107ce0 0x7f378c10a0d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 --2- 172.21.15.57:0/609339935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f378c1053b0 0x7f378c1077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 --2- 172.21.15.57:0/609339935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f378c102a80 0x7f378c104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 -- 172.21.15.57:0/609339935 >> 172.21.15.57:0/609339935 conn(0x7f378c0fc410 msgr2=0x7f378c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 -- 172.21.15.57:0/609339935 shutdown_connections 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.356+0000 7f3794584640 1 -- 172.21.15.57:0/609339935 wait complete. 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 Processor -- start 2024-04-21T08:42:34.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 -- start start 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 0x7f378c1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f378c1053b0 0x7f378c100a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f378c107ce0 0x7f378c100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f378c10dae0 con 0x7f378c102a80 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f378c10d960 con 0x7f378c107ce0 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3794584640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f378c10dc60 con 0x7f378c1053b0 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 0x7f378c1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 0x7f378c1023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37352/0 (socket says 172.21.15.57:37352) 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 -- 172.21.15.57:0/716188149 learned_addr learned my addr 172.21.15.57:0/716188149 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:34.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3791b1f640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f378c1053b0 0x7f378c100a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792b21640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f378c107ce0 0x7f378c100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 -- 172.21.15.57:0/716188149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f378c1053b0 msgr2=0x7f378c100a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f378c1053b0 0x7f378c100a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 -- 172.21.15.57:0/716188149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f378c107ce0 msgr2=0x7f378c100f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f378c107ce0 0x7f378c100f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 -- 172.21.15.57:0/716188149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c101800 con 0x7f378c102a80 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.357+0000 7f3792320640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 0x7f378c1023f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f377c00e9f0 tx=0x7f377c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.358+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c00cd80 con 0x7f378c102a80 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.358+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378c1a2280 con 0x7f378c102a80 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.358+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f377c0054d0 con 0x7f378c102a80 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.358+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c010690 con 0x7f378c102a80 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.358+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378c1a2780 con 0x7f378c102a80 2024-04-21T08:42:34.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.359+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378c0fdf00 con 0x7f378c102a80 2024-04-21T08:42:34.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.360+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f377c004e90 con 0x7f378c102a80 2024-04-21T08:42:34.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.360+0000 7f37837fe640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f37780a3bb0 0x7f37780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:34.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.360+0000 7f3791b1f640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f37780a3bb0 0x7f37780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:34.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.361+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f377c0d0460 con 0x7f378c102a80 2024-04-21T08:42:34.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.361+0000 7f3791b1f640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f37780a3bb0 0x7f37780a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f378c102160 tx=0x7f378800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:34.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.365+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f377c09d2e0 con 0x7f378c102a80 2024-04-21T08:42:34.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.594+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f378c02d5c0 con 0x7f378c102a80 2024-04-21T08:42:34.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.594+0000 7f37837fe640 1 -- 172.21.15.57:0/716188149 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f377c09cbe0 con 0x7f378c102a80 2024-04-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:42:34.594 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 7 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f37780a3bb0 msgr2=0x7f37780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f37780a3bb0 0x7f37780a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f378c102160 tx=0x7f378800b410 comp rx=0 tx=0).stop 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 msgr2=0x7f378c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 0x7f378c1023f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f377c00e9f0 tx=0x7f377c00edf0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 shutdown_connections 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f37780a3bb0 0x7f37780a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f378c107ce0 0x7f378c100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f378c1053b0 0x7f378c100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 --2- 172.21.15.57:0/716188149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c102a80 0x7f378c1023f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:34.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 >> 172.21.15.57:0/716188149 conn(0x7f378c0fc410 msgr2=0x7f378c1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:34.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 shutdown_connections 2024-04-21T08:42:34.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:34.596+0000 7f3794584640 1 -- 172.21.15.57:0/716188149 wait complete. 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:34.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:35 smithi136 ceph-mon[31791]: Deploying daemon mds.c on smithi136 2024-04-21T08:42:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:35 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.136:6834/1669978943,v1:172.21.15.136:6835/1669978943] up:boot 2024-04-21T08:42:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:35 smithi136 ceph-mon[31791]: fsmap 6 up:standby 2024-04-21T08:42:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:42:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:35 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/716188149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:35.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:35 smithi129 ceph-mon[31272]: Deploying daemon mds.c on smithi136 2024-04-21T08:42:35.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:35 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.136:6834/1669978943,v1:172.21.15.136:6835/1669978943] up:boot 2024-04-21T08:42:35.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:35 smithi129 ceph-mon[31272]: fsmap 6 up:standby 2024-04-21T08:42:35.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:42:35.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:35 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/716188149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:35 smithi057 ceph-mon[27968]: Deploying daemon mds.c on smithi136 2024-04-21T08:42:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:35 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.136:6834/1669978943,v1:172.21.15.136:6835/1669978943] up:boot 2024-04-21T08:42:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:35 smithi057 ceph-mon[27968]: fsmap 6 up:standby 2024-04-21T08:42:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:42:35.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:35 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/716188149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:36.172 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 systemd[1]: Starting Ceph mds.c for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:36.426 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:36 smithi136 ceph-mon[31791]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:36.426 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 podman[61736]: 2024-04-21 08:42:36.205620757 +0000 UTC m=+0.018192205 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:36.426 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 podman[61736]: 2024-04-21 08:42:36.313245061 +0000 UTC m=+0.125816511 container create ea94fe2f9854badfa6beace68724735db789dd260dadb89e969026d7e64c2d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:42:36.490 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:36 smithi057 ceph-mon[27968]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:36 smithi129 ceph-mon[31272]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:36.608 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.725+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/2141555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 msgr2=0x7ff2a41109f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.725+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/2141555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41109f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff29c009ca0 tx=0x7ff29c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/2141555558 shutdown_connections 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/2141555558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41109f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/2141555558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2a41097a0 0x7ff2a4109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/2141555558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2a41085b0 0x7ff2a41089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/2141555558 >> 172.21.15.57:0/2141555558 conn(0x7ff2a4077560 msgr2=0x7ff2a4079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/2141555558 shutdown_connections 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/2141555558 wait complete. 2024-04-21T08:42:36.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 Processor -- start 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.726+0000 7ff2ac1ce640 1 -- start start 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2ac1ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2a41085b0 0x7ff2a419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2ac1ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2a41097a0 0x7ff2a419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2ac1ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2ac1ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a41140e0 con 0x7ff2a410a160 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2ac1ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a4113f60 con 0x7ff2a41097a0 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2ac1ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a4114260 con 0x7ff2a41085b0 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37366/0 (socket says 172.21.15.57:37366) 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 -- 172.21.15.57:0/4058622813 learned_addr learned my addr 172.21.15.57:0/4058622813 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 -- 172.21.15.57:0/4058622813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2a41085b0 msgr2=0x7ff2a419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2a41085b0 0x7ff2a419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 -- 172.21.15.57:0/4058622813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2a41097a0 msgr2=0x7ff2a419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2a41097a0 0x7ff2a419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 -- 172.21.15.57:0/4058622813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a41a6380 con 0x7ff2a410a160 2024-04-21T08:42:36.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.727+0000 7ff2aa76b640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41a5d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff29c02c700 tx=0x7ff29c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.728+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c041020 con 0x7ff2a410a160 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.728+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff29c034d60 con 0x7ff2a410a160 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.728+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a41a6610 con 0x7ff2a410a160 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.728+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c039040 con 0x7ff2a410a160 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.728+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a41a6b50 con 0x7ff2a410a160 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.729+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2a410f8d0 con 0x7ff2a410a160 2024-04-21T08:42:36.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.731+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff29c044020 con 0x7ff2a410a160 2024-04-21T08:42:36.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.731+0000 7ff29affd640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2900a3c30 0x7ff2900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:36.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.731+0000 7ff2a9f6a640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2900a3c30 0x7ff2900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:36.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.731+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff29c0f2370 con 0x7ff2a410a160 2024-04-21T08:42:36.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.732+0000 7ff2a9f6a640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2900a3c30 0x7ff2900a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005fd0 tx=0x7ff2a000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:36.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.735+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff29c04b020 con 0x7ff2a410a160 2024-04-21T08:42:36.885 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 podman[61736]: 2024-04-21 08:42:36.549713133 +0000 UTC m=+0.362284580 container init ea94fe2f9854badfa6beace68724735db789dd260dadb89e969026d7e64c2d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T08:42:36.885 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 podman[61736]: 2024-04-21 08:42:36.552399446 +0000 UTC m=+0.364970900 container start ea94fe2f9854badfa6beace68724735db789dd260dadb89e969026d7e64c2d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:42:36.885 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 ceph-mds[61755]: -- 172.21.15.136:0/3331335793 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7d99d5a40 con 0x55f7d99a9000 2024-04-21T08:42:36.886 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c[61751]: starting mds.c at 2024-04-21T08:42:36.886 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 bash[61736]: ea94fe2f9854badfa6beace68724735db789dd260dadb89e969026d7e64c2d51 2024-04-21T08:42:36.886 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:42:36 smithi136 systemd[1]: Started Ceph mds.c for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:36.972 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.972+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2a402d5c0 con 0x7ff2a410a160 2024-04-21T08:42:36.972 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.973+0000 7ff29affd640 1 -- 172.21.15.57:0/4058622813 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7ff29c0bf1f0 con 0x7ff2a410a160 2024-04-21T08:42:36.972 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:36.973 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-21T08:42:36.973 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 7 2024-04-21T08:42:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2900a3c30 msgr2=0x7ff2900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2900a3c30 0x7ff2900a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005fd0 tx=0x7ff2a000b410 comp rx=0 tx=0).stop 2024-04-21T08:42:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 msgr2=0x7ff2a41a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41a5d10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff29c02c700 tx=0x7ff29c0093f0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 shutdown_connections 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2900a3c30 0x7ff2900a60f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2a410a160 0x7ff2a41a5d10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2a41097a0 0x7ff2a419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 --2- 172.21.15.57:0/4058622813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2a41085b0 0x7ff2a419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.975+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 >> 172.21.15.57:0/4058622813 conn(0x7ff2a4077560 msgr2=0x7ff2a410caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.976+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 shutdown_connections 2024-04-21T08:42:36.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:36.976+0000 7ff2ac1ce640 1 -- 172.21.15.57:0/4058622813 wait complete. 2024-04-21T08:42:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:37.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:37.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:37.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:37.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: Deploying daemon mds.h on smithi129 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4058622813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.136:6836/1005985303,v1:172.21.15.136:6837/1005985303] up:boot 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: fsmap 7 up:standby 2024-04-21T08:42:37.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:37 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:42:38.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: Deploying daemon mds.h on smithi129 2024-04-21T08:42:38.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4058622813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:38.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.136:6836/1005985303,v1:172.21.15.136:6837/1005985303] up:boot 2024-04-21T08:42:38.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: fsmap 7 up:standby 2024-04-21T08:42:38.076 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:37 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:42:38.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: Deploying daemon mds.h on smithi129 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4058622813' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.136:6836/1005985303,v1:172.21.15.136:6837/1005985303] up:boot 2024-04-21T08:42:38.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: fsmap 7 up:standby 2024-04-21T08:42:38.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:37 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:42:38.987 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 -- 172.21.15.57:0/3090604697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 msgr2=0x7f8ec010a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 --2- 172.21.15.57:0/3090604697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec010a0d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009ca0 tx=0x7f8eb802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 -- 172.21.15.57:0/3090604697 shutdown_connections 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 --2- 172.21.15.57:0/3090604697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec010a0d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 --2- 172.21.15.57:0/3090604697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 0x7f8ec0107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 --2- 172.21.15.57:0/3090604697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec0102a20 0x7f8ec0104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 -- 172.21.15.57:0/3090604697 >> 172.21.15.57:0/3090604697 conn(0x7f8ec00fc410 msgr2=0x7f8ec00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 -- 172.21.15.57:0/3090604697 shutdown_connections 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.105+0000 7f8ec5843640 1 -- 172.21.15.57:0/3090604697 wait complete. 2024-04-21T08:42:39.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 Processor -- start 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 -- start start 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec0102a20 0x7f8ec0102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 0x7f8ec01009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec0100f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec010dae0 con 0x7f8ec0107c80 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec010d960 con 0x7f8ec0105350 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ec5843640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec010dc60 con 0x7f8ec0102a20 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ebf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec0100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ebf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec0100f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37372/0 (socket says 172.21.15.57:37372) 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.106+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/2484984099 learned_addr learned my addr 172.21.15.57:0/2484984099 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebe7fc640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 0x7f8ec01009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/2484984099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec0102a20 msgr2=0x7f8ec0102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebf7fe640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec0102a20 0x7f8ec0102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/2484984099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 msgr2=0x7f8ec01009d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebf7fe640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 0x7f8ec01009d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebf7fe640 1 -- 172.21.15.57:0/2484984099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec01017a0 con 0x7f8ec0107c80 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebe7fc640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 0x7f8ec01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ebf7fe640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec0100f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8eb802c700 tx=0x7f8eb8034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8039040 con 0x7f8ec0107c80 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eb8044070 con 0x7f8ec0107c80 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec01a21d0 con 0x7f8ec0107c80 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb80042a0 con 0x7f8ec0107c80 2024-04-21T08:42:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.107+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec01a2720 con 0x7f8ec0107c80 2024-04-21T08:42:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.108+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec00fdf00 con 0x7f8ec0107c80 2024-04-21T08:42:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.109+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8eb8003970 con 0x7f8ec0107c80 2024-04-21T08:42:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.110+0000 7f8ec4841640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eac0a3bb0 0x7f8eac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.110+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8eb80f2e70 con 0x7f8ec0107c80 2024-04-21T08:42:39.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.110+0000 7f8ebeffd640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eac0a3bb0 0x7f8eac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:39.111 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.111+0000 7f8ebeffd640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eac0a3bb0 0x7f8eac0a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000a8f0 tx=0x7f8eb0005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:39.112 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.113+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8eb80bfcf0 con 0x7f8ec0107c80 2024-04-21T08:42:39.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.343+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ec002d5c0 con 0x7f8ec0107c80 2024-04-21T08:42:39.343 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.344+0000 7f8ec4841640 1 -- 172.21.15.57:0/2484984099 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6358 (secure 0 0 0) 0x7f8eb80bf5f0 con 0x7f8ec0107c80 2024-04-21T08:42:39.343 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:39.343 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:42:39.344 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 8 2024-04-21T08:42:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eac0a3bb0 msgr2=0x7f8eac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eac0a3bb0 0x7f8eac0a6070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000a8f0 tx=0x7f8eb0005bf0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 msgr2=0x7f8ec0100f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:39.345 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec0100f10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8eb802c700 tx=0x7f8eb8034760 comp rx=0 tx=0).stop 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 shutdown_connections 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eac0a3bb0 0x7f8eac0a6070 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec0107c80 0x7f8ec0100f10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec0105350 0x7f8ec01009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 --2- 172.21.15.57:0/2484984099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec0102a20 0x7f8ec0102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 >> 172.21.15.57:0/2484984099 conn(0x7f8ec00fc410 msgr2=0x7f8ec0103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 shutdown_connections 2024-04-21T08:42:39.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:39.346+0000 7f8ec5843640 1 -- 172.21.15.57:0/2484984099 wait complete. 2024-04-21T08:42:40.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:39 smithi129 ceph-mon[31272]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:40.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:39 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2484984099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:39 smithi057 ceph-mon[27968]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:40.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:39 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2484984099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:40.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:39 smithi136 ceph-mon[31791]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:40.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:39 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2484984099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.129:6836/1005091659,v1:172.21.15.129:6837/1005091659] up:boot 2024-04-21T08:42:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: fsmap 8 up:standby 2024-04-21T08:42:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:42:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:41.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:41.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:40 smithi129 ceph-mon[31272]: Deploying daemon mds.i on smithi136 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.129:6836/1005091659,v1:172.21.15.129:6837/1005091659] up:boot 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: fsmap 8 up:standby 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:40 smithi057 ceph-mon[27968]: Deploying daemon mds.i on smithi136 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.129:6836/1005091659,v1:172.21.15.129:6837/1005091659] up:boot 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: fsmap 8 up:standby 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:41.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:41.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:40 smithi136 ceph-mon[31791]: Deploying daemon mds.i on smithi136 2024-04-21T08:42:41.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:41.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 -- 172.21.15.57:0/505964006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 msgr2=0x7fd5241092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 --2- 172.21.15.57:0/505964006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd5241092d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009ca0 tx=0x7fd51c02c100 comp rx=0 tx=0).stop 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 -- 172.21.15.57:0/505964006 shutdown_connections 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 --2- 172.21.15.57:0/505964006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd5241092d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 --2- 172.21.15.57:0/505964006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd524101ff0 0x7fd524102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 --2- 172.21.15.57:0/505964006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd524100ea0 0x7fd5241012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 -- 172.21.15.57:0/505964006 >> 172.21.15.57:0/505964006 conn(0x7fd5240fc410 msgr2=0x7fd5240fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 -- 172.21.15.57:0/505964006 shutdown_connections 2024-04-21T08:42:41.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.472+0000 7fd528fa4640 1 -- 172.21.15.57:0/505964006 wait complete. 2024-04-21T08:42:41.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 Processor -- start 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 -- start start 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd524100ea0 0x7fd524198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd524101ff0 0x7fd524198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd52419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd52410db10 con 0x7fd5241029b0 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd52410d990 con 0x7fd524101ff0 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd528fa4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd52410dc90 con 0x7fd524100ea0 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd52419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd52419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37390/0 (socket says 172.21.15.57:37390) 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 -- 172.21.15.57:0/2660920773 learned_addr learned my addr 172.21.15.57:0/2660920773 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd52259c640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd524100ea0 0x7fd524198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd521d9b640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd524101ff0 0x7fd524198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 -- 172.21.15.57:0/2660920773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd524100ea0 msgr2=0x7fd524198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd524100ea0 0x7fd524198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 -- 172.21.15.57:0/2660920773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd524101ff0 msgr2=0x7fd524198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd524101ff0 0x7fd524198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.473+0000 7fd522d9d640 1 -- 172.21.15.57:0/2660920773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52419fc40 con 0x7fd5241029b0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.474+0000 7fd522d9d640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd52419f5d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd51c02c710 tx=0x7fd51c035dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.474+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c039040 con 0x7fd5241029b0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.474+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd51c044070 con 0x7fd5241029b0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.474+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0046f0 con 0x7fd5241029b0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.474+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52419fed0 con 0x7fd5241029b0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.474+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5241a0330 con 0x7fd5241029b0 2024-04-21T08:42:41.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.475+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5241080e0 con 0x7fd5241029b0 2024-04-21T08:42:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.476+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd51c003970 con 0x7fd5241029b0 2024-04-21T08:42:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.477+0000 7fd5137fe640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd50c0a3bb0 0x7fd50c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.477+0000 7fd52259c640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd50c0a3bb0 0x7fd50c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:41.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.477+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd51c0f2950 con 0x7fd5241029b0 2024-04-21T08:42:41.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.477+0000 7fd52259c640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd50c0a3bb0 0x7fd50c0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd514005e10 tx=0x7fd514005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:41.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.481+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd51c0bf7d0 con 0x7fd5241029b0 2024-04-21T08:42:41.710 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.710+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd52402d5c0 con 0x7fd5241029b0 2024-04-21T08:42:41.710 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.711+0000 7fd5137fe640 1 -- 172.21.15.57:0/2660920773 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7fd51c0098b0 con 0x7fd5241029b0 2024-04-21T08:42:41.710 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:41.711 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-21T08:42:41.711 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 9 2024-04-21T08:42:41.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd50c0a3bb0 msgr2=0x7fd50c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd50c0a3bb0 0x7fd50c0a6070 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd514005e10 tx=0x7fd514005da0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 msgr2=0x7fd52419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd52419f5d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd51c02c710 tx=0x7fd51c035dd0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 shutdown_connections 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd50c0a3bb0 0x7fd50c0a6070 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd5241029b0 0x7fd52419f5d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd524101ff0 0x7fd524198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 --2- 172.21.15.57:0/2660920773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd524100ea0 0x7fd524198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 >> 172.21.15.57:0/2660920773 conn(0x7fd5240fc410 msgr2=0x7fd5241052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 shutdown_connections 2024-04-21T08:42:41.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:41.713+0000 7fd528fa4640 1 -- 172.21.15.57:0/2660920773 wait complete. 2024-04-21T08:42:42.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:41 smithi129 ceph-mon[31272]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:42.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:41 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2660920773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:42.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:41 smithi057 ceph-mon[27968]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:42.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:41 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2660920773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:42.122 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:41 smithi136 ceph-mon[31791]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:42.122 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:41 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2660920773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:42.122 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:41 smithi136 systemd[1]: Starting Ceph mds.i for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:42.385 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 podman[62051]: 2024-04-21 08:42:42.026089264 +0000 UTC m=+0.017849080 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:42.385 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 podman[62051]: 2024-04-21 08:42:42.145583371 +0000 UTC m=+0.137343195 container create d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-21T08:42:42.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 podman[62051]: 2024-04-21 08:42:42.405765762 +0000 UTC m=+0.397525578 container init d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:42:42.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 podman[62051]: 2024-04-21 08:42:42.408371505 +0000 UTC m=+0.400131321 container start d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T08:42:42.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 ceph-mds[62069]: -- 172.21.15.136:0/676603623 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561e69d79a40 con 0x561e69d4d000 2024-04-21T08:42:42.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i[62065]: starting mds.i at 2024-04-21T08:42:42.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 bash[62051]: d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb 2024-04-21T08:42:42.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:42:42 smithi136 systemd[1]: Started Ceph mds.i for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:43.724 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: Deploying daemon mds.g on smithi057 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.136:6838/4293224778,v1:172.21.15.136:6839/4293224778] up:boot 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: fsmap 9 up:standby 2024-04-21T08:42:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:43 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- 172.21.15.57:0/2947430436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078100e70 msgr2=0x7f2078101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- 172.21.15.57:0/2947430436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078100e70 0x7f2078101290 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2068009900 tx=0x7f206802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- 172.21.15.57:0/2947430436 shutdown_connections 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- 172.21.15.57:0/2947430436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 0x7f20781092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- 172.21.15.57:0/2947430436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2078101fc0 0x7f2078102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- 172.21.15.57:0/2947430436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078100e70 0x7f2078101290 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- 172.21.15.57:0/2947430436 >> 172.21.15.57:0/2947430436 conn(0x7f20780fc3e0 msgr2=0x7f20780fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- 172.21.15.57:0/2947430436 shutdown_connections 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- 172.21.15.57:0/2947430436 wait complete. 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 Processor -- start 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- start start 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2078100e70 0x7f207819a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 0x7f207819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 0x7f20781a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f207810dad0 con 0x7f2078101fc0 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f207810d950 con 0x7f2078102980 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.831+0000 7f207ed2c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f207810dc50 con 0x7f2078100e70 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 0x7f207819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 0x7f207819a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46304/0 (socket says 172.21.15.57:46304) 2024-04-21T08:42:43.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 -- 172.21.15.57:0/3667033050 learned_addr learned my addr 172.21.15.57:0/3667033050 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 -- 172.21.15.57:0/3667033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2078100e70 msgr2=0x7f207819a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f207d2c9640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 0x7f20781a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f207cac8640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2078100e70 0x7f207819a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2078100e70 0x7f207819a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 -- 172.21.15.57:0/3667033050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 msgr2=0x7f20781a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 0x7f20781a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 -- 172.21.15.57:0/3667033050 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20781a1f20 con 0x7f2078101fc0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f207d2c9640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 0x7f20781a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2077fff640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 0x7f207819a840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f206c00c970 tx=0x7f206c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c007c10 con 0x7f2078101fc0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f206c007db0 con 0x7f2078101fc0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20781a2210 con 0x7f2078101fc0 2024-04-21T08:42:43.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c00f870 con 0x7f2078101fc0 2024-04-21T08:42:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.832+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20781a2760 con 0x7f2078101fc0 2024-04-21T08:42:43.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.833+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20781a2a20 con 0x7f2078101fc0 2024-04-21T08:42:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.834+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f206c00fa10 con 0x7f2078101fc0 2024-04-21T08:42:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.835+0000 7f2075ffb640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20600a3bb0 0x7f20600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.835+0000 7f207cac8640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20600a3bb0 0x7f20600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.835+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f206c0d0420 con 0x7f2078101fc0 2024-04-21T08:42:43.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.836+0000 7f207cac8640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20600a3bb0 0x7f20600a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f20680096c0 tx=0x7f2068000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:43.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: Deploying daemon mds.g on smithi057 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.136:6838/4293224778,v1:172.21.15.136:6839/4293224778] up:boot 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: fsmap 9 up:standby 2024-04-21T08:42:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:43 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:42:43.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:43.840+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f206c09d290 con 0x7f2078101fc0 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: Deploying daemon mds.g on smithi057 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.136:6838/4293224778,v1:172.21.15.136:6839/4293224778] up:boot 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: fsmap 9 up:standby 2024-04-21T08:42:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:43 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:42:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.067+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20781a2bf0 con 0x7f2078101fc0 2024-04-21T08:42:44.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.068+0000 7f2075ffb640 1 -- 172.21.15.57:0/3667033050 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f206c09cb90 con 0x7f2078101fc0 2024-04-21T08:42:44.068 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:44.068 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:44.068 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 10 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20600a3bb0 msgr2=0x7f20600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20600a3bb0 0x7f20600a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f20680096c0 tx=0x7f2068000bc0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 msgr2=0x7f207819a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 0x7f207819a840 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f206c00c970 tx=0x7f206c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 shutdown_connections 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f20600a3bb0 0x7f20600a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078102980 0x7f20781a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2078101fc0 0x7f207819a840 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 --2- 172.21.15.57:0/3667033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2078100e70 0x7f207819a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.070+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 >> 172.21.15.57:0/3667033050 conn(0x7f20780fc3e0 msgr2=0x7f2078109c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.071+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 shutdown_connections 2024-04-21T08:42:44.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:44.071+0000 7f207ed2c640 1 -- 172.21.15.57:0/3667033050 wait complete. 2024-04-21T08:42:44.777 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:44 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3667033050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:44 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3667033050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.056 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 podman[70058]: 2024-04-21 08:42:44.902496516 +0000 UTC m=+0.018633180 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:45.056 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 podman[70058]: 2024-04-21 08:42:45.057745997 +0000 UTC m=+0.173882693 container create d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-21T08:42:45.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:44 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3667033050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:45.592 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 podman[70058]: 2024-04-21 08:42:45.373332609 +0000 UTC m=+0.489469261 container init d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:42:45.592 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 podman[70058]: 2024-04-21 08:42:45.376695658 +0000 UTC m=+0.492832311 container start d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-21T08:42:45.593 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mds[70078]: -- 172.21.15.57:0/2617565856 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628e9f43a40 con 0x5628e9f16c00 2024-04-21T08:42:45.593 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g[70074]: starting mds.g at 2024-04-21T08:42:45.593 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 bash[70058]: d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 2024-04-21T08:42:45.593 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:42:45 smithi057 systemd[1]: Started Ceph mds.g for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:45.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:46.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:46.080 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:46.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:46.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:46.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:46.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:46.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:46.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:46.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 -- 172.21.15.57:0/1971222146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740103b50 msgr2=0x7f874010a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 --2- 172.21.15.57:0/1971222146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740103b50 0x7f874010a3e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8738009ca0 tx=0x7f873802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 -- 172.21.15.57:0/1971222146 shutdown_connections 2024-04-21T08:42:46.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 --2- 172.21.15.57:0/1971222146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740103b50 0x7f874010a3e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 --2- 172.21.15.57:0/1971222146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8740103190 0x7f8740103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 --2- 172.21.15.57:0/1971222146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8740101fa0 0x7f87401023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.194+0000 7f8747470640 1 -- 172.21.15.57:0/1971222146 >> 172.21.15.57:0/1971222146 conn(0x7f87400fd530 msgr2=0x7f87400ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 -- 172.21.15.57:0/1971222146 shutdown_connections 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 -- 172.21.15.57:0/1971222146 wait complete. 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 Processor -- start 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 -- start start 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 0x7f8740075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8740103190 0x7f8740074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8740103b50 0x7f8740074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f874010dae0 con 0x7f8740101fa0 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f874010d960 con 0x7f8740103b50 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8747470640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f874010dc60 con 0x7f8740103190 2024-04-21T08:42:46.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f874520c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 0x7f8740075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f874520c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 0x7f8740075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46346/0 (socket says 172.21.15.57:46346) 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f874520c640 1 -- 172.21.15.57:0/970245971 learned_addr learned my addr 172.21.15.57:0/970245971 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.195+0000 7f8745a0d640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8740103b50 0x7f8740074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f874520c640 1 -- 172.21.15.57:0/970245971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8740103190 msgr2=0x7f8740074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f874520c640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8740103190 0x7f8740074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f874520c640 1 -- 172.21.15.57:0/970245971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8740103b50 msgr2=0x7f8740074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f874520c640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8740103b50 0x7f8740074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f874520c640 1 -- 172.21.15.57:0/970245971 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8740075110 con 0x7f8740101fa0 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f874520c640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 0x7f8740075d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f873c00d770 tx=0x7f873c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c017070 con 0x7f8740101fa0 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f873c009c30 con 0x7f8740101fa0 2024-04-21T08:42:46.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87401a2200 con 0x7f8740101fa0 2024-04-21T08:42:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c003cc0 con 0x7f8740101fa0 2024-04-21T08:42:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.196+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87401a2750 con 0x7f8740101fa0 2024-04-21T08:42:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.197+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8740109310 con 0x7f8740101fa0 2024-04-21T08:42:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.199+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f873c003e60 con 0x7f8740101fa0 2024-04-21T08:42:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.199+0000 7f87367fc640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f872c0a3bb0 0x7f872c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.199+0000 7f8744a0b640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f872c0a3bb0 0x7f872c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.199+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f873c0cfde0 con 0x7f8740101fa0 2024-04-21T08:42:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.200+0000 7f8744a0b640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f872c0a3bb0 0x7f872c0a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8730005fd0 tx=0x7f8730005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.203+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f873c009de0 con 0x7f8740101fa0 2024-04-21T08:42:46.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.432+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f874002d870 con 0x7f8740101fa0 2024-04-21T08:42:46.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.432+0000 7f87367fc640 1 -- 172.21.15.57:0/970245971 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7f873c09cc60 con 0x7f8740101fa0 2024-04-21T08:42:46.432 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:46.432 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:46.433 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 11 2024-04-21T08:42:46.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f872c0a3bb0 msgr2=0x7f872c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f872c0a3bb0 0x7f872c0a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8730005fd0 tx=0x7f8730005c70 comp rx=0 tx=0).stop 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 msgr2=0x7f8740075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 0x7f8740075d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f873c00d770 tx=0x7f873c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 shutdown_connections 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f872c0a3bb0 0x7f872c0a6070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8740103b50 0x7f8740074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8740103190 0x7f8740074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 --2- 172.21.15.57:0/970245971 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8740101fa0 0x7f8740075d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 >> 172.21.15.57:0/970245971 conn(0x7f87400fd530 msgr2=0x7f87401064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 shutdown_connections 2024-04-21T08:42:46.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:46.435+0000 7f8747470640 1 -- 172.21.15.57:0/970245971 wait complete. 2024-04-21T08:42:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:46 smithi057 ceph-mon[27968]: Deploying daemon mds.a on smithi057 2024-04-21T08:42:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:46 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.57:6838/1393580593,v1:172.21.15.57:6839/1393580593] up:boot 2024-04-21T08:42:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:46 smithi057 ceph-mon[27968]: fsmap 10 up:standby 2024-04-21T08:42:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:42:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:46 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/970245971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:46.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:46 smithi136 ceph-mon[31791]: Deploying daemon mds.a on smithi057 2024-04-21T08:42:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:46 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.57:6838/1393580593,v1:172.21.15.57:6839/1393580593] up:boot 2024-04-21T08:42:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:46 smithi136 ceph-mon[31791]: fsmap 10 up:standby 2024-04-21T08:42:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:42:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:46 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/970245971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:47.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:46 smithi129 ceph-mon[31272]: Deploying daemon mds.a on smithi057 2024-04-21T08:42:47.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:46 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.57:6838/1393580593,v1:172.21.15.57:6839/1393580593] up:boot 2024-04-21T08:42:47.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:46 smithi129 ceph-mon[31272]: fsmap 10 up:standby 2024-04-21T08:42:47.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:42:47.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:46 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/970245971' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:47.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:47 smithi136 ceph-mon[31791]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:48.063 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:47 smithi057 ceph-mon[27968]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:48.065 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:47 smithi057 systemd[1]: Starting Ceph mds.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:48.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:47 smithi129 ceph-mon[31272]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:48.339 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 podman[70408]: 2024-04-21 08:42:47.966148497 +0000 UTC m=+0.018538773 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:48.339 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 podman[70408]: 2024-04-21 08:42:48.102463146 +0000 UTC m=+0.154853396 container create d9b3e4ca368b65bee0d2063f20f1739f79321bc3ee01d355e5a485138a835d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-21T08:42:48.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 -- 172.21.15.57:0/2181424142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79101029f0 msgr2=0x7f7910104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 --2- 172.21.15.57:0/2181424142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79101029f0 0x7f7910104de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7900009900 tx=0x7f790002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 -- 172.21.15.57:0/2181424142 shutdown_connections 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 --2- 172.21.15.57:0/2181424142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7910107c50 0x7f791010a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 --2- 172.21.15.57:0/2181424142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910105320 0x7f7910107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 --2- 172.21.15.57:0/2181424142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79101029f0 0x7f7910104de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 -- 172.21.15.57:0/2181424142 >> 172.21.15.57:0/2181424142 conn(0x7f79100fc3e0 msgr2=0x7f79100fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 -- 172.21.15.57:0/2181424142 shutdown_connections 2024-04-21T08:42:48.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.557+0000 7f7915611640 1 -- 172.21.15.57:0/2181424142 wait complete. 2024-04-21T08:42:48.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 Processor -- start 2024-04-21T08:42:48.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 -- start start 2024-04-21T08:42:48.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79101029f0 0x7f7910102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 0x7f79101009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910107c50 0x7f7910100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f791010dab0 con 0x7f7910105320 2024-04-21T08:42:48.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f791010d930 con 0x7f79101029f0 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f7915611640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f791010dc30 con 0x7f7910107c50 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f790e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 0x7f79101009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f790e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 0x7f79101009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46376/0 (socket says 172.21.15.57:46376) 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f790e7fc640 1 -- 172.21.15.57:0/747915664 learned_addr learned my addr 172.21.15.57:0/747915664 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.558+0000 7f790f7fe640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910107c50 0x7f7910100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790effd640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79101029f0 0x7f7910102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790e7fc640 1 -- 172.21.15.57:0/747915664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910107c50 msgr2=0x7f7910100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790e7fc640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910107c50 0x7f7910100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790e7fc640 1 -- 172.21.15.57:0/747915664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79101029f0 msgr2=0x7f7910102360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790e7fc640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79101029f0 0x7f7910102360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790e7fc640 1 -- 172.21.15.57:0/747915664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7910101770 con 0x7f7910105320 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790effd640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79101029f0 0x7f7910102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790f7fe640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910107c50 0x7f7910100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:48.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.559+0000 7f790e7fc640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 0x7f79101009a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f790400e9f0 tx=0x7f790400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.561+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790400ccc0 con 0x7f7910105320 2024-04-21T08:42:48.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.561+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79101a22e0 con 0x7f7910105320 2024-04-21T08:42:48.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.561+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79040054d0 con 0x7f7910105320 2024-04-21T08:42:48.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.561+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904010430 con 0x7f7910105320 2024-04-21T08:42:48.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.561+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79101a2830 con 0x7f7910105320 2024-04-21T08:42:48.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.562+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79100fded0 con 0x7f7910105320 2024-04-21T08:42:48.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.563+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7904005020 con 0x7f7910105320 2024-04-21T08:42:48.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.563+0000 7f78f7fff640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f78f80a3c30 0x7f78f80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:48.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.563+0000 7f790effd640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f78f80a3c30 0x7f78f80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:48.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.563+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79040d0410 con 0x7f7910105320 2024-04-21T08:42:48.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.564+0000 7f790effd640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f78f80a3c30 0x7f78f80a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7900006010 tx=0x7f790002c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:48.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.568+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7904016330 con 0x7f7910105320 2024-04-21T08:42:48.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.801+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f791002d770 con 0x7f7910105320 2024-04-21T08:42:48.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.801+0000 7f78f7fff640 1 -- 172.21.15.57:0/747915664 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7f790409d290 con 0x7f7910105320 2024-04-21T08:42:48.801 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:48.801 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:48.801 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f78f80a3c30 msgr2=0x7f78f80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f78f80a3c30 0x7f78f80a60f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7900006010 tx=0x7f790002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 msgr2=0x7f79101009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 0x7f79101009a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f790400e9f0 tx=0x7f790400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 shutdown_connections 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f78f80a3c30 0x7f78f80a60f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7910107c50 0x7f7910100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7910105320 0x7f79101009a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 --2- 172.21.15.57:0/747915664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79101029f0 0x7f7910102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:48.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.803+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 >> 172.21.15.57:0/747915664 conn(0x7f79100fc3e0 msgr2=0x7f79101034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:48.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.804+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 shutdown_connections 2024-04-21T08:42:48.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:48.804+0000 7f7915611640 1 -- 172.21.15.57:0/747915664 wait complete. 2024-04-21T08:42:48.813 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 podman[70408]: 2024-04-21 08:42:48.352515385 +0000 UTC m=+0.404905635 container init d9b3e4ca368b65bee0d2063f20f1739f79321bc3ee01d355e5a485138a835d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:42:48.813 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 podman[70408]: 2024-04-21 08:42:48.355713012 +0000 UTC m=+0.408103262 container start d9b3e4ca368b65bee0d2063f20f1739f79321bc3ee01d355e5a485138a835d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, ceph=True) 2024-04-21T08:42:48.813 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 ceph-mds[70427]: -- 172.21.15.57:0/1082510409 <== mon.0 v2:172.21.15.57:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556ef16d5a40 con 0x556ef16a9000 2024-04-21T08:42:48.813 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a[70423]: starting mds.a at 2024-04-21T08:42:48.813 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 bash[70408]: d9b3e4ca368b65bee0d2063f20f1739f79321bc3ee01d355e5a485138a835d69 2024-04-21T08:42:48.813 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:42:48 smithi057 systemd[1]: Started Ceph mds.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:49.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: Deploying daemon mds.b on smithi129 2024-04-21T08:42:49.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:49.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.57:6840/177564568,v1:172.21.15.57:6841/177564568] up:boot 2024-04-21T08:42:49.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: fsmap 11 up:standby 2024-04-21T08:42:49.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:42:49.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/747915664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: Deploying daemon mds.b on smithi129 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.57:6840/177564568,v1:172.21.15.57:6841/177564568] up:boot 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: fsmap 11 up:standby 2024-04-21T08:42:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:42:49.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/747915664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: Deploying daemon mds.b on smithi129 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.57:6840/177564568,v1:172.21.15.57:6841/177564568] up:boot 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: fsmap 11 up:standby 2024-04-21T08:42:49.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:42:49.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/747915664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:50.815 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 -- 172.21.15.57:0/1220192981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 msgr2=0x7f1f30109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 --2- 172.21.15.57:0/1220192981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30109330 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1f28009ca0 tx=0x7f1f2802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 -- 172.21.15.57:0/1220192981 shutdown_connections 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 --2- 172.21.15.57:0/1220192981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30109330 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 --2- 172.21.15.57:0/1220192981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f30102050 0x7f1f301024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 --2- 172.21.15.57:0/1220192981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f30100e70 0x7f1f30101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 -- 172.21.15.57:0/1220192981 >> 172.21.15.57:0/1220192981 conn(0x7f1f300fc470 msgr2=0x7f1f300fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 -- 172.21.15.57:0/1220192981 shutdown_connections 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 -- 172.21.15.57:0/1220192981 wait complete. 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 Processor -- start 2024-04-21T08:42:50.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.932+0000 7f1f35be6640 1 -- start start 2024-04-21T08:42:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f35be6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f30100e70 0x7f1f30104e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f35be6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f30102050 0x7f1f301034c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f35be6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30103a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f35be6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3010ca20 con 0x7f1f30102a10 2024-04-21T08:42:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f35be6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3010c8a0 con 0x7f1f30100e70 2024-04-21T08:42:50.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f35be6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3010cba0 con 0x7f1f30102050 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30103a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46394/0 (socket says 172.21.15.57:46394) 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 -- 172.21.15.57:0/2131527492 learned_addr learned my addr 172.21.15.57:0/2131527492 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2effd640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f30102050 0x7f1f301034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2f7fe640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f30100e70 0x7f1f30104e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 -- 172.21.15.57:0/2131527492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f30102050 msgr2=0x7f1f301034c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f30102050 0x7f1f301034c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 -- 172.21.15.57:0/2131527492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f30100e70 msgr2=0x7f1f30104e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f30100e70 0x7f1f30104e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 -- 172.21.15.57:0/2131527492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f30104290 con 0x7f1f30102a10 2024-04-21T08:42:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2ffff640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30103a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f2802c700 tx=0x7f1f280346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.933+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28041020 con 0x7f1f30102a10 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.934+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f28034ca0 con 0x7f1f30102a10 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.934+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f301a2210 con 0x7f1f30102a10 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.934+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28039040 con 0x7f1f30102a10 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.934+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f301a27b0 con 0x7f1f30102a10 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.935+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f301a2a10 con 0x7f1f30102a10 2024-04-21T08:42:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.936+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1f28044020 con 0x7f1f30102a10 2024-04-21T08:42:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.937+0000 7f1f2cff9640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1f180a3c30 0x7f1f180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.937+0000 7f1f2f7fe640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1f180a3c30 0x7f1f180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.937+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1f280f2110 con 0x7f1f30102a10 2024-04-21T08:42:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.937+0000 7f1f2f7fe640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1f180a3c30 0x7f1f180a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f30101bc0 tx=0x7f1f20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:50.941+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f2804b020 con 0x7f1f30102a10 2024-04-21T08:42:51.021 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:50 smithi129 systemd[1]: Starting Ceph mds.b for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:42:51.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.168+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f301a2be0 con 0x7f1f30102a10 2024-04-21T08:42:51.168 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.169+0000 7f1f2cff9640 1 -- 172.21.15.57:0/2131527492 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7f1f280bef70 con 0x7f1f30102a10 2024-04-21T08:42:51.168 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:51.168 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-21T08:42:51.169 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 12 2024-04-21T08:42:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1f180a3c30 msgr2=0x7f1f180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1f180a3c30 0x7f1f180a60f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f30101bc0 tx=0x7f1f20005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 msgr2=0x7f1f30103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30103a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1f2802c700 tx=0x7f1f280346d0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 shutdown_connections 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1f180a3c30 0x7f1f180a60f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1f30102a10 0x7f1f30103a00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f30102050 0x7f1f301034c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 --2- 172.21.15.57:0/2131527492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f30100e70 0x7f1f30104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 >> 172.21.15.57:0/2131527492 conn(0x7f1f300fc470 msgr2=0x7f1f300fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 shutdown_connections 2024-04-21T08:42:51.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:51.171+0000 7f1f35be6640 1 -- 172.21.15.57:0/2131527492 wait complete. 2024-04-21T08:42:51.273 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 podman[61078]: 2024-04-21 08:42:50.924793881 +0000 UTC m=+0.018049211 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:42:51.274 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 podman[61078]: 2024-04-21 08:42:51.030382168 +0000 UTC m=+0.123637491 container create 302607c3a1b7baed623e7dc6035112b31665926fa2ae5f72a1191d8dd74ba9f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:42:51.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:51 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2131527492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:51.573 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 podman[61078]: 2024-04-21 08:42:51.291257354 +0000 UTC m=+0.384512678 container init 302607c3a1b7baed623e7dc6035112b31665926fa2ae5f72a1191d8dd74ba9f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T08:42:51.573 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 podman[61078]: 2024-04-21 08:42:51.294061282 +0000 UTC m=+0.387316608 container start 302607c3a1b7baed623e7dc6035112b31665926fa2ae5f72a1191d8dd74ba9f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:42:51.573 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 ceph-mds[61097]: -- 172.21.15.129:0/4025030372 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563520a47a40 con 0x563520a1ac00 2024-04-21T08:42:51.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: starting mds.b at 2024-04-21T08:42:51.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 bash[61078]: 302607c3a1b7baed623e7dc6035112b31665926fa2ae5f72a1191d8dd74ba9f1 2024-04-21T08:42:51.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:42:51 smithi129 systemd[1]: Started Ceph mds.b for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:42:51.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:51 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2131527492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:51.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:51 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2131527492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:52 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:52 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:52 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:53.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.296+0000 7fbfd9a63640 1 -- 172.21.15.57:0/3488502402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4101fa0 msgr2=0x7fbfd41023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.296+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/3488502402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4101fa0 0x7fbfd41023c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009900 tx=0x7fbfc402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 -- 172.21.15.57:0/3488502402 shutdown_connections 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/3488502402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfd4103b50 0x7fbfd410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/3488502402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfd4103190 0x7fbfd4103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/3488502402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4101fa0 0x7fbfd41023c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 -- 172.21.15.57:0/3488502402 >> 172.21.15.57:0/3488502402 conn(0x7fbfd40fd530 msgr2=0x7fbfd40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:53.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 -- 172.21.15.57:0/3488502402 shutdown_connections 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 -- 172.21.15.57:0/3488502402 wait complete. 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.297+0000 7fbfd9a63640 1 Processor -- start 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 -- start start 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfd4101fa0 0x7fbfd41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 0x7fbfd4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfd4103b50 0x7fbfd419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410db20 con 0x7fbfd4103190 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410d9a0 con 0x7fbfd4101fa0 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd9a63640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd410dca0 con 0x7fbfd4103b50 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 0x7fbfd4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 0x7fbfd4198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40114/0 (socket says 172.21.15.57:40114) 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 -- 172.21.15.57:0/851508951 learned_addr learned my addr 172.21.15.57:0/851508951 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd3fff640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfd4103b50 0x7fbfd419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd37fe640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfd4101fa0 0x7fbfd41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 -- 172.21.15.57:0/851508951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfd4103b50 msgr2=0x7fbfd419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfd4103b50 0x7fbfd419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 -- 172.21.15.57:0/851508951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfd4101fa0 msgr2=0x7fbfd41981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfd4101fa0 0x7fbfd41981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 -- 172.21.15.57:0/851508951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd419fe00 con 0x7fbfd4103190 2024-04-21T08:42:53.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.298+0000 7fbfd2ffd640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 0x7fbfd4198720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800b4a0 tx=0x7fbfc800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:53.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.303+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8014070 con 0x7fbfd4103190 2024-04-21T08:42:53.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.304+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfc8010ac0 con 0x7fbfd4103190 2024-04-21T08:42:53.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.304+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd41a00f0 con 0x7fbfd4103190 2024-04-21T08:42:53.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.304+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8017c30 con 0x7fbfd4103190 2024-04-21T08:42:53.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.304+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd41a0710 con 0x7fbfd4103190 2024-04-21T08:42:53.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.305+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd41092c0 con 0x7fbfd4103190 2024-04-21T08:42:53.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.306+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbfc8017070 con 0x7fbfd4103190 2024-04-21T08:42:53.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.307+0000 7fbfd0ff9640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbfbc0a3c30 0x7fbfbc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.307+0000 7fbfd37fe640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbfbc0a3c30 0x7fbfbc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.307+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbfc80cf4f0 con 0x7fbfd4103190 2024-04-21T08:42:53.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.307+0000 7fbfd37fe640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbfbc0a3c30 0x7fbfbc0a60f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4005ab0 tx=0x7fbfc4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:53.310 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.311+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbfc809bd20 con 0x7fbfd4103190 2024-04-21T08:42:53.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.569+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfd402d790 con 0x7fbfd4103190 2024-04-21T08:42:53.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.570+0000 7fbfd0ff9640 1 -- 172.21.15.57:0/851508951 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7fbfc80a0020 con 0x7fbfd4103190 2024-04-21T08:42:53.570 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:53.570 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24503,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-21T08:42:53.570 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 13 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbfbc0a3c30 msgr2=0x7fbfbc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbfbc0a3c30 0x7fbfbc0a60f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4005ab0 tx=0x7fbfc4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 msgr2=0x7fbfd4198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 0x7fbfd4198720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbfc800b4a0 tx=0x7fbfc800b960 comp rx=0 tx=0).stop 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 shutdown_connections 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbfbc0a3c30 0x7fbfbc0a60f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfd4103b50 0x7fbfd419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfd4103190 0x7fbfd4198720 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 --2- 172.21.15.57:0/851508951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfd4101fa0 0x7fbfd41981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 >> 172.21.15.57:0/851508951 conn(0x7fbfd40fd530 msgr2=0x7fbfd4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:53.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 shutdown_connections 2024-04-21T08:42:53.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.572+0000 7fbfd9a63640 1 -- 172.21.15.57:0/851508951 wait complete. 2024-04-21T08:42:53.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:53 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] up:boot 2024-04-21T08:42:53.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:53 smithi129 ceph-mon[31272]: fsmap 12 up:standby 2024-04-21T08:42:53.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:53 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:42:53.582 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:53 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] up:boot 2024-04-21T08:42:53.582 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:53 smithi057 ceph-mon[27968]: fsmap 12 up:standby 2024-04-21T08:42:53.582 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:53 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:42:53.583 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:42:53.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:53 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] up:boot 2024-04-21T08:42:53.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:53 smithi136 ceph-mon[31791]: fsmap 12 up:standby 2024-04-21T08:42:53.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:53 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.687+0000 7fe005a77640 1 -- 172.21.15.57:0/2036495476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 msgr2=0x7fe000104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.687+0000 7fe005a77640 1 --2- 172.21.15.57:0/2036495476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe000104de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009ca0 tx=0x7fdff002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 -- 172.21.15.57:0/2036495476 shutdown_connections 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 --2- 172.21.15.57:0/2036495476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe000107c50 0x7fe00010a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 --2- 172.21.15.57:0/2036495476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe000105320 0x7fe000107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 --2- 172.21.15.57:0/2036495476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe000104de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 -- 172.21.15.57:0/2036495476 >> 172.21.15.57:0/2036495476 conn(0x7fe0000fc3e0 msgr2=0x7fe0000fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:53.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 -- 172.21.15.57:0/2036495476 shutdown_connections 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 -- 172.21.15.57:0/2036495476 wait complete. 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 Processor -- start 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.688+0000 7fe005a77640 1 -- start start 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe0001981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe000105320 0x7fe000198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe000107c50 0x7fe00019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe00010dab0 con 0x7fe0001029f0 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe00010d930 con 0x7fe000107c50 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe00010dc30 con 0x7fe000105320 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe0001981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe0001981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40132/0 (socket says 172.21.15.57:40132) 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 -- 172.21.15.57:0/2152788594 learned_addr learned my addr 172.21.15.57:0/2152788594 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:53.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdffffff640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe000107c50 0x7fe00019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdffeffd640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe000105320 0x7fe000198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 -- 172.21.15.57:0/2152788594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe000105320 msgr2=0x7fe000198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe000105320 0x7fe000198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 -- 172.21.15.57:0/2152788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe000107c50 msgr2=0x7fe00019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe000107c50 0x7fe00019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 -- 172.21.15.57:0/2152788594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00019fde0 con 0x7fe0001029f0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdfff7fe640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe0001981c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdff0005e30 tx=0x7fdff0034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0039040 con 0x7fe0001029f0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff00325c0 con 0x7fe0001029f0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0001a0070 con 0x7fe0001029f0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff00042a0 con 0x7fe0001029f0 2024-04-21T08:42:53.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.689+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0001a0580 con 0x7fe0001029f0 2024-04-21T08:42:53.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.690+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0000fded0 con 0x7fe0001029f0 2024-04-21T08:42:53.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.691+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdff0046030 con 0x7fe0001029f0 2024-04-21T08:42:53.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.692+0000 7fdffcff9640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdfe80a3c30 0x7fdfe80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:53.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.692+0000 7fdffeffd640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdfe80a3c30 0x7fdfe80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:53.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.692+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdff00f1960 con 0x7fe0001029f0 2024-04-21T08:42:53.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.693+0000 7fdffeffd640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdfe80a3c30 0x7fdfe80a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe0001996d0 tx=0x7fdff4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:53.696 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.697+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdff004d060 con 0x7fe0001029f0 2024-04-21T08:42:53.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.885+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe00002d7e0 con 0x7fe0001029f0 2024-04-21T08:42:53.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.886+0000 7fdffcff9640 1 -- 172.21.15.57:0/2152788594 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18432 (secure 0 0 0) 0x7fdff00be7d0 con 0x7fe0001029f0 2024-04-21T08:42:53.886 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:53.887 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":81,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:42:04.373173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:42:53.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdfe80a3c30 msgr2=0x7fdfe80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdfe80a3c30 0x7fdfe80a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe0001996d0 tx=0x7fdff4005f90 comp rx=0 tx=0).stop 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 msgr2=0x7fe0001981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe0001981c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdff0005e30 tx=0x7fdff0034760 comp rx=0 tx=0).stop 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 shutdown_connections 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fdfe80a3c30 0x7fdfe80a60f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe000107c50 0x7fe00019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe000105320 0x7fe000198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 --2- 172.21.15.57:0/2152788594 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0001029f0 0x7fe0001981c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:53.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 >> 172.21.15.57:0/2152788594 conn(0x7fe0000fc3e0 msgr2=0x7fe0001034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:53.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 shutdown_connections 2024-04-21T08:42:53.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:53.889+0000 7fe005a77640 1 -- 172.21.15.57:0/2152788594 wait complete. 2024-04-21T08:42:53.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-21T08:42:53.899 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:42:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 -- 172.21.15.57:0/2647312828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 msgr2=0x7fc400104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 --2- 172.21.15.57:0/2647312828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc400104e20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc009900 tx=0x7fc3fc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:54.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 -- 172.21.15.57:0/2647312828 shutdown_connections 2024-04-21T08:42:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 --2- 172.21.15.57:0/2647312828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 0x7fc40010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 --2- 172.21.15.57:0/2647312828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc400105360 0x7fc400107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 --2- 172.21.15.57:0/2647312828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc400104e20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 -- 172.21.15.57:0/2647312828 >> 172.21.15.57:0/2647312828 conn(0x7fc4000fc440 msgr2=0x7fc4000fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 -- 172.21.15.57:0/2647312828 shutdown_connections 2024-04-21T08:42:54.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.008+0000 7fc4079fb640 1 -- 172.21.15.57:0/2647312828 wait complete. 2024-04-21T08:42:54.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 Processor -- start 2024-04-21T08:42:54.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 -- start start 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc4001023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc400105360 0x7fc4001009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 0x7fc400100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc40010daf0 con 0x7fc400102a30 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc40010d970 con 0x7fc400107c90 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc4079fb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc40010dc70 con 0x7fc400105360 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc4001023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc4001023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40152/0 (socket says 172.21.15.57:40152) 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 -- 172.21.15.57:0/38450153 learned_addr learned my addr 172.21.15.57:0/38450153 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc404f96640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc400105360 0x7fc4001009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405f98640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 0x7fc400100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 -- 172.21.15.57:0/38450153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc400105360 msgr2=0x7fc4001009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc400105360 0x7fc4001009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 -- 172.21.15.57:0/38450153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 msgr2=0x7fc400100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 0x7fc400100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405797640 1 -- 172.21.15.57:0/38450153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4001017b0 con 0x7fc400102a30 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.009+0000 7fc405f98640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 0x7fc400100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.010+0000 7fc405797640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc4001023a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc006010 tx=0x7fc3fc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.010+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc043020 con 0x7fc400102a30 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.010+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3fc004440 con 0x7fc400102a30 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.010+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc003bf0 con 0x7fc400102a30 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.010+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4001a2380 con 0x7fc400102a30 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.010+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4001a28d0 con 0x7fc400102a30 2024-04-21T08:42:54.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.011+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4000fdf30 con 0x7fc400102a30 2024-04-21T08:42:54.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.013+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc3fc046020 con 0x7fc400102a30 2024-04-21T08:42:54.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.013+0000 7fc3f67fc640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc3ec0a3bb0 0x7fc3ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.013+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc3fc0f2940 con 0x7fc400102a30 2024-04-21T08:42:54.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.013+0000 7fc404f96640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc3ec0a3bb0 0x7fc3ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.014+0000 7fc404f96640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc3ec0a3bb0 0x7fc3ec0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc400102110 tx=0x7fc3f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.017+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3fc04d020 con 0x7fc400102a30 2024-04-21T08:42:54.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.205+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc400062b50 con 0x7fc400102a30 2024-04-21T08:42:54.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.206+0000 7fc3f67fc640 1 -- 172.21.15.57:0/38450153 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fc3fc0bf7c0 con 0x7fc400102a30 2024-04-21T08:42:54.205 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:42:54.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc3ec0a3bb0 msgr2=0x7fc3ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc3ec0a3bb0 0x7fc3ec0a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc400102110 tx=0x7fc3f000b390 comp rx=0 tx=0).stop 2024-04-21T08:42:54.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 msgr2=0x7fc4001023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc4001023a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc006010 tx=0x7fc3fc004040 comp rx=0 tx=0).stop 2024-04-21T08:42:54.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 shutdown_connections 2024-04-21T08:42:54.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc3ec0a3bb0 0x7fc3ec0a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc400107c90 0x7fc400100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc400105360 0x7fc4001009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 --2- 172.21.15.57:0/38450153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc400102a30 0x7fc4001023a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 >> 172.21.15.57:0/38450153 conn(0x7fc4000fc440 msgr2=0x7fc400103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 shutdown_connections 2024-04-21T08:42:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.208+0000 7fc4079fb640 1 -- 172.21.15.57:0/38450153 wait complete. 2024-04-21T08:42:54.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-21T08:42:54.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 -- 172.21.15.57:0/1455730096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c100e90 msgr2=0x7fbc7c1012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 --2- 172.21.15.57:0/1455730096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c100e90 0x7fbc7c1012b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009900 tx=0x7fbc7802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 -- 172.21.15.57:0/1455730096 shutdown_connections 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 --2- 172.21.15.57:0/1455730096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 0x7fbc7c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 --2- 172.21.15.57:0/1455730096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 --2- 172.21.15.57:0/1455730096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c100e90 0x7fbc7c1012b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 -- 172.21.15.57:0/1455730096 >> 172.21.15.57:0/1455730096 conn(0x7fbc7c0fc400 msgr2=0x7fbc7c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 -- 172.21.15.57:0/1455730096 shutdown_connections 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.322+0000 7fbc84414640 1 -- 172.21.15.57:0/1455730096 wait complete. 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 Processor -- start 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 -- start start 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c100e90 0x7fbc7c1970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 0x7fbc7c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10c9c0 con 0x7fbc7c101fe0 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10c840 con 0x7fbc7c100e90 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc84414640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10cb40 con 0x7fbc7c1029a0 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc819af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc819af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40160/0 (socket says 172.21.15.57:40160) 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc819af640 1 -- 172.21.15.57:0/4091875703 learned_addr learned my addr 172.21.15.57:0/4091875703 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc829b1640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 0x7fbc7c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.323+0000 7fbc821b0640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c100e90 0x7fbc7c1970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc819af640 1 -- 172.21.15.57:0/4091875703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 msgr2=0x7fbc7c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc819af640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 0x7fbc7c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc819af640 1 -- 172.21.15.57:0/4091875703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c100e90 msgr2=0x7fbc7c1970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc819af640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c100e90 0x7fbc7c1970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc819af640 1 -- 172.21.15.57:0/4091875703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c19ed10 con 0x7fbc7c101fe0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc829b1640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 0x7fbc7c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc819af640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c197630 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009970 tx=0x7fbc6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00d030 con 0x7fbc7c101fe0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc6c007d10 con 0x7fbc7c101fe0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c069140 con 0x7fbc7c101fe0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00cbe0 con 0x7fbc7c101fe0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.324+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c069690 con 0x7fbc7c101fe0 2024-04-21T08:42:54.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.325+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c068d70 con 0x7fbc7c101fe0 2024-04-21T08:42:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.326+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbc6c007860 con 0x7fbc7c101fe0 2024-04-21T08:42:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.327+0000 7fbc737fe640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbc680a3bb0 0x7fbc680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.328+0000 7fbc821b0640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbc680a3bb0 0x7fbc680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.328+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbc6c0d1a60 con 0x7fbc7c101fe0 2024-04-21T08:42:54.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.328+0000 7fbc821b0640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbc680a3bb0 0x7fbc680a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbc78000c00 tx=0x7fbc78005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.331+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc6c09e8c0 con 0x7fbc7c101fe0 2024-04-21T08:42:54.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.523+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbc7c19eee0 con 0x7fbc7c101fe0 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.523+0000 7fbc737fe640 1 -- 172.21.15.57:0/4091875703 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fbc6c09e1c0 con 0x7fbc7c101fe0 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:42:54.523 INFO:teuthology.orchestra.run.smithi057.stdout:] 2024-04-21T08:42:54.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbc680a3bb0 msgr2=0x7fbc680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbc680a3bb0 0x7fbc680a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbc78000c00 tx=0x7fbc78005be0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 msgr2=0x7fbc7c197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c197630 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009970 tx=0x7fbc6c009e30 comp rx=0 tx=0).stop 2024-04-21T08:42:54.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 shutdown_connections 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbc680a3bb0 0x7fbc680a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc7c1029a0 0x7fbc7c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbc7c101fe0 0x7fbc7c197630 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 --2- 172.21.15.57:0/4091875703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c100e90 0x7fbc7c1970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 >> 172.21.15.57:0/4091875703 conn(0x7fbc7c0fc400 msgr2=0x7fbc7c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 shutdown_connections 2024-04-21T08:42:54.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.525+0000 7fbc84414640 1 -- 172.21.15.57:0/4091875703 wait complete. 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/851508951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2152788594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:54.535 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:54.536 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.536 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/38450153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:54.536 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:54 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.537 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-21T08:42:54.537 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-21T08:42:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/851508951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2152788594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:54.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:54.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/38450153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:54.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:54 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/128317529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec101fe0 msgr2=0x7fb7ec102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/128317529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec101fe0 0x7fb7ec102460 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009900 tx=0x7fb7d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/128317529 shutdown_connections 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/128317529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7ec1029a0 0x7fb7ec1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/128317529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec101fe0 0x7fb7ec102460 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/128317529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec100e90 0x7fb7ec1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/128317529 >> 172.21.15.57:0/128317529 conn(0x7fb7ec0fc400 msgr2=0x7fb7ec0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/128317529 shutdown_connections 2024-04-21T08:42:54.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/128317529 wait complete. 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 Processor -- start 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 -- start start 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.646+0000 7fb7f2d8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 0x7fb7ec104e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f2d8e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec101fe0 0x7fb7ec103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f2d8e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7ec1029a0 0x7fb7ec103990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f2d8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec10ca30 con 0x7fb7ec100e90 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f2d8e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec10c8b0 con 0x7fb7ec101fe0 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f2d8e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec10cbb0 con 0x7fb7ec1029a0 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 0x7fb7ec104e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 0x7fb7ec104e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40180/0 (socket says 172.21.15.57:40180) 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 -- 172.21.15.57:0/1660111643 learned_addr learned my addr 172.21.15.57:0/1660111643 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f132b640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7ec1029a0 0x7fb7ec103990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec101fe0 0x7fb7ec103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 -- 172.21.15.57:0/1660111643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7ec1029a0 msgr2=0x7fb7ec103990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7ec1029a0 0x7fb7ec103990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 -- 172.21.15.57:0/1660111643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec101fe0 msgr2=0x7fb7ec103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec101fe0 0x7fb7ec103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 -- 172.21.15.57:0/1660111643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec104220 con 0x7fb7ec100e90 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f0b2a640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 0x7fb7ec104e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400ebc0 tx=0x7fb7e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4010630 con 0x7fb7ec100e90 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec1a2320 con 0x7fb7ec100e90 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7e4010c80 con 0x7fb7ec100e90 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.647+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4018b60 con 0x7fb7ec100e90 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.648+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec1a2890 con 0x7fb7ec100e90 2024-04-21T08:42:54.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.648+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ec1a2b50 con 0x7fb7ec100e90 2024-04-21T08:42:54.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.650+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb7e4018d00 con 0x7fb7ec100e90 2024-04-21T08:42:54.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.651+0000 7fb7e9ffb640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb7d80a3c30 0x7fb7d80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:54.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.651+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb7d80a3c30 0x7fb7d80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:54.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.651+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb7e40d0970 con 0x7fb7ec100e90 2024-04-21T08:42:54.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.651+0000 7fb7ebfff640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb7d80a3c30 0x7fb7d80a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb7d402c240 tx=0x7fb7d402c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:54.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.655+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7e409d140 con 0x7fb7ec100e90 2024-04-21T08:42:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/851508951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2152788594' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/38450153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:42:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:54 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:42:54.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:54.850+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb7ec1a2d20 con 0x7fb7ec100e90 2024-04-21T08:42:55.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.616+0000 7fb7e9ffb640 1 -- 172.21.15.57:0/1660111643 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fb7e40a1440 con 0x7fb7ec100e90 2024-04-21T08:42:55.616 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.619+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb7d80a3c30 msgr2=0x7fb7d80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.619+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb7d80a3c30 0x7fb7d80a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb7d402c240 tx=0x7fb7d402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.619+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 msgr2=0x7fb7ec104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.619+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 0x7fb7ec104e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400ebc0 tx=0x7fb7e400c6a0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 shutdown_connections 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb7d80a3c30 0x7fb7d80a60f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7ec1029a0 0x7fb7ec103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec101fe0 0x7fb7ec103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 --2- 172.21.15.57:0/1660111643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb7ec100e90 0x7fb7ec104e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 >> 172.21.15.57:0/1660111643 conn(0x7fb7ec0fc400 msgr2=0x7fb7ec0fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:55.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 shutdown_connections 2024-04-21T08:42:55.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.620+0000 7fb7f2d8e640 1 -- 172.21.15.57:0/1660111643 wait complete. 2024-04-21T08:42:55.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:55 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4091875703' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:42:55.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:55 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:55.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:55 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1660111643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:55.635 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:55 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4091875703' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:42:55.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:55 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:55.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:55 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1660111643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:55.637 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-21T08:42:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/593182146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64102a00 msgr2=0x7fbd64104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/593182146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64102a00 0x7fbd64104df0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009900 tx=0x7fbd5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/593182146 shutdown_connections 2024-04-21T08:42:55.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/593182146 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd64107c60 0x7fbd6410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/593182146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64105330 0x7fbd64107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/593182146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64102a00 0x7fbd64104df0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/593182146 >> 172.21.15.57:0/593182146 conn(0x7fbd640fc3f0 msgr2=0x7fbd640fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/593182146 shutdown_connections 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/593182146 wait complete. 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.755+0000 7fbd6aaf2640 1 Processor -- start 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 -- start start 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd64102a00 0x7fbd64198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 0x7fbd64198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64107c60 0x7fbd6419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410dab0 con 0x7fbd64105330 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410d930 con 0x7fbd64107c60 2024-04-21T08:42:55.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6aaf2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6410dc30 con 0x7fbd64102a00 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 0x7fbd64198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 0x7fbd64198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40200/0 (socket says 172.21.15.57:40200) 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 -- 172.21.15.57:0/3706131695 learned_addr learned my addr 172.21.15.57:0/3706131695 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6888e640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd64102a00 0x7fbd64198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd6908f640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64107c60 0x7fbd6419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 -- 172.21.15.57:0/3706131695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd64102a00 msgr2=0x7fbd64198210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd64102a00 0x7fbd64198210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 -- 172.21.15.57:0/3706131695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64107c60 msgr2=0x7fbd6419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64107c60 0x7fbd6419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 -- 172.21.15.57:0/3706131695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6419fe30 con 0x7fbd64105330 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.756+0000 7fbd63fff640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 0x7fbd64198750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbd50005fd0 tx=0x7fbd5000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:55.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.757+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50017070 con 0x7fbd64105330 2024-04-21T08:42:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.757+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd50009c30 con 0x7fbd64105330 2024-04-21T08:42:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.757+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd641a0120 con 0x7fbd64105330 2024-04-21T08:42:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.757+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50003d30 con 0x7fbd64105330 2024-04-21T08:42:55.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.757+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd641a0670 con 0x7fbd64105330 2024-04-21T08:42:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.758+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd640fdee0 con 0x7fbd64105330 2024-04-21T08:42:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.759+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbd50003ed0 con 0x7fbd64105330 2024-04-21T08:42:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.760+0000 7fbd61ffb640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbd540a3bb0 0x7fbd540a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.760+0000 7fbd6888e640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbd540a3bb0 0x7fbd540a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.760+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fbd500d04f0 con 0x7fbd64105330 2024-04-21T08:42:55.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.760+0000 7fbd6888e640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbd540a3bb0 0x7fbd540a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009230 tx=0x7fbd58005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:55.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.764+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd50020030 con 0x7fbd64105330 2024-04-21T08:42:55.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:55 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4091875703' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-21T08:42:55.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:55 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:55.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:55 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1660111643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-21T08:42:55.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:55.977+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fbd6402d5c0 con 0x7fbd64105330 2024-04-21T08:42:56.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.618+0000 7fbd61ffb640 1 -- 172.21.15.57:0/3706131695 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fbd50009de0 con 0x7fbd64105330 2024-04-21T08:42:56.618 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbd540a3bb0 msgr2=0x7fbd540a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbd540a3bb0 0x7fbd540a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009230 tx=0x7fbd58005fd0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 msgr2=0x7fbd64198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 0x7fbd64198750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbd50005fd0 tx=0x7fbd5000dde0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 shutdown_connections 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbd540a3bb0 0x7fbd540a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd64107c60 0x7fbd6419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd64105330 0x7fbd64198750 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 --2- 172.21.15.57:0/3706131695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd64102a00 0x7fbd64198210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 >> 172.21.15.57:0/3706131695 conn(0x7fbd640fc3f0 msgr2=0x7fbd641033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:56.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 shutdown_connections 2024-04-21T08:42:56.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.620+0000 7fbd6aaf2640 1 -- 172.21.15.57:0/3706131695 wait complete. 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1660111643' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3706131695' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:56.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:56.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:56.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1660111643' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:42:56.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:42:56.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.639 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3706131695' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:56.640 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.641 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-21T08:42:56.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.749+0000 7f67b76fd640 1 -- 172.21.15.57:0/3360131348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0107c90 msgr2=0x7f67b010a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.749+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3360131348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0107c90 0x7f67b010a0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009ca0 tx=0x7f67a002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- 172.21.15.57:0/3360131348 shutdown_connections 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3360131348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0107c90 0x7f67b010a0e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3360131348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b0105360 0x7f67b0107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3360131348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0102a30 0x7f67b0104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- 172.21.15.57:0/3360131348 >> 172.21.15.57:0/3360131348 conn(0x7f67b00fc440 msgr2=0x7f67b00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- 172.21.15.57:0/3360131348 shutdown_connections 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- 172.21.15.57:0/3360131348 wait complete. 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 Processor -- start 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- start start 2024-04-21T08:42:56.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 0x7f67b01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0105360 0x7f67b01986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b0107c90 0x7f67b019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f67b010daf0 con 0x7f67b0102a30 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67b010d970 con 0x7f67b0105360 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.750+0000 7f67b76fd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f67b010dc70 con 0x7f67b0107c90 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 0x7f67b01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 0x7f67b01981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40208/0 (socket says 172.21.15.57:40208) 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 -- 172.21.15.57:0/3294098315 learned_addr learned my addr 172.21.15.57:0/3294098315 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b4c98640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0105360 0x7f67b01986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5c9a640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b0107c90 0x7f67b019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 -- 172.21.15.57:0/3294098315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b0107c90 msgr2=0x7f67b019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b0107c90 0x7f67b019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 -- 172.21.15.57:0/3294098315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0105360 msgr2=0x7f67b01986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0105360 0x7f67b01986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 -- 172.21.15.57:0/3294098315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b019fdc0 con 0x7f67b0102a30 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b4c98640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0105360 0x7f67b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b5499640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 0x7f67b01981a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f67a800e9f0 tx=0x7f67a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a800cd80 con 0x7f67b0102a30 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67a80108e0 con 0x7f67b0102a30 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b01a00b0 con 0x7f67b0102a30 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8018980 con 0x7f67b0102a30 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.751+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b01a0600 con 0x7f67b0102a30 2024-04-21T08:42:56.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.752+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b00fdf30 con 0x7f67b0102a30 2024-04-21T08:42:56.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.754+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f67a8010430 con 0x7f67b0102a30 2024-04-21T08:42:56.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.754+0000 7f67ae7fc640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67980a3bb0 0x7f67980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:56.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.754+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f67a80d0a20 con 0x7f67b0102a30 2024-04-21T08:42:56.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.754+0000 7f67b4c98640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67980a3bb0 0x7f67980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:56.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.755+0000 7f67b4c98640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67980a3bb0 0x7f67980a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67b01996b0 tx=0x7f67a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:56.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.758+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67a809cf60 con 0x7f67b0102a30 2024-04-21T08:42:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1660111643' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-21T08:42:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: osdmap e82: 12 total, 12 up, 12 in 2024-04-21T08:42:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3706131695' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:42:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:42:56.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:56.987+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f67b002d5c0 con 0x7f67b0102a30 2024-04-21T08:42:57.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.619+0000 7f67ae7fc640 1 -- 172.21.15.57:0/3294098315 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f67a80a1260 con 0x7f67b0102a30 2024-04-21T08:42:57.620 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 2 and data pool 3 2024-04-21T08:42:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67980a3bb0 msgr2=0x7f67980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67980a3bb0 0x7f67980a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67b01996b0 tx=0x7f67a400b410 comp rx=0 tx=0).stop 2024-04-21T08:42:57.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 msgr2=0x7f67b01981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 0x7f67b01981a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f67a800e9f0 tx=0x7f67a800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 shutdown_connections 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67980a3bb0 0x7f67980a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b0107c90 0x7f67b019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b0105360 0x7f67b01986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 --2- 172.21.15.57:0/3294098315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67b0102a30 0x7f67b01981a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 >> 172.21.15.57:0/3294098315 conn(0x7f67b00fc440 msgr2=0x7f67b0103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 shutdown_connections 2024-04-21T08:42:57.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.623+0000 7f67b76fd640 1 -- 172.21.15.57:0/3294098315 wait complete. 2024-04-21T08:42:57.640 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-21T08:42:57.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-21T08:42:57.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 -- 172.21.15.57:0/1899798426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450102a30 msgr2=0x7f6450104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 --2- 172.21.15.57:0/1899798426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450102a30 0x7f6450104e20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f644c009ca0 tx=0x7f644c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 -- 172.21.15.57:0/1899798426 shutdown_connections 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 --2- 172.21.15.57:0/1899798426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 0x7f645010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 --2- 172.21.15.57:0/1899798426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450105360 0x7f6450107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 --2- 172.21.15.57:0/1899798426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450102a30 0x7f6450104e20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 -- 172.21.15.57:0/1899798426 >> 172.21.15.57:0/1899798426 conn(0x7f64500fc440 msgr2=0x7f64500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 -- 172.21.15.57:0/1899798426 shutdown_connections 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 -- 172.21.15.57:0/1899798426 wait complete. 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.754+0000 7f64586f2640 1 Processor -- start 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 -- start start 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450102a30 0x7f64501023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 0x7f64501009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 0x7f6450100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f645010db70 con 0x7f6450105360 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f645010d9f0 con 0x7f6450107c90 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64586f2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f645010dcf0 con 0x7f6450102a30 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 0x7f64501009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f645648e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450102a30 0x7f64501023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 0x7f64501009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40228/0 (socket says 172.21.15.57:40228) 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 -- 172.21.15.57:0/964793882 learned_addr learned my addr 172.21.15.57:0/964793882 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6456c8f640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 0x7f6450100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 -- 172.21.15.57:0/964793882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450102a30 msgr2=0x7f64501023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450102a30 0x7f64501023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 -- 172.21.15.57:0/964793882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 msgr2=0x7f6450100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 0x7f6450100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:57.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 -- 172.21.15.57:0/964793882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64501017b0 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6456c8f640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 0x7f6450100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f645648e640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450102a30 0x7f64501023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f6455c8d640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 0x7f64501009e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f644000e9f0 tx=0x7f644000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.755+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644000ccc0 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.756+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64400054d0 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.756+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64501a23a0 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.756+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440010430 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.756+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64501a28a0 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.756+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64500fdf30 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.758+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f64400046c0 con 0x7f6450105360 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.758+0000 7f64477fe640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f643c0a3bb0 0x7f643c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.758+0000 7f645648e640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f643c0a3bb0 0x7f643c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:57.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.758+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f64400d0940 con 0x7f6450105360 2024-04-21T08:42:57.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.759+0000 7f645648e640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f643c0a3bb0 0x7f643c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f644c000f80 tx=0x7f644c032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:57.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.762+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6440053ff0 con 0x7f6450105360 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: pgmap v288: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3706131695' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3294098315' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:57.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:57 smithi136 ceph-mon[31791]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:42:57.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:57.956+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f645002d5c0 con 0x7f6450105360 2024-04-21T08:42:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: pgmap v288: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3706131695' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:42:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:42:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3294098315' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:42:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:42:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:58.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:57 smithi129 ceph-mon[31272]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: pgmap v288: 33 pgs: 10 creating+peering, 22 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3706131695' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: osdmap e83: 12 total, 12 up, 12 in 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3294098315' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:42:58.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:57 smithi057 ceph-mon[27968]: osdmap e84: 12 total, 12 up, 12 in 2024-04-21T08:42:58.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.381+0000 7f64477fe640 1 -- 172.21.15.57:0/964793882 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f644009d470 con 0x7f6450105360 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f643c0a3bb0 msgr2=0x7f643c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f643c0a3bb0 0x7f643c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f644c000f80 tx=0x7f644c032340 comp rx=0 tx=0).stop 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 msgr2=0x7f64501009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 0x7f64501009e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f644000e9f0 tx=0x7f644000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 shutdown_connections 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f643c0a3bb0 0x7f643c0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6450107c90 0x7f6450100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6450105360 0x7f64501009e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 --2- 172.21.15.57:0/964793882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6450102a30 0x7f64501023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 >> 172.21.15.57:0/964793882 conn(0x7f64500fc440 msgr2=0x7f6450103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:58.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 shutdown_connections 2024-04-21T08:42:58.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.383+0000 7f64586f2640 1 -- 172.21.15.57:0/964793882 wait complete. 2024-04-21T08:42:58.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 -- 172.21.15.57:0/2454797846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14101031b0 msgr2=0x7f1410103630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 --2- 172.21.15.57:0/2454797846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14101031b0 0x7f1410103630 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1400009900 tx=0x7f140002bc30 comp rx=0 tx=0).stop 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 -- 172.21.15.57:0/2454797846 shutdown_connections 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 --2- 172.21.15.57:0/2454797846 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1410103b70 0x7f141010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 --2- 172.21.15.57:0/2454797846 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14101031b0 0x7f1410103630 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 --2- 172.21.15.57:0/2454797846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1410101fc0 0x7f14101023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 -- 172.21.15.57:0/2454797846 >> 172.21.15.57:0/2454797846 conn(0x7f14100fd550 msgr2=0x7f14100ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 -- 172.21.15.57:0/2454797846 shutdown_connections 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.508+0000 7f1416767640 1 -- 172.21.15.57:0/2454797846 wait complete. 2024-04-21T08:42:58.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 Processor -- start 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 -- start start 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1410101fc0 0x7f14101981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14101031b0 0x7f1410198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 0x7f141019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f141010db50 con 0x7f1410103b70 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f141010d9d0 con 0x7f14101031b0 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1416767640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f141010dcd0 con 0x7f1410101fc0 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1414d04640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 0x7f141019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1414d04640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 0x7f141019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40244/0 (socket says 172.21.15.57:40244) 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f1414d04640 1 -- 172.21.15.57:0/1054610423 learned_addr learned my addr 172.21.15.57:0/1054610423 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.509+0000 7f140ffff640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1410101fc0 0x7f14101981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f140f7fe640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14101031b0 0x7f1410198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:58.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1414d04640 1 -- 172.21.15.57:0/1054610423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1410101fc0 msgr2=0x7f14101981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1414d04640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1410101fc0 0x7f14101981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1414d04640 1 -- 172.21.15.57:0/1054610423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14101031b0 msgr2=0x7f1410198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1414d04640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14101031b0 0x7f1410198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1414d04640 1 -- 172.21.15.57:0/1054610423 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141019fdf0 con 0x7f1410103b70 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1414d04640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 0x7f141019f780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f140800eaa0 tx=0x7f140800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800cdb0 con 0x7f1410103b70 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14101a00e0 con 0x7f1410103b70 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1408005020 con 0x7f1410103b70 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080116d0 con 0x7f1410103b70 2024-04-21T08:42:58.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.510+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14101a0600 con 0x7f1410103b70 2024-04-21T08:42:58.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.511+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1410109360 con 0x7f1410103b70 2024-04-21T08:42:58.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.512+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f14080054a0 con 0x7f1410103b70 2024-04-21T08:42:58.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.513+0000 7f140d7fa640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13fc0a3bb0 0x7f13fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:42:58.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.513+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f14080d0980 con 0x7f1410103b70 2024-04-21T08:42:58.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.513+0000 7f140ffff640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13fc0a3bb0 0x7f13fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:42:58.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.513+0000 7f140ffff640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13fc0a3bb0 0x7f13fc0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1410103010 tx=0x7f13f800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:42:58.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.516+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1408016030 con 0x7f1410103b70 2024-04-21T08:42:58.708 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:42:58.709+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f141002d5c0 con 0x7f1410103b70 2024-04-21T08:42:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:58 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3294098315' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:42:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:58 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:42:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:58 smithi057 ceph-mon[27968]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:42:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:58 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/964793882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:42:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:58 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/964793882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:42:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:58 smithi057 ceph-mon[27968]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:42:58.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:58 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3294098315' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:42:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:58 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:42:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:58 smithi136 ceph-mon[31791]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:42:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:58 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/964793882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:42:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:58 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/964793882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:42:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:58 smithi136 ceph-mon[31791]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:42:58.975 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:58 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3294098315' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-21T08:42:58.975 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:58 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-21T08:42:58.975 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:58 smithi129 ceph-mon[31272]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-21T08:42:58.975 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:58 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/964793882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-21T08:42:58.975 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:58 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/964793882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-21T08:42:58.975 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:58 smithi129 ceph-mon[31272]: osdmap e85: 12 total, 12 up, 12 in 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: pgmap v292: 97 pgs: 10 creating+peering, 69 unknown, 18 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] up:active 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1054610423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:43:00.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:42:59 smithi129 ceph-mon[31272]: Cluster is now healthy 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: pgmap v292: 97 pgs: 10 creating+peering, 69 unknown, 18 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] up:active 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1054610423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:43:00.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:42:59 smithi057 ceph-mon[27968]: Cluster is now healthy 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: pgmap v292: 97 pgs: 10 creating+peering, 69 unknown, 18 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] up:active 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1054610423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:43:00.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:42:59 smithi136 ceph-mon[31791]: Cluster is now healthy 2024-04-21T08:43:00.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.654+0000 7f140d7fa640 1 -- 172.21.15.57:0/1054610423 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f140809d330 con 0x7f1410103b70 2024-04-21T08:43:00.654 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data_ec' created 2024-04-21T08:43:00.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13fc0a3bb0 msgr2=0x7f13fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:00.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13fc0a3bb0 0x7f13fc0a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1410103010 tx=0x7f13f800b390 comp rx=0 tx=0).stop 2024-04-21T08:43:00.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 msgr2=0x7f141019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:00.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 0x7f141019f780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f140800eaa0 tx=0x7f140800ef60 comp rx=0 tx=0).stop 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 shutdown_connections 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f13fc0a3bb0 0x7f13fc0a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1410103b70 0x7f141019f780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14101031b0 0x7f1410198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 --2- 172.21.15.57:0/1054610423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1410101fc0 0x7f14101981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 >> 172.21.15.57:0/1054610423 conn(0x7f14100fd550 msgr2=0x7f1410106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 shutdown_connections 2024-04-21T08:43:00.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.657+0000 7f1416767640 1 -- 172.21.15.57:0/1054610423 wait complete. 2024-04-21T08:43:00.676 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 -- 172.21.15.57:0/1521458144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4238101fe0 msgr2=0x7f4238102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 --2- 172.21.15.57:0/1521458144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4238101fe0 0x7f4238102460 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4228009900 tx=0x7f422802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 -- 172.21.15.57:0/1521458144 shutdown_connections 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 --2- 172.21.15.57:0/1521458144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42381029a0 0x7f42381092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 --2- 172.21.15.57:0/1521458144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4238101fe0 0x7f4238102460 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 --2- 172.21.15.57:0/1521458144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 0x7f42381012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 -- 172.21.15.57:0/1521458144 >> 172.21.15.57:0/1521458144 conn(0x7f42380fc400 msgr2=0x7f42380fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 -- 172.21.15.57:0/1521458144 shutdown_connections 2024-04-21T08:43:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 -- 172.21.15.57:0/1521458144 wait complete. 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 Processor -- start 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.789+0000 7f42401ce640 1 -- start start 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 0x7f4238105070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238101fe0 0x7f42381036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 0x7f4238103bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f423810cac0 con 0x7f42381029a0 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f423810c940 con 0x7f4238101fe0 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f423810cc40 con 0x7f4238100e90 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 0x7f4238103bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 0x7f4238103bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40254/0 (socket says 172.21.15.57:40254) 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 -- 172.21.15.57:0/3274703727 learned_addr learned my addr 172.21.15.57:0/3274703727 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423df6a640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 0x7f4238105070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 -- 172.21.15.57:0/3274703727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 msgr2=0x7f4238105070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 0x7f4238105070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 -- 172.21.15.57:0/3274703727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238101fe0 msgr2=0x7f42381036b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238101fe0 0x7f42381036b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 -- 172.21.15.57:0/3274703727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4238104480 con 0x7f42381029a0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423df6a640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 0x7f4238105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f423e76b640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 0x7f4238103bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f423000b970 tx=0x7f423000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230019070 con 0x7f42381029a0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4230014440 con 0x7f42381029a0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42381a2490 con 0x7f42381029a0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.790+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42300075d0 con 0x7f42381029a0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.791+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42381a2a60 con 0x7f42381029a0 2024-04-21T08:43:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.792+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42381a2d80 con 0x7f42381029a0 2024-04-21T08:43:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.793+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f423001c070 con 0x7f42381029a0 2024-04-21T08:43:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.793+0000 7f422effd640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f42240a3c30 0x7f42240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.793+0000 7f423df6a640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f42240a3c30 0x7f42240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.793+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f42300cfc70 con 0x7f42381029a0 2024-04-21T08:43:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.794+0000 7f423df6a640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f42240a3c30 0x7f42240a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4234009930 tx=0x7f4234008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:00.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.798+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f423009c430 con 0x7f42381029a0 2024-04-21T08:43:00.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:00.984+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f42381a2f50 con 0x7f42381029a0 2024-04-21T08:43:01.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:00 smithi129 ceph-mon[31272]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:43:01.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:00 smithi057 ceph-mon[27968]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:43:01.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:00 smithi136 ceph-mon[31791]: osdmap e86: 12 total, 12 up, 12 in 2024-04-21T08:43:01.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.657+0000 7f422effd640 1 -- 172.21.15.57:0/3274703727 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f423009bd30 con 0x7f42381029a0 2024-04-21T08:43:01.656 INFO:teuthology.orchestra.run.smithi057.stderr:set pool 4 allow_ec_overwrites to true 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f42240a3c30 msgr2=0x7f42240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f42240a3c30 0x7f42240a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4234009930 tx=0x7f4234008040 comp rx=0 tx=0).stop 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 msgr2=0x7f4238103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 0x7f4238103bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f423000b970 tx=0x7f423000be30 comp rx=0 tx=0).stop 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 shutdown_connections 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f42240a3c30 0x7f42240a60f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42381029a0 0x7f4238103bf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4238101fe0 0x7f42381036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 --2- 172.21.15.57:0/3274703727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4238100e90 0x7f4238105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 >> 172.21.15.57:0/3274703727 conn(0x7f42380fc400 msgr2=0x7f42380690e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 shutdown_connections 2024-04-21T08:43:01.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.660+0000 7f42401ce640 1 -- 172.21.15.57:0/3274703727 wait complete. 2024-04-21T08:43:01.682 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-21T08:43:01.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 -- 172.21.15.57:0/4041604931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc107c70 msgr2=0x7f10cc10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:01.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 --2- 172.21.15.57:0/4041604931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc107c70 0x7f10cc10a0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10c4009ca0 tx=0x7f10c402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 -- 172.21.15.57:0/4041604931 shutdown_connections 2024-04-21T08:43:01.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 --2- 172.21.15.57:0/4041604931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc107c70 0x7f10cc10a0c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 --2- 172.21.15.57:0/4041604931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10cc105340 0x7f10cc107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 --2- 172.21.15.57:0/4041604931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc102a10 0x7f10cc104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 -- 172.21.15.57:0/4041604931 >> 172.21.15.57:0/4041604931 conn(0x7f10cc0fc400 msgr2=0x7f10cc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 -- 172.21.15.57:0/4041604931 shutdown_connections 2024-04-21T08:43:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.795+0000 7f10d2315640 1 -- 172.21.15.57:0/4041604931 wait complete. 2024-04-21T08:43:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 Processor -- start 2024-04-21T08:43:01.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 -- start start 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc102a10 0x7f10cc198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 0x7f10cc198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10cc107c70 0x7f10cc19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc10db30 con 0x7f10cc105340 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc10d9b0 con 0x7f10cc102a10 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10d2315640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc10dcb0 con 0x7f10cc107c70 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.796+0000 7f10cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 0x7f10cc198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 0x7f10cc198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40270/0 (socket says 172.21.15.57:40270) 2024-04-21T08:43:01.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 -- 172.21.15.57:0/2018272113 learned_addr learned my addr 172.21.15.57:0/2018272113 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cbfff640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc102a10 0x7f10cc198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 -- 172.21.15.57:0/2018272113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10cc107c70 msgr2=0x7f10cc19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10cc107c70 0x7f10cc19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 -- 172.21.15.57:0/2018272113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc102a10 msgr2=0x7f10cc198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc102a10 0x7f10cc198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 -- 172.21.15.57:0/2018272113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc19fe50 con 0x7f10cc105340 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10cb7fe640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 0x7f10cc198770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b400b450 tx=0x7f10b400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4015020 con 0x7f10cc105340 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10cc1a0140 con 0x7f10cc105340 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10b4011d40 con 0x7f10cc105340 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b400f340 con 0x7f10cc105340 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.797+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10cc1a0690 con 0x7f10cc105340 2024-04-21T08:43:01.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.798+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10cc0fdef0 con 0x7f10cc105340 2024-04-21T08:43:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.799+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f10b4018020 con 0x7f10cc105340 2024-04-21T08:43:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.800+0000 7f10c97fa640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10b80a3bb0 0x7f10b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.800+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f10b40d0740 con 0x7f10cc105340 2024-04-21T08:43:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.800+0000 7f10cbfff640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10b80a3bb0 0x7f10b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:01.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.800+0000 7f10cbfff640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10b80a3bb0 0x7f10b80a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f10bc005fd0 tx=0x7f10bc005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:01.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:01.803+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10b401f020 con 0x7f10cc105340 2024-04-21T08:43:02.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.029+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f10cc02d5c0 con 0x7f10cc105340 2024-04-21T08:43:02.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:01 smithi129 ceph-mon[31272]: pgmap v294: 97 pgs: 10 creating+peering, 24 unknown, 63 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-21T08:43:02.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:01 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1054610423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:43:02.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:01 smithi129 ceph-mon[31272]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:43:02.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:01 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3274703727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:43:02.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:01 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3274703727' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:43:02.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:01 smithi129 ceph-mon[31272]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:43:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:01 smithi057 ceph-mon[27968]: pgmap v294: 97 pgs: 10 creating+peering, 24 unknown, 63 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-21T08:43:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:01 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1054610423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:43:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:01 smithi057 ceph-mon[27968]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:43:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:01 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3274703727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:43:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:01 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3274703727' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:43:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:01 smithi057 ceph-mon[27968]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:43:02.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:01 smithi136 ceph-mon[31791]: pgmap v294: 97 pgs: 10 creating+peering, 24 unknown, 63 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-21T08:43:02.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:01 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1054610423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-21T08:43:02.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:01 smithi136 ceph-mon[31791]: osdmap e87: 12 total, 12 up, 12 in 2024-04-21T08:43:02.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:01 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3274703727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-21T08:43:02.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:01 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3274703727' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-21T08:43:02.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:01 smithi136 ceph-mon[31791]: osdmap e88: 12 total, 12 up, 12 in 2024-04-21T08:43:02.573 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:43:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: 2024-04-21T08:43:02.293+0000 7fc71da1a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-21T08:43:02.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.679+0000 7f10c97fa640 1 -- 172.21.15.57:0/2018272113 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f10b400d040 con 0x7f10cc105340 2024-04-21T08:43:02.678 INFO:teuthology.orchestra.run.smithi057.stderr:added data pool 4 to fsmap 2024-04-21T08:43:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10b80a3bb0 msgr2=0x7f10b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10b80a3bb0 0x7f10b80a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f10bc005fd0 tx=0x7f10bc005c70 comp rx=0 tx=0).stop 2024-04-21T08:43:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 msgr2=0x7f10cc198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 0x7f10cc198770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b400b450 tx=0x7f10b400b910 comp rx=0 tx=0).stop 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 shutdown_connections 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f10b80a3bb0 0x7f10b80a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10cc107c70 0x7f10cc19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10cc105340 0x7f10cc198770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 --2- 172.21.15.57:0/2018272113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc102a10 0x7f10cc198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 >> 172.21.15.57:0/2018272113 conn(0x7f10cc0fc400 msgr2=0x7f10cc1033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 shutdown_connections 2024-04-21T08:43:02.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.682+0000 7f10d2315640 1 -- 172.21.15.57:0/2018272113 wait complete. 2024-04-21T08:43:02.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 -- 172.21.15.57:0/3893340415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 msgr2=0x7f12c010a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 --2- 172.21.15.57:0/3893340415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c010a540 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009cd0 tx=0x7f12b802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 -- 172.21.15.57:0/3893340415 shutdown_connections 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 --2- 172.21.15.57:0/3893340415 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c010a540 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 --2- 172.21.15.57:0/3893340415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c01032f0 0x7f12c0103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 --2- 172.21.15.57:0/3893340415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c0100280 0x7f12c01006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 -- 172.21.15.57:0/3893340415 >> 172.21.15.57:0/3893340415 conn(0x7f12c00fbb90 msgr2=0x7f12c00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 -- 172.21.15.57:0/3893340415 shutdown_connections 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 -- 172.21.15.57:0/3893340415 wait complete. 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.805+0000 7f12c7631640 1 Processor -- start 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 -- start start 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c0100280 0x7f12c0102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01032f0 0x7f12c01012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c01017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010dbd0 con 0x7f12c0103cb0 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010da50 con 0x7f12c0100280 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010dd50 con 0x7f12c01032f0 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c01017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c01017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58902/0 (socket says 172.21.15.57:58902) 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 -- 172.21.15.57:0/241688270 learned_addr learned my addr 172.21.15.57:0/241688270 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:02.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c53cd640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c0100280 0x7f12c0102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c4bcc640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01032f0 0x7f12c01012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 -- 172.21.15.57:0/241688270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01032f0 msgr2=0x7f12c01012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01032f0 0x7f12c01012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 -- 172.21.15.57:0/241688270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c0100280 msgr2=0x7f12c0102c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c0100280 0x7f12c0102c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 -- 172.21.15.57:0/241688270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c0102070 con 0x7f12c0103cb0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c53cd640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c0100280 0x7f12c0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c4bcc640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01032f0 0x7f12c01012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c5bce640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c01017e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f12b8037040 tx=0x7f12b8035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8039040 con 0x7f12c0103cb0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12b8044070 con 0x7f12c0103cb0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.806+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c01a2340 con 0x7f12c0103cb0 2024-04-21T08:43:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.807+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b80043a0 con 0x7f12c0103cb0 2024-04-21T08:43:02.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.807+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c01a2960 con 0x7f12c0103cb0 2024-04-21T08:43:02.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.808+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c0109420 con 0x7f12c0103cb0 2024-04-21T08:43:02.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.809+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f12b8004540 con 0x7f12c0103cb0 2024-04-21T08:43:02.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.809+0000 7f12b67fc640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f12ac0a3c30 0x7f12ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:02.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.809+0000 7f12c53cd640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f12ac0a3c30 0x7f12ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:02.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.809+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f12b80f2da0 con 0x7f12c0103cb0 2024-04-21T08:43:02.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.810+0000 7f12c53cd640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f12ac0a3c30 0x7f12ac0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12c006a380 tx=0x7f12bc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:02.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:02.814+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12b80bf4e0 con 0x7f12c0103cb0 2024-04-21T08:43:03.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.041+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12c0062b50 con 0x7f12c0103cb0 2024-04-21T08:43:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.042+0000 7f12b67fc640 1 -- 172.21.15.57:0/241688270 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f12b80bede0 con 0x7f12c0103cb0 2024-04-21T08:43:03.041 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:03.041 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:02.031338+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:43:03.041 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 17 2024-04-21T08:43:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f12ac0a3c30 msgr2=0x7f12ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f12ac0a3c30 0x7f12ac0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f12c006a380 tx=0x7f12bc008040 comp rx=0 tx=0).stop 2024-04-21T08:43:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 msgr2=0x7f12c01017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c01017e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f12b8037040 tx=0x7f12b8035d60 comp rx=0 tx=0).stop 2024-04-21T08:43:03.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 shutdown_connections 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f12ac0a3c30 0x7f12ac0a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f12c0103cb0 0x7f12c01017e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01032f0 0x7f12c01012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 --2- 172.21.15.57:0/241688270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c0100280 0x7f12c0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 >> 172.21.15.57:0/241688270 conn(0x7f12c00fbb90 msgr2=0x7f12c01065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 shutdown_connections 2024-04-21T08:43:03.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.044+0000 7f12c7631640 1 -- 172.21.15.57:0/241688270 wait complete. 2024-04-21T08:43:03.053 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:02 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2018272113' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:43:03.054 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:03.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:02 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2018272113' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:43:03.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:02 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2018272113' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-21T08:43:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 -- 172.21.15.57:0/3717087228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 msgr2=0x7feed8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 --2- 172.21.15.57:0/3717087228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8101290 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feec8009900 tx=0x7feec802bc50 comp rx=0 tx=0).stop 2024-04-21T08:43:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 -- 172.21.15.57:0/3717087228 shutdown_connections 2024-04-21T08:43:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 --2- 172.21.15.57:0/3717087228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8102a10 0x7feed8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 --2- 172.21.15.57:0/3717087228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 0x7feed81024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 --2- 172.21.15.57:0/3717087228 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8101290 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 -- 172.21.15.57:0/3717087228 >> 172.21.15.57:0/3717087228 conn(0x7feed80fc470 msgr2=0x7feed80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 -- 172.21.15.57:0/3717087228 shutdown_connections 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.156+0000 7feeddec0640 1 -- 172.21.15.57:0/3717087228 wait complete. 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 Processor -- start 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 -- start start 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 0x7feed81988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8102a10 0x7feed819f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feed810cb00 con 0x7feed8100e70 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feed810c980 con 0x7feed8102050 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feeddec0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feed810cc80 con 0x7feed8102a10 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feed77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feed77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58916/0 (socket says 172.21.15.57:58916) 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feed77fe640 1 -- 172.21.15.57:0/490322903 learned_addr learned my addr 172.21.15.57:0/490322903 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feed6ffd640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 0x7feed81988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.157+0000 7feed77fe640 1 -- 172.21.15.57:0/490322903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8102a10 msgr2=0x7feed819f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed77fe640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8102a10 0x7feed819f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed77fe640 1 -- 172.21.15.57:0/490322903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 msgr2=0x7feed81988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed77fe640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 0x7feed81988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed77fe640 1 -- 172.21.15.57:0/490322903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed819ffa0 con 0x7feed8100e70 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed6ffd640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 0x7feed81988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed77fe640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8198380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feec8000c00 tx=0x7feec8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8043020 con 0x7feed8100e70 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feec80052b0 con 0x7feed8100e70 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feec8003d90 con 0x7feed8100e70 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feed81a0230 con 0x7feed8100e70 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.158+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feed81a0850 con 0x7feed8100e70 2024-04-21T08:43:03.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.159+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feed81a0b70 con 0x7feed8100e70 2024-04-21T08:43:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.160+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feec8046020 con 0x7feed8100e70 2024-04-21T08:43:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.161+0000 7feed4ff9640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feec00a3c30 0x7feec00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.161+0000 7feed6ffd640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feec00a3c30 0x7feec00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.161+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feec80f3160 con 0x7feed8100e70 2024-04-21T08:43:03.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.161+0000 7feed6ffd640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feec00a3c30 0x7feec00a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feed8199890 tx=0x7feecc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:03.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.164+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feec804d020 con 0x7feed8100e70 2024-04-21T08:43:03.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.351+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feed81a0d40 con 0x7feed8100e70 2024-04-21T08:43:03.352 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.353+0000 7feed4ff9640 1 -- 172.21.15.57:0/490322903 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7feec80bf9c0 con 0x7feed8100e70 2024-04-21T08:43:03.353 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:03.353 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feec00a3c30 msgr2=0x7feec00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feec00a3c30 0x7feec00a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7feed8199890 tx=0x7feecc008040 comp rx=0 tx=0).stop 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 msgr2=0x7feed8198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8198380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feec8000c00 tx=0x7feec8004270 comp rx=0 tx=0).stop 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 shutdown_connections 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feec00a3c30 0x7feec00a60f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8102a10 0x7feed819f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feed8102050 0x7feed81988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 --2- 172.21.15.57:0/490322903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feed8100e70 0x7feed8198380 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 >> 172.21.15.57:0/490322903 conn(0x7feed80fc470 msgr2=0x7feed806a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:03.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 shutdown_connections 2024-04-21T08:43:03.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.355+0000 7feeddec0640 1 -- 172.21.15.57:0/490322903 wait complete. 2024-04-21T08:43:03.367 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 -- 172.21.15.57:0/3342598322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 msgr2=0x7f75b8103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/3342598322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8103610 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009900 tx=0x7f75a802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 -- 172.21.15.57:0/3342598322 shutdown_connections 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/3342598322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b8103b50 0x7f75b810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/3342598322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8103610 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/3342598322 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8101fa0 0x7f75b81023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 -- 172.21.15.57:0/3342598322 >> 172.21.15.57:0/3342598322 conn(0x7f75b80fd530 msgr2=0x7f75b80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 -- 172.21.15.57:0/3342598322 shutdown_connections 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.469+0000 7f75bf5ef640 1 -- 172.21.15.57:0/3342598322 wait complete. 2024-04-21T08:43:03.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 Processor -- start 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 -- start start 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b8101fa0 0x7f75b8075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8103b50 0x7f75b8074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f75b810db20 con 0x7f75b8103190 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f75b810d9a0 con 0x7f75b8101fa0 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bf5ef640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f75b810dca0 con 0x7f75b8103b50 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58920/0 (socket says 172.21.15.57:58920) 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 -- 172.21.15.57:0/2235457751 learned_addr learned my addr 172.21.15.57:0/2235457751 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bdb8c640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8103b50 0x7f75b8074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 -- 172.21.15.57:0/2235457751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8103b50 msgr2=0x7f75b8074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bd38b640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b8101fa0 0x7f75b8075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8103b50 0x7f75b8074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 -- 172.21.15.57:0/2235457751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b8101fa0 msgr2=0x7f75b8075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b8101fa0 0x7f75b8075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 -- 172.21.15.57:0/2235457751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b8075110 con 0x7f75b8103190 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bdb8c640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8103b50 0x7f75b8074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.470+0000 7f75bcb8a640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8074340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f75a8036040 tx=0x7f75a80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.471+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8043020 con 0x7f75b8103190 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.471+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b81a2260 con 0x7f75b8103190 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.471+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75a80054c0 con 0x7f75b8103190 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.471+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a8003db0 con 0x7f75b8103190 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.471+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b81a2830 con 0x7f75b8103190 2024-04-21T08:43:03.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.472+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b8109260 con 0x7f75b8103190 2024-04-21T08:43:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.473+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f75a8046020 con 0x7f75b8103190 2024-04-21T08:43:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.474+0000 7f75ae7fc640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f75a40a3bb0 0x7f75a40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.474+0000 7f75bd38b640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f75a40a3bb0 0x7f75a40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.474+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f75a80f2ff0 con 0x7f75b8103190 2024-04-21T08:43:03.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.474+0000 7f75bd38b640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f75a40a3bb0 0x7f75a40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f75b4005fd0 tx=0x7f75b4005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:03.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.478+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75a804d020 con 0x7f75b8103190 2024-04-21T08:43:03.665 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.666+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75b802d830 con 0x7f75b8103190 2024-04-21T08:43:03.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.667+0000 7f75ae7fc640 1 -- 172.21.15.57:0/2235457751 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7f75a80bf730 con 0x7f75b8103190 2024-04-21T08:43:03.667 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:03.667 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f75a40a3bb0 msgr2=0x7f75a40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f75a40a3bb0 0x7f75a40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f75b4005fd0 tx=0x7f75b4005ce0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 msgr2=0x7f75b8074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8074340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f75a8036040 tx=0x7f75a80042f0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 shutdown_connections 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f75a40a3bb0 0x7f75a40a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75b8103b50 0x7f75b8074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75b8103190 0x7f75b8074340 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 --2- 172.21.15.57:0/2235457751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b8101fa0 0x7f75b8075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 >> 172.21.15.57:0/2235457751 conn(0x7f75b80fd530 msgr2=0x7f75b8106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 shutdown_connections 2024-04-21T08:43:03.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.669+0000 7f75bf5ef640 1 -- 172.21.15.57:0/2235457751 wait complete. 2024-04-21T08:43:03.695 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:43:03.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 -- 172.21.15.57:0/949728388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1878102aa0 msgr2=0x7f1878104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 --2- 172.21.15.57:0/949728388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1878102aa0 0x7f1878104e90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1868009900 tx=0x7f186802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:03.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 -- 172.21.15.57:0/949728388 shutdown_connections 2024-04-21T08:43:03.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 --2- 172.21.15.57:0/949728388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1878107d00 0x7f187810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 --2- 172.21.15.57:0/949728388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18781053d0 0x7f18781077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 --2- 172.21.15.57:0/949728388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1878102aa0 0x7f1878104e90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 -- 172.21.15.57:0/949728388 >> 172.21.15.57:0/949728388 conn(0x7f18780fc410 msgr2=0x7f18780fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 -- 172.21.15.57:0/949728388 shutdown_connections 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.797+0000 7f187d0e4640 1 -- 172.21.15.57:0/949728388 wait complete. 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 Processor -- start 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 -- start start 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1878102aa0 0x7f187819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 0x7f187819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1878107d00 0x7f18781a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f187810db70 con 0x7f18781053d0 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f187810d9f0 con 0x7f1878102aa0 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187d0e4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f187810dcf0 con 0x7f1878107d00 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 0x7f187819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 0x7f187819aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58942/0 (socket says 172.21.15.57:58942) 2024-04-21T08:43:03.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 -- 172.21.15.57:0/597923227 learned_addr learned my addr 172.21.15.57:0/597923227 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f1876d9d640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1878102aa0 0x7f187819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f187759e640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1878107d00 0x7f18781a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 -- 172.21.15.57:0/597923227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1878107d00 msgr2=0x7f18781a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1878107d00 0x7f18781a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 -- 172.21.15.57:0/597923227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1878102aa0 msgr2=0x7f187819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1878102aa0 0x7f187819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.798+0000 7f186ffff640 1 -- 172.21.15.57:0/597923227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18781a20e0 con 0x7f18781053d0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f187759e640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1878107d00 0x7f18781a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f186ffff640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 0x7f187819aa00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f186000b450 tx=0x7f186000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860015020 con 0x7f18781053d0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1860011d40 con 0x7f18781053d0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18781a23d0 con 0x7f18781053d0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186000f340 con 0x7f18781053d0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.799+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18781a2920 con 0x7f18781053d0 2024-04-21T08:43:03.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.800+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18781097f0 con 0x7f18781053d0 2024-04-21T08:43:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.801+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1860018020 con 0x7f18781053d0 2024-04-21T08:43:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.801+0000 7f1874d99640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18640a3bb0 0x7f18640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.802+0000 7f1876d9d640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18640a3bb0 0x7f18640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:03.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.802+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f18600d00e0 con 0x7f18781053d0 2024-04-21T08:43:03.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.802+0000 7f1876d9d640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18640a3bb0 0x7f18640a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1868000c00 tx=0x7f186802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:03.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:03.806+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f186001f020 con 0x7f18781053d0 2024-04-21T08:43:04.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.033+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f187802d5c0 con 0x7f18781053d0 2024-04-21T08:43:04.033 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.033+0000 7f1874d99640 1 -- 172.21.15.57:0/597923227 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11809 (secure 0 0 0) 0x7f186000d040 con 0x7f18781053d0 2024-04-21T08:43:04.033 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:04.033 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:02.031338+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:43:04.033 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 17 2024-04-21T08:43:04.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18640a3bb0 msgr2=0x7f18640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18640a3bb0 0x7f18640a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1868000c00 tx=0x7f186802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 msgr2=0x7f187819aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 0x7f187819aa00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f186000b450 tx=0x7f186000b910 comp rx=0 tx=0).stop 2024-04-21T08:43:04.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 shutdown_connections 2024-04-21T08:43:04.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f18640a3bb0 0x7f18640a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1878107d00 0x7f18781a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18781053d0 0x7f187819aa00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 --2- 172.21.15.57:0/597923227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1878102aa0 0x7f187819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 >> 172.21.15.57:0/597923227 conn(0x7f18780fc410 msgr2=0x7f18781022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 shutdown_connections 2024-04-21T08:43:04.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.036+0000 7f187d0e4640 1 -- 172.21.15.57:0/597923227 wait complete. 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: pgmap v297: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2018272113' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/241688270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/490322903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.045 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2235457751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.047 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: pgmap v297: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2018272113' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/241688270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/490322903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2235457751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: pgmap v297: 129 pgs: 5 creating+peering, 27 unknown, 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: osdmap e89: 12 total, 12 up, 12 in 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2018272113' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/241688270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/490322903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2235457751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:04.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 -- 172.21.15.57:0/1687018909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc105410 msgr2=0x7f3dfc107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/1687018909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc105410 0x7f3dfc107800 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3df0009900 tx=0x7f3df002bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 -- 172.21.15.57:0/1687018909 shutdown_connections 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/1687018909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 0x7f3dfc10a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/1687018909 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc105410 0x7f3dfc107800 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/1687018909 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 -- 172.21.15.57:0/1687018909 >> 172.21.15.57:0/1687018909 conn(0x7f3dfc0fc470 msgr2=0x7f3dfc0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 -- 172.21.15.57:0/1687018909 shutdown_connections 2024-04-21T08:43:04.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.149+0000 7f3e00cc0640 1 -- 172.21.15.57:0/1687018909 wait complete. 2024-04-21T08:43:04.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 Processor -- start 2024-04-21T08:43:04.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 -- start start 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc102450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc105410 0x7f3dfc100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 0x7f3dfc100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc10db70 con 0x7f3dfc102ae0 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc10d9f0 con 0x7f3dfc107d40 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3e00cc0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3dfc10dcf0 con 0x7f3dfc105410 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc102450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58966/0 (socket says 172.21.15.57:58966) 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 -- 172.21.15.57:0/3807161657 learned_addr learned my addr 172.21.15.57:0/3807161657 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfad9d640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 0x7f3dfc100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3df9d9b640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc105410 0x7f3dfc100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 -- 172.21.15.57:0/3807161657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc105410 msgr2=0x7f3dfc100a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc105410 0x7f3dfc100a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 -- 172.21.15.57:0/3807161657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 msgr2=0x7f3dfc100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 0x7f3dfc100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 -- 172.21.15.57:0/3807161657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dfc101860 con 0x7f3dfc102ae0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfad9d640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 0x7f3dfc100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3df9d9b640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc105410 0x7f3dfc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.150+0000 7f3dfa59c640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc102450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3df400e9f0 tx=0x7f3df400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.151+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df400cd80 con 0x7f3dfc102ae0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.151+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3df40054d0 con 0x7f3dfc102ae0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.151+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dfc1a2300 con 0x7f3dfc102ae0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.151+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4010690 con 0x7f3dfc102ae0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.151+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dfc1a2800 con 0x7f3dfc102ae0 2024-04-21T08:43:04.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.152+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dfc0fdf60 con 0x7f3dfc102ae0 2024-04-21T08:43:04.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.153+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3df4004e90 con 0x7f3dfc102ae0 2024-04-21T08:43:04.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.153+0000 7f3deb7fe640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3de40a3bb0 0x7f3de40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.153+0000 7f3df9d9b640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3de40a3bb0 0x7f3de40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.153+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3df40d0be0 con 0x7f3dfc102ae0 2024-04-21T08:43:04.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.154+0000 7f3df9d9b640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3de40a3bb0 0x7f3de40a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3df0036040 tx=0x7f3df0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.158+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3df409d380 con 0x7f3dfc102ae0 2024-04-21T08:43:04.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.347+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3dfc062b50 con 0x7f3dfc102ae0 2024-04-21T08:43:04.348 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.349+0000 7f3deb7fe640 1 -- 172.21.15.57:0/3807161657 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7f3df409cc80 con 0x7f3dfc102ae0 2024-04-21T08:43:04.349 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:04.349 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:04.350 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3de40a3bb0 msgr2=0x7f3de40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3de40a3bb0 0x7f3de40a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3df0036040 tx=0x7f3df0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 msgr2=0x7f3dfc102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc102450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3df400e9f0 tx=0x7f3df400edf0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 shutdown_connections 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3de40a3bb0 0x7f3de40a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dfc107d40 0x7f3dfc100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dfc105410 0x7f3dfc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 --2- 172.21.15.57:0/3807161657 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dfc102ae0 0x7f3dfc102450 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 >> 172.21.15.57:0/3807161657 conn(0x7f3dfc0fc470 msgr2=0x7f3dfc103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 shutdown_connections 2024-04-21T08:43:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.351+0000 7f3e00cc0640 1 -- 172.21.15.57:0/3807161657 wait complete. 2024-04-21T08:43:04.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-21T08:42:56.988323+0000', 'modified': '2024-04-21T08:43:02.031338+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6839/123336774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 123336774}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 123336774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24503, 'qdb_cluster': [24503]} 2024-04-21T08:43:04.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:43:04.362 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:43:04.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 -- 172.21.15.57:0/2696521575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4107d50 msgr2=0x7f2ac410a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 --2- 172.21.15.57:0/2696521575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4107d50 0x7f2ac410a140 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 -- 172.21.15.57:0/2696521575 shutdown_connections 2024-04-21T08:43:04.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 --2- 172.21.15.57:0/2696521575 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4107d50 0x7f2ac410a140 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 --2- 172.21.15.57:0/2696521575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ac4105420 0x7f2ac4107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 --2- 172.21.15.57:0/2696521575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4102af0 0x7f2ac4104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 -- 172.21.15.57:0/2696521575 >> 172.21.15.57:0/2696521575 conn(0x7f2ac40fc440 msgr2=0x7f2ac40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 -- 172.21.15.57:0/2696521575 shutdown_connections 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.466+0000 7f2acc281640 1 -- 172.21.15.57:0/2696521575 wait complete. 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 Processor -- start 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 -- start start 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 0x7f2ac4198210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4105420 0x7f2ac4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ac4107d50 0x7f2ac419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac410db00 con 0x7f2ac4102af0 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac410d980 con 0x7f2ac4105420 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2acc281640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac410dc80 con 0x7f2ac4107d50 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 0x7f2ac4198210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 0x7f2ac4198210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58992/0 (socket says 172.21.15.57:58992) 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 -- 172.21.15.57:0/369528059 learned_addr learned my addr 172.21.15.57:0/369528059 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca81e640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ac4107d50 0x7f2ac419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2ac981c640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4105420 0x7f2ac4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 -- 172.21.15.57:0/369528059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ac4107d50 msgr2=0x7f2ac419f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ac4107d50 0x7f2ac419f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 -- 172.21.15.57:0/369528059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4105420 msgr2=0x7f2ac4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4105420 0x7f2ac4198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 -- 172.21.15.57:0/369528059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac419fe30 con 0x7f2ac4102af0 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2ac981c640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4105420 0x7f2ac4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.467+0000 7f2aca01d640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 0x7f2ac4198210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000d950 tx=0x7f2ac000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.468+0000 7f2abb7fe640 1 -- 172.21.15.57:0/369528059 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0017070 con 0x7f2ac4102af0 2024-04-21T08:43:04.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.468+0000 7f2abb7fe640 1 -- 172.21.15.57:0/369528059 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ac0009c30 con 0x7f2ac4102af0 2024-04-21T08:43:04.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.468+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac41a0120 con 0x7f2ac4102af0 2024-04-21T08:43:04.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.468+0000 7f2abb7fe640 1 -- 172.21.15.57:0/369528059 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000f620 con 0x7f2ac4102af0 2024-04-21T08:43:04.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.468+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac41a0670 con 0x7f2ac4102af0 2024-04-21T08:43:04.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.470+0000 7f2abb7fe640 1 -- 172.21.15.57:0/369528059 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2ac0010460 con 0x7f2ac4102af0 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.470+0000 7f2acee73740 1 Processor -- start 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.470+0000 7f2acee73740 1 -- start start 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.470+0000 7f2acee73740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459d95260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.470+0000 7f2acee73740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d95830 0x557459d95cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2acee73740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459d9ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2acee73740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557459cf9b10 con 0x557459d94e40 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2acee73740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557459c97240 con 0x557459d95830 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2acee73740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557459cf5ad0 con 0x557459d962d0 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459d95260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459d95260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59000/0 (socket says 172.21.15.57:59000) 2024-04-21T08:43:04.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 -- 172.21.15.57:0/3544998608 learned_addr learned my addr 172.21.15.57:0/3544998608 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2abb7fe640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2ab00a3bb0 0x7f2ab00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2ac981c640 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459d9ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca01d640 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d95830 0x557459d95cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 -- 172.21.15.57:0/3544998608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 msgr2=0x557459d9ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459d9ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 -- 172.21.15.57:0/3544998608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d95830 msgr2=0x557459d95cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d95830 0x557459d95cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 -- 172.21.15.57:0/3544998608 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557459a0c2d0 con 0x557459d94e40 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2ab00a3bb0 0x7f2ab00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2abb7fe640 1 -- 172.21.15.57:0/369528059 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2ac00d0270 con 0x7f2ac4102af0 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.471+0000 7f2aca81e640 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459d95260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2abc02c700 tx=0x7f2abc0099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd6862166a857a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2ab8bfe640 1 -- 172.21.15.57:0/3544998608 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc039040 con 0x557459d94e40 2024-04-21T08:43:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2aca81e640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2ab00a3bb0 0x7f2ab00a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2abc048000 tx=0x7f2abc045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2ab8bfe640 1 -- 172.21.15.57:0/3544998608 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc032e80 con 0x557459d94e40 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2ab8bfe640 1 -- 172.21.15.57:0/3544998608 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc04a440 con 0x557459d94e40 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 -- 172.21.15.57:0/3544998608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 msgr2=0x557459d95260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459d95260 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2abc02c700 tx=0x7f2abc0099b0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 -- 172.21.15.57:0/3544998608 shutdown_connections 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459d9ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d95830 0x557459d95cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 --2- 172.21.15.57:0/3544998608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459d95260 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 -- 172.21.15.57:0/3544998608 >> 172.21.15.57:0/3544998608 conn(0x557459d90f40 msgr2=0x557459d8f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 -- 172.21.15.57:0/3544998608 shutdown_connections 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.472+0000 7f2acee73740 1 -- 172.21.15.57:0/3544998608 wait complete. 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 -- start start 2024-04-21T08:43:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 Processor -- start 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459d9ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 0x557459d9f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459da6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557459d9d4b0 con 0x557459d95830 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557459c82190 con 0x557459d94e40 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557459d9d630 con 0x557459d962d0 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2acee73740 10 client.0 _ll_register_callbacks cb 0x557459c7d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 0x557459d9f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 0x557459d9f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59016/0 (socket says 172.21.15.57:59016) 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 -- 172.21.15.57:0/2387933816 learned_addr learned my addr 172.21.15.57:0/2387933816 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2aca01d640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459d9ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 -- 172.21.15.57:0/2387933816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 msgr2=0x557459da6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459da6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 -- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 msgr2=0x557459d9ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459d9ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2ac981c640 1 -- 172.21.15.57:0/2387933816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557459ad66b0 con 0x557459d95830 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.473+0000 7f2aca01d640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459d9ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2ac981c640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 0x557459d9f4b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ac406aff0 tx=0x7f2ab40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525e510a92bd279f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2aadffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2aadffb640 1 -- 172.21.15.57:0/2387933816 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4012650 con 0x557459d95830 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2aadffb640 1 -- 172.21.15.57:0/2387933816 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2ab40127f0 con 0x557459d95830 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2abc032e80 con 0x557459d95830 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2aadffb640 1 -- 172.21.15.57:0/2387933816 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2ab4011e60 con 0x557459d95830 2024-04-21T08:43:04.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 10 client.14748 fetch_fsmap learned FSMap version 17 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x557459da9910 con 0x557459d95830 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2aadffb640 1 -- 172.21.15.57:0/2387933816 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc032e80 con 0x557459d95830 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2aadffb640 1 -- 172.21.15.57:0/2387933816 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f2ab401ab70 con 0x557459d95830 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 10 client.14748 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 10 client.14748 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 20 client.14748 populate_metadata read hostname 'smithi057' 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459dac090 0x557459dad500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 4 client.14748 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.474+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557459d07530 con 0x557459dac090 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.475+0000 7f2aca01d640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459dac090 0x557459dad500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.475+0000 7f2aca01d640 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459dac090 0x557459dad500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=584d7a2b2beaf440 server_cookie=542d2a86f1d2bcf7 in_seq=0 out_seq=0 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.475+0000 7f2aadffb640 10 client.14748 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2aadffb640 1 -- 172.21.15.57:0/2387933816 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f2ac0081ae0 con 0x557459dac090 2024-04-21T08:43:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2aadffb640 10 client.14748 handle_command_reply: tid=0 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 client.14748 shutdown 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 msgr2=0x557459d9f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 0x557459d9f4b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ac406aff0 tx=0x7f2ab40117b0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 shutdown_connections 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459dac090 0x557459dad500 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d962d0 0x557459da6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d95830 0x557459d9f4b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2aadffb640 0 client.14748 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 --2- 172.21.15.57:0/2387933816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459d9ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.476+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 >> 172.21.15.57:0/2387933816 conn(0x557459d90f40 msgr2=0x557459d8ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.477+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 shutdown_connections 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.477+0000 7f2acee73740 1 -- 172.21.15.57:0/2387933816 wait complete. 2024-04-21T08:43:04.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.477+0000 7f2acee73740 20 client.14748 trim_cache size 0 max 16384 2024-04-21T08:43:04.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 Processor -- start 2024-04-21T08:43:04.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 -- start start 2024-04-21T08:43:04.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459da6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 0x557459d8a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557459dab840 con 0x557459d962d0 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557459dabf10 con 0x557459d94e40 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2acee73740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557459dac5e0 con 0x557459d95830 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 0x557459d8a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 0x557459d8a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59022/0 (socket says 172.21.15.57:59022) 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 -- 172.21.15.57:0/182569104 learned_addr learned my addr 172.21.15.57:0/182569104 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca01d640 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459da6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 -- 172.21.15.57:0/182569104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 msgr2=0x557459da6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 -- 172.21.15.57:0/182569104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 msgr2=0x557459da6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459da6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca81e640 1 -- 172.21.15.57:0/182569104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557459d8fca0 con 0x557459d962d0 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.495+0000 7f2aca01d640 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459da6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2aca81e640 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 0x557459d8a800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89da9dfdb80cbcb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2aaffff640 1 -- 172.21.15.57:0/182569104 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc057020 con 0x557459d962d0 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2aaffff640 1 -- 172.21.15.57:0/182569104 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc04a440 con 0x557459d962d0 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2aaffff640 1 -- 172.21.15.57:0/182569104 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2abc039070 con 0x557459d962d0 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 -- 172.21.15.57:0/182569104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 msgr2=0x557459d8a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 0x557459d8a800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2abc009ca0 tx=0x7f2abc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 -- 172.21.15.57:0/182569104 shutdown_connections 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d962d0 0x557459d8a800 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 --2- 172.21.15.57:0/182569104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d94e40 0x557459da6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 -- 172.21.15.57:0/182569104 >> 172.21.15.57:0/182569104 conn(0x557459d90f40 msgr2=0x557459d8f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 -- 172.21.15.57:0/182569104 shutdown_connections 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.496+0000 7f2acee73740 1 -- 172.21.15.57:0/182569104 wait complete. 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 -- start start 2024-04-21T08:43:04.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 Processor -- start 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459da2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d962d0 0x557459da4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557459dad380 con 0x557459d94e40 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557459daccb0 con 0x557459d962d0 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557459dada50 con 0x557459d95830 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2acee73740 10 client.0 _ll_register_callbacks cb 0x7f2ab400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459da2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459da2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59038/0 (socket says 172.21.15.57:59038) 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 -- 172.21.15.57:0/155271700 learned_addr learned my addr 172.21.15.57:0/155271700 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2ac981c640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca81e640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d962d0 0x557459da4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 -- 172.21.15.57:0/155271700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 msgr2=0x557459da4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 -- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d962d0 msgr2=0x557459da4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d962d0 0x557459da4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 -- 172.21.15.57:0/155271700 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557459da55f0 con 0x557459d94e40 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca81e640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d962d0 0x557459da4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aca01d640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459da2420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0000c00 tx=0x7f2ac0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4365a37ffe0d0ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.497+0000 7f2aadffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2aadffb640 1 -- 172.21.15.57:0/155271700 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0020040 con 0x557459d94e40 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2aadffb640 1 -- 172.21.15.57:0/155271700 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2ac0098980 con 0x557459d94e40 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2abc04a440 con 0x557459d94e40 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2aadffb640 1 -- 172.21.15.57:0/155271700 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2ac009c310 con 0x557459d94e40 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 10 client.14754 fetch_fsmap learned FSMap version 17 2024-04-21T08:43:04.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x557459e12040 con 0x557459d94e40 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2aadffb640 1 -- 172.21.15.57:0/155271700 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2abc04a440 con 0x557459d94e40 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2aadffb640 1 -- 172.21.15.57:0/155271700 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f2ac0098060 con 0x557459d94e40 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 10 client.14754 fetch_fsmap finished waiting for FSMap version 17 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 10 client.14754 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 20 client.14754 populate_metadata read hostname 'smithi057' 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459e16b90 0x557459e158a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 4 client.14754 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "status"}) -- 0x557459d013a0 con 0x557459e16b90 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.498+0000 7f2ac981c640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459e16b90 0x557459e158a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.499+0000 7f2aadffb640 10 client.14754 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.499+0000 7f2ac981c640 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459e16b90 0x557459e158a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34a7d0f2186ef543 server_cookie=ba66c440f824ca2e in_seq=0 out_seq=0 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.499+0000 7f2aadffb640 1 -- 172.21.15.57:0/155271700 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x557459d013a0 con 0x557459e16b90 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.499+0000 7f2aadffb640 10 client.14754 handle_command_reply: tid=0 2024-04-21T08:43:04.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.499+0000 7f2acee73740 1 client.14754 shutdown 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 msgr2=0x557459da2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459da2420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0000c00 tx=0x7f2ac0005880 comp rx=0 tx=0).stop 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 shutdown_connections 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557459e16b90 0x557459e158a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557459d962d0 0x557459da4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2aadffb640 0 client.14754 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557459d95830 0x557459da4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 --2- 172.21.15.57:0/155271700 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557459d94e40 0x557459da2420 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 >> 172.21.15.57:0/155271700 conn(0x557459d90f40 msgr2=0x557459d8a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 shutdown_connections 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 1 -- 172.21.15.57:0/155271700 wait complete. 2024-04-21T08:43:04.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acee73740 20 client.14754 trim_cache size 0 max 16384 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 24503, 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 6.878312803, 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 17, 2024-04-21T08:43:04.507 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 89, 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 13.181152515000001 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2ab00a3bb0 msgr2=0x7f2ab00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2ab00a3bb0 0x7f2ab00a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2abc048000 tx=0x7f2abc045000 comp rx=0 tx=0).stop 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 msgr2=0x7f2ac4198210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:04.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 0x7f2ac4198210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ac000d950 tx=0x7f2ac000de10 comp rx=0 tx=0).stop 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 shutdown_connections 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2ab00a3bb0 0x7f2ab00a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ac4107d50 0x7f2ac419f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ac4105420 0x7f2ac4198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 --2- 172.21.15.57:0/369528059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac4102af0 0x7f2ac4198210 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 >> 172.21.15.57:0/369528059 conn(0x7f2ac40fc440 msgr2=0x7f2ac4103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 shutdown_connections 2024-04-21T08:43:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:04.500+0000 7f2acc281640 1 -- 172.21.15.57:0/369528059 wait complete. 2024-04-21T08:43:04.510 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:43:04.510 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:04.511 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:43:04.757 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:05.032 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:04 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/597923227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.032 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:04 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3807161657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:04 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/597923227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:04 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3807161657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:04 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/597923227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:05.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:04 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3807161657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:06.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:05 smithi129 ceph-mon[31272]: pgmap v299: 129 pgs: 5 creating+peering, 17 unknown, 107 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-21T08:43:06.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:05 smithi129 ceph-mon[31272]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:43:06.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:05 smithi129 ceph-mon[31272]: Cluster is now healthy 2024-04-21T08:43:06.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:05 smithi057 ceph-mon[27968]: pgmap v299: 129 pgs: 5 creating+peering, 17 unknown, 107 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-21T08:43:06.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:05 smithi057 ceph-mon[27968]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:43:06.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:05 smithi057 ceph-mon[27968]: Cluster is now healthy 2024-04-21T08:43:06.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:05 smithi136 ceph-mon[31791]: pgmap v299: 129 pgs: 5 creating+peering, 17 unknown, 107 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-21T08:43:06.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:05 smithi136 ceph-mon[31791]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-21T08:43:06.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:05 smithi136 ceph-mon[31791]: Cluster is now healthy 2024-04-21T08:43:06.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 -- 172.21.15.57:0/4148233737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 msgr2=0x7f5f40101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 --2- 172.21.15.57:0/4148233737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40101db0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009990 tx=0x7f5f34009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 -- 172.21.15.57:0/4148233737 shutdown_connections 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 --2- 172.21.15.57:0/4148233737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f40107980 0x7f5f40107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 --2- 172.21.15.57:0/4148233737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f401022f0 0x7f5f4010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 --2- 172.21.15.57:0/4148233737 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40101db0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 -- 172.21.15.57:0/4148233737 >> 172.21.15.57:0/4148233737 conn(0x7f5f400788b0 msgr2=0x7f5f40078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 -- 172.21.15.57:0/4148233737 shutdown_connections 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.495+0000 7f5f48696700 1 -- 172.21.15.57:0/4148233737 wait complete. 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 Processor -- start 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 -- start start 2024-04-21T08:43:06.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f401022f0 0x7f5f4019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f40107980 0x7f5f401a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5f40111ad0 con 0x7f5f40101930 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5f40111950 con 0x7f5f401022f0 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f48696700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5f40111c50 con 0x7f5f40107980 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f46432700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f46432700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59064/0 (socket says 172.21.15.57:59064) 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f46432700 1 -- 172.21.15.57:0/2355691792 learned_addr learned my addr 172.21.15.57:0/2355691792 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f45c31700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f401022f0 0x7f5f4019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.496+0000 7f5f46c33700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f40107980 0x7f5f401a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46432700 1 -- 172.21.15.57:0/2355691792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f40107980 msgr2=0x7f5f401a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46432700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f40107980 0x7f5f401a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46432700 1 -- 172.21.15.57:0/2355691792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f401022f0 msgr2=0x7f5f4019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46432700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f401022f0 0x7f5f4019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46432700 1 -- 172.21.15.57:0/2355691792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f401a54f0 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46c33700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f40107980 0x7f5f401a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f46432700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40199f30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5f38007f10 tx=0x7f5f3800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3800ddd0 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f38004bb0 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f401a5720 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f380046a0 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.497+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f401a5c30 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.498+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f40103050 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.499+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5f3801c020 con 0x7f5f40101930 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.500+0000 7f5f3f7fe700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5f280a3c40 0x7f5f280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.500+0000 7f5f45c31700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5f280a3c40 0x7f5f280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:06.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.500+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5f380cff00 con 0x7f5f40101930 2024-04-21T08:43:06.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.500+0000 7f5f45c31700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5f280a3c40 0x7f5f280a6100 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400b5c0 tx=0x7f5f34005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:06.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.504+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f38010070 con 0x7f5f40101930 2024-04-21T08:43:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.761+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5f4002df80 con 0x7f5f40101930 2024-04-21T08:43:06.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.761+0000 7f5f3f7fe700 1 -- 172.21.15.57:0/2355691792 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f5f3809c640 con 0x7f5f40101930 2024-04-21T08:43:06.761 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:43:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5f280a3c40 msgr2=0x7f5f280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5f280a3c40 0x7f5f280a6100 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400b5c0 tx=0x7f5f34005960 comp rx=0 tx=0).stop 2024-04-21T08:43:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 msgr2=0x7f5f40199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40199f30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5f38007f10 tx=0x7f5f3800d3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 shutdown_connections 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5f280a3c40 0x7f5f280a6100 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f40107980 0x7f5f401a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5f40101930 0x7f5f40199f30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 --2- 172.21.15.57:0/2355691792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f401022f0 0x7f5f4019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 >> 172.21.15.57:0/2355691792 conn(0x7f5f400788b0 msgr2=0x7f5f400ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 shutdown_connections 2024-04-21T08:43:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:06.764+0000 7f5f48696700 1 -- 172.21.15.57:0/2355691792 wait complete. 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:43:07.596 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:43:07.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:07.597 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:43:07.615 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42204sec preferred_lft 42204sec 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:43:07.616 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2024-04-21T08:43:07.617 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr flush dev ceph-brx 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set ceph-brx up 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:43:07.617 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:43:07.708 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:43:07.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:07.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:07.809 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:43:07.817 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:07 smithi057 ceph-mon[27968]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-21T08:43:07.817 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:07 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2355691792' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:43:07.834 INFO:teuthology.orchestra.run.smithi057.stdout:1 2024-04-21T08:43:07.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:07.836 DEBUG:teuthology.orchestra.run.smithi057:> route 2024-04-21T08:43:07.909 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2024-04-21T08:43:07.910 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:43:07.910 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T08:43:07.910 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T08:43:07.910 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:43:07.910 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:43:07.911 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:43:07.911 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T08:43:07.911 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T08:43:07.911 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T08:43:07.911 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:43:08.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:43:08.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:07 smithi129 ceph-mon[31272]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-21T08:43:08.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:07 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2355691792' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:43:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:07 smithi136 ceph-mon[31791]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-21T08:43:08.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:07 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2355691792' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:43:08.166 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:08.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:08.173 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2024-04-21T08:43:08.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:08.189 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2024-04-21T08:43:08.244 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:43:08.244 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:43:08.245 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:43:08.245 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-21T08:43:08.245 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:43:08.334 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:43:08.357 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:08.364 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-21T08:43:08.364 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:43:08.414 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:43:08.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:08.490 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:43:08.490 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:43:08.490 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2024-04-21T08:43:08.490 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:43:08.490 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:43:08.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:43:08.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:08.568 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-21T08:43:08.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:43:08.568 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:08.584 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-21T08:43:08.584 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:43:08.584 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:08.639 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2024-04-21T08:43:08.705 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2024-04-21T08:43:08.761 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2024-04-21T08:43:08.762 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/aecd3bd6fb05bb72400d554c2a922ac5fcd22cb15e514f7f8d8bcd525282cd7f/merged 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/b7d7f881fd0fc570a0089059c4f62a4ff64004b873e64a8bdfe2243468839007/merged 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/0 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/18fdd7414298896357144884b3765c4eb6a931c6d3b3aca3e032139439822a77/merged 2024-04-21T08:43:08.763 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/cbe495251376818f18be36cbbff591b680de54cd701db6a16213142227c7778b/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/5f7708329f3271139947a710ebaafb8c9669c1882e4ae9ffeffa205a51f1863a/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/7caf121605e9fa79fb0030d0812d3b77c54187451561e111abb4022822e74522/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/f7f8d40b10efcdbdcfb5beff5c8d73136584a7f59c55ecb1984cdc386f505e3f/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/a9013f0f98304d3f64666293554b7c968d893b1b2a05f961386e989eface7656/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/6d41099c4b748046550a78bb0c14c1e93c50d8a1e8b532b105c188ffa5e145ba/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/bde5902fe085e21d41c0534716f85621d7ff92fd408f42e95eb598f7eecd7b9b/merged 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:43:08.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:08.765 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2024-04-21T08:43:08.817 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-21T08:43:08.817 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-21T08:43:08.859 DEBUG:teuthology.orchestra.run.smithi057:> sudo modprobe fuse 2024-04-21T08:43:08.884 DEBUG:teuthology.orchestra.run.smithi057:> cat /proc/self/mounts | awk '{print $2}' 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/proc 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/sys 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/dev 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/security 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/shm 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/pts 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/run 2024-04-21T08:43:08.940 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/cgroup 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/pstore 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/bpf 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/ 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/selinux 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/hugepages 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/dev/mqueue 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/debug 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/tracing 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysusers.service 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/kernel/config 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/sys/fs/fuse/connections 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-sysctl.service 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-21T08:43:08.941 INFO:teuthology.orchestra.run.smithi057.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/nfs/rpc_pipefs 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/1000 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/ceph 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/aecd3bd6fb05bb72400d554c2a922ac5fcd22cb15e514f7f8d8bcd525282cd7f/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/b7d7f881fd0fc570a0089059c4f62a4ff64004b873e64a8bdfe2243468839007/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/run/user/0 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/proc/sys/fs/binfmt_misc 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/18fdd7414298896357144884b3765c4eb6a931c6d3b3aca3e032139439822a77/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/cbe495251376818f18be36cbbff591b680de54cd701db6a16213142227c7778b/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/5f7708329f3271139947a710ebaafb8c9669c1882e4ae9ffeffa205a51f1863a/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/7caf121605e9fa79fb0030d0812d3b77c54187451561e111abb4022822e74522/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/f7f8d40b10efcdbdcfb5beff5c8d73136584a7f59c55ecb1984cdc386f505e3f/merged 2024-04-21T08:43:08.942 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/a9013f0f98304d3f64666293554b7c968d893b1b2a05f961386e989eface7656/merged 2024-04-21T08:43:08.943 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/6d41099c4b748046550a78bb0c14c1e93c50d8a1e8b532b105c188ffa5e145ba/merged 2024-04-21T08:43:08.943 INFO:teuthology.orchestra.run.smithi057.stdout:/var/lib/containers/storage/overlay/bde5902fe085e21d41c0534716f85621d7ff92fd408f42e95eb598f7eecd7b9b/merged 2024-04-21T08:43:08.943 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns 2024-04-21T08:43:08.943 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:43:08.943 INFO:teuthology.orchestra.run.smithi057.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:43:08.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:08.943 DEBUG:teuthology.orchestra.run.smithi057:> ls /sys/fs/fuse/connections 2024-04-21T08:43:08.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:2024-04-21T08:43:08.958+0000 7fa437441480 -1 init, newargv = 0x559e6b5fbf70 newargc=15 2024-04-21T08:43:08.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[72690]: starting ceph client 2024-04-21T08:43:08.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[72690]: starting fuse 2024-04-21T08:43:08.973 INFO:teuthology.orchestra.run.smithi057.stdout:115 2024-04-21T08:43:08.973 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> sudo stdin-killer -- python3 -c ' 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> import glob 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> import re 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> import os 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> import subprocess 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> def _find_admin_socket(client_name): 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> files = glob.glob(asok_path) 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> # Given a non-glob path, it better be there 2024-04-21T08:43:08.974 DEBUG:teuthology.orchestra.run.smithi057:> if "*" not in asok_path: 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> assert(len(files) == 1) 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> return files[0] 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> for f in files: 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> contents = proc_f.read() 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> if mountpoint in contents: 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> return f 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> print(_find_admin_socket("client.admin")) 2024-04-21T08:43:08.975 DEBUG:teuthology.orchestra.run.smithi057:> ' 2024-04-21T08:43:09.094 INFO:teuthology.orchestra.run.smithi057.stdout:/var/run/ceph/ceph-client.admin.72690.asok 2024-04-21T08:43:09.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:09.104 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72690.asok 2024-04-21T08:43:09.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:09.104 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72690.asok status 2024-04-21T08:43:09.240 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:43:09.240 INFO:teuthology.orchestra.run.smithi057.stdout: "metadata": { 2024-04-21T08:43:09.240 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_sha1": "a630765c1f93c6e61d9b40c4b76437611d5ef29e", 2024-04-21T08:43:09.240 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_version": "ceph version 19.0.0-3154-ga630765c (a630765c1f93c6e61d9b40c4b76437611d5ef29e) squid (dev)", 2024-04-21T08:43:09.240 INFO:teuthology.orchestra.run.smithi057.stdout: "entity_id": "admin", 2024-04-21T08:43:09.240 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": "smithi057", 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "pid": "72690", 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "root": "/" 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_count": 0, 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "dentry_pinned_count": 0, 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 14766, 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "inst": { 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "name": { 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "client", 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "num": 14766 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2024-04-21T08:43:09.241 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 821274997 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": { 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "192.168.0.1:0", 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 821274997 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "inst_str": "client.14766 192.168.0.1:0/821274997", 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "addr_str": "192.168.0.1:0/821274997", 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "inode_count": 1, 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_epoch": 17, 2024-04-21T08:43:09.242 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch": 89, 2024-04-21T08:43:09.243 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_epoch_barrier": 0, 2024-04-21T08:43:09.243 INFO:teuthology.orchestra.run.smithi057.stdout: "blocklisted": false, 2024-04-21T08:43:09.243 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs" 2024-04-21T08:43:09.243 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:43:09.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:09.249 DEBUG:teuthology.orchestra.run.smithi057:> stat --file-system '--printf=%T 2024-04-21T08:43:09.249 DEBUG:teuthology.orchestra.run.smithi057:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:09.264 INFO:teuthology.orchestra.run.smithi057.stdout:fuseblk 2024-04-21T08:43:09.264 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:09.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:09.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:09.330 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-21T08:43:09.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:43:09.443 INFO:teuthology.orchestra.run.smithi057.stdout:# file: . 2024-04-21T08:43:09.443 INFO:teuthology.orchestra.run.smithi057.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-21T08:43:09.444 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:09.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:43:09.449 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:43:09.449 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:43:09.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 -- 172.21.15.57:0/2687166836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 msgr2=0x7f5c2c104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 --2- 172.21.15.57:0/2687166836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c104e30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009900 tx=0x7f5c1c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 -- 172.21.15.57:0/2687166836 shutdown_connections 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 --2- 172.21.15.57:0/2687166836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c2c107ca0 0x7f5c2c10a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 --2- 172.21.15.57:0/2687166836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c2c105370 0x7f5c2c107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 --2- 172.21.15.57:0/2687166836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c104e30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 -- 172.21.15.57:0/2687166836 >> 172.21.15.57:0/2687166836 conn(0x7f5c2c0fc470 msgr2=0x7f5c2c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:09.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 -- 172.21.15.57:0/2687166836 shutdown_connections 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.608+0000 7f5c32150640 1 -- 172.21.15.57:0/2687166836 wait complete. 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 Processor -- start 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 -- start start 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c1023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c2c105370 0x7f5c2c1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c2c107ca0 0x7f5c2c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c10db00 con 0x7f5c2c102a40 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c10d980 con 0x7f5c2c105370 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c32150640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c10dc80 con 0x7f5c2c107ca0 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c1023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c1023b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59088/0 (socket says 172.21.15.57:59088) 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2b7fe640 1 -- 172.21.15.57:0/1231599450 learned_addr learned my addr 172.21.15.57:0/1231599450 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:09.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2affd640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c2c105370 0x7f5c2c1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2bfff640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c2c107ca0 0x7f5c2c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2b7fe640 1 -- 172.21.15.57:0/1231599450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c2c107ca0 msgr2=0x7f5c2c100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.609+0000 7f5c2b7fe640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c2c107ca0 0x7f5c2c100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c2b7fe640 1 -- 172.21.15.57:0/1231599450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c2c105370 msgr2=0x7f5c2c1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c2b7fe640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c2c105370 0x7f5c2c1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c2b7fe640 1 -- 172.21.15.57:0/1231599450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c1017c0 con 0x7f5c2c102a40 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c2b7fe640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c1023b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c006010 tx=0x7f5c1c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c043020 con 0x7f5c2c102a40 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c1a22b0 con 0x7f5c2c102a40 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c1c0043d0 con 0x7f5c2c102a40 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c003b80 con 0x7f5c2c102a40 2024-04-21T08:43:09.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.610+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c1a2800 con 0x7f5c2c102a40 2024-04-21T08:43:09.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.611+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c2c0fdf60 con 0x7f5c2c102a40 2024-04-21T08:43:09.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.612+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c1c046020 con 0x7f5c2c102a40 2024-04-21T08:43:09.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.613+0000 7f5c28ff9640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5c140a3bb0 0x7f5c140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.613+0000 7f5c2affd640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5c140a3bb0 0x7f5c140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.613+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5c1c0f2fd0 con 0x7f5c2c102a40 2024-04-21T08:43:09.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.613+0000 7f5c2affd640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5c140a3bb0 0x7f5c140a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c102120 tx=0x7f5c20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:09.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.616+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c1c04d020 con 0x7f5c2c102a40 2024-04-21T08:43:09.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.809+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c2c062b50 con 0x7f5c2c102a40 2024-04-21T08:43:09.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.811+0000 7f5c28ff9640 1 -- 172.21.15.57:0/1231599450 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7f5c1c0bf710 con 0x7f5c2c102a40 2024-04-21T08:43:09.810 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:09.810 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5c140a3bb0 msgr2=0x7f5c140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.812 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5c140a3bb0 0x7f5c140a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c102120 tx=0x7f5c20005f90 comp rx=0 tx=0).stop 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 msgr2=0x7f5c2c1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c1023b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c006010 tx=0x7f5c1c005e40 comp rx=0 tx=0).stop 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 shutdown_connections 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5c140a3bb0 0x7f5c140a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c2c107ca0 0x7f5c2c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c2c105370 0x7f5c2c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 --2- 172.21.15.57:0/1231599450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c2c102a40 0x7f5c2c1023b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 >> 172.21.15.57:0/1231599450 conn(0x7f5c2c0fc470 msgr2=0x7f5c2c1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 shutdown_connections 2024-04-21T08:43:09.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.813+0000 7f5c32150640 1 -- 172.21.15.57:0/1231599450 wait complete. 2024-04-21T08:43:09.823 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:43:09.824 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:43:09.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 -- 172.21.15.57:0/3004025886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102040 msgr2=0x7fc9ec1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/3004025886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102040 0x7fc9ec1024c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009900 tx=0x7fc9e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 -- 172.21.15.57:0/3004025886 shutdown_connections 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/3004025886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9ec102a00 0x7fc9ec109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/3004025886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102040 0x7fc9ec1024c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/3004025886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9ec100e60 0x7fc9ec101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 -- 172.21.15.57:0/3004025886 >> 172.21.15.57:0/3004025886 conn(0x7fc9ec0fc440 msgr2=0x7fc9ec0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 -- 172.21.15.57:0/3004025886 shutdown_connections 2024-04-21T08:43:09.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.930+0000 7fc9f0dce640 1 -- 172.21.15.57:0/3004025886 wait complete. 2024-04-21T08:43:09.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 Processor -- start 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 -- start start 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9ec100e60 0x7fc9ec1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9ec102040 0x7fc9ec1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 0x7fc9ec19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10db30 con 0x7fc9ec102a00 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10d9b0 con 0x7fc9ec100e60 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9f0dce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ec10dcb0 con 0x7fc9ec102040 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 0x7fc9ec19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 0x7fc9ec19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59118/0 (socket says 172.21.15.57:59118) 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 -- 172.21.15.57:0/2980193782 learned_addr learned my addr 172.21.15.57:0/2980193782 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ea59c640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9ec100e60 0x7fc9ec1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9e9d9b640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9ec102040 0x7fc9ec1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 -- 172.21.15.57:0/2980193782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9ec102040 msgr2=0x7fc9ec1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9ec102040 0x7fc9ec1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 -- 172.21.15.57:0/2980193782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9ec100e60 msgr2=0x7fc9ec1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9ec100e60 0x7fc9ec1981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.931+0000 7fc9ead9d640 1 -- 172.21.15.57:0/2980193782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9ec19fdd0 con 0x7fc9ec102a00 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.932+0000 7fc9ead9d640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 0x7fc9ec19f760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400cce0 tx=0x7fc9e400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.932+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400f450 con 0x7fc9ec102a00 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.932+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e400faa0 con 0x7fc9ec102a00 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.932+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4018b70 con 0x7fc9ec102a00 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.932+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9ec1a00c0 con 0x7fc9ec102a00 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.932+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9ec1a0590 con 0x7fc9ec102a00 2024-04-21T08:43:09.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.933+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9ec108200 con 0x7fc9ec102a00 2024-04-21T08:43:09.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.934+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc9e4018d10 con 0x7fc9ec102a00 2024-04-21T08:43:09.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.934+0000 7fc9db7fe640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc9d40a3bb0 0x7fc9d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:09.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.935+0000 7fc9ea59c640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc9d40a3bb0 0x7fc9d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:09.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.935+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc9e40d09e0 con 0x7fc9ec102a00 2024-04-21T08:43:09.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.935+0000 7fc9ea59c640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc9d40a3bb0 0x7fc9d40a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec101bb0 tx=0x7fc9dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:09.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:09.939+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9e4014030 con 0x7fc9ec102a00 2024-04-21T08:43:10.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:09 smithi129 ceph-mon[31272]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 386 B/s wr, 1 op/s 2024-04-21T08:43:10.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:09 smithi057 ceph-mon[27968]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 386 B/s wr, 1 op/s 2024-04-21T08:43:10.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.125+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc9ec062b50 con 0x7fc9ec102a00 2024-04-21T08:43:10.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.125+0000 7fc9db7fe640 1 -- 172.21.15.57:0/2980193782 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fc9e409d180 con 0x7fc9ec102a00 2024-04-21T08:43:10.125 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc9d40a3bb0 msgr2=0x7fc9d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc9d40a3bb0 0x7fc9d40a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec101bb0 tx=0x7fc9dc005f90 comp rx=0 tx=0).stop 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 msgr2=0x7fc9ec19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 0x7fc9ec19f760 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400cce0 tx=0x7fc9e400a620 comp rx=0 tx=0).stop 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 shutdown_connections 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc9d40a3bb0 0x7fc9d40a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9ec102a00 0x7fc9ec19f760 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9ec102040 0x7fc9ec1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 --2- 172.21.15.57:0/2980193782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9ec100e60 0x7fc9ec1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 >> 172.21.15.57:0/2980193782 conn(0x7fc9ec0fc440 msgr2=0x7fc9ec1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:10.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 shutdown_connections 2024-04-21T08:43:10.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.127+0000 7fc9f0dce640 1 -- 172.21.15.57:0/2980193782 wait complete. 2024-04-21T08:43:10.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:09 smithi136 ceph-mon[31791]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 386 B/s wr, 1 op/s 2024-04-21T08:43:10.137 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.242+0000 7fa40f2c8640 1 -- 172.21.15.57:0/289465745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 msgr2=0x7fa408102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.242+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/289465745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa408102460 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009900 tx=0x7fa3fc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.242+0000 7fa40f2c8640 1 -- 172.21.15.57:0/289465745 shutdown_connections 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.242+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/289465745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 0x7fa4081092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.242+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/289465745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa408102460 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.242+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/289465745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa408100e90 0x7fa4081012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- 172.21.15.57:0/289465745 >> 172.21.15.57:0/289465745 conn(0x7fa4080fc400 msgr2=0x7fa4080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- 172.21.15.57:0/289465745 shutdown_connections 2024-04-21T08:43:10.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- 172.21.15.57:0/289465745 wait complete. 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 Processor -- start 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- start start 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa408100e90 0x7fa408198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa4081987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 0x7fa40819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810db90 con 0x7fa408101fe0 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810da10 con 0x7fa408100e90 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40f2c8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa40810dd10 con 0x7fa4081029a0 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.243+0000 7fa40c863640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa4081987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa4081987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59130/0 (socket says 172.21.15.57:59130) 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 -- 172.21.15.57:0/2155944065 learned_addr learned my addr 172.21.15.57:0/2155944065 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40d064640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa408100e90 0x7fa408198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40d865640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 0x7fa40819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 -- 172.21.15.57:0/2155944065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 msgr2=0x7fa40819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 0x7fa40819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 -- 172.21.15.57:0/2155944065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa408100e90 msgr2=0x7fa408198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa408100e90 0x7fa408198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 -- 172.21.15.57:0/2155944065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40819fe80 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40d865640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 0x7fa40819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40c863640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa4081987a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc000c00 tx=0x7fa3fc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc043020 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3fc0044c0 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4081a0110 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc003c90 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.244+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4081a0660 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.245+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa408108310 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.246+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa3fc046020 con 0x7fa408101fe0 2024-04-21T08:43:10.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.247+0000 7fa4067fc640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.247+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa3fc0f2ce0 con 0x7fa408101fe0 2024-04-21T08:43:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.247+0000 7fa40d064640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.248+0000 7fa40d064640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa400009bd0 tx=0x7fa400005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:10.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.251+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3fc04d020 con 0x7fa408101fe0 2024-04-21T08:43:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.436+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa40802d830 con 0x7fa408101fe0 2024-04-21T08:43:10.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.437+0000 7fa4067fc640 1 -- 172.21.15.57:0/2155944065 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fa3fc0bf480 con 0x7fa408101fe0 2024-04-21T08:43:10.436 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:43:10.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f00a3bb0 msgr2=0x7fa3f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa400009bd0 tx=0x7fa400005f90 comp rx=0 tx=0).stop 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 msgr2=0x7fa4081987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa4081987a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc000c00 tx=0x7fa3fc0040c0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 shutdown_connections 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f00a3bb0 0x7fa3f00a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4081029a0 0x7fa40819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa408101fe0 0x7fa4081987a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 --2- 172.21.15.57:0/2155944065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa408100e90 0x7fa408198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 >> 172.21.15.57:0/2155944065 conn(0x7fa4080fc400 msgr2=0x7fa4081054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 shutdown_connections 2024-04-21T08:43:10.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.439+0000 7fa40f2c8640 1 -- 172.21.15.57:0/2155944065 wait complete. 2024-04-21T08:43:10.449 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 -- 172.21.15.57:0/1161190571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c102ad0 msgr2=0x7f8c6c104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 --2- 172.21.15.57:0/1161190571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c102ad0 0x7f8c6c104ec0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009900 tx=0x7f8c6802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 -- 172.21.15.57:0/1161190571 shutdown_connections 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 --2- 172.21.15.57:0/1161190571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 0x7f8c6c10a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 --2- 172.21.15.57:0/1161190571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c6c105400 0x7f8c6c1077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 --2- 172.21.15.57:0/1161190571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c102ad0 0x7f8c6c104ec0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 -- 172.21.15.57:0/1161190571 >> 172.21.15.57:0/1161190571 conn(0x7f8c6c0fc440 msgr2=0x7f8c6c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 -- 172.21.15.57:0/1161190571 shutdown_connections 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.555+0000 7f8c73293640 1 -- 172.21.15.57:0/1161190571 wait complete. 2024-04-21T08:43:10.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 Processor -- start 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 -- start start 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c6c102ad0 0x7f8c6c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 0x7f8c6c1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 0x7f8c6c19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10dbd0 con 0x7f8c6c105400 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10da50 con 0x7f8c6c102ad0 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c73293640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10dd50 con 0x7f8c6c107d30 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c7082e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 0x7f8c6c1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c7082e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 0x7f8c6c1988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59152/0 (socket says 172.21.15.57:59152) 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c7082e640 1 -- 172.21.15.57:0/347866628 learned_addr learned my addr 172.21.15.57:0/347866628 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c7102f640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c6c102ad0 0x7f8c6c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.556+0000 7f8c71830640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 0x7f8c6c19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c7082e640 1 -- 172.21.15.57:0/347866628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 msgr2=0x7f8c6c19f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c7082e640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 0x7f8c6c19f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c7082e640 1 -- 172.21.15.57:0/347866628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c6c102ad0 msgr2=0x7f8c6c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c7082e640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c6c102ad0 0x7f8c6c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c7082e640 1 -- 172.21.15.57:0/347866628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6c19ffb0 con 0x7f8c6c105400 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c71830640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 0x7f8c6c19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c7082e640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 0x7f8c6c1988d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00b450 tx=0x7f8c5c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c015020 con 0x7f8c6c105400 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c5c011d40 con 0x7f8c6c105400 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00f340 con 0x7f8c6c105400 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6c1a02a0 con 0x7f8c6c105400 2024-04-21T08:43:10.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.557+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6c1a0870 con 0x7f8c6c105400 2024-04-21T08:43:10.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.558+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c6c0fdf30 con 0x7f8c6c105400 2024-04-21T08:43:10.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.559+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8c5c018020 con 0x7f8c6c105400 2024-04-21T08:43:10.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.560+0000 7f8c627fc640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8c580a3bb0 0x7f8c580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.560+0000 7f8c7102f640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8c580a3bb0 0x7f8c580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.560+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8c5c0d01e0 con 0x7f8c6c105400 2024-04-21T08:43:10.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.560+0000 7f8c7102f640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8c580a3bb0 0x7f8c580a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8c68000c00 tx=0x7f8c6802c800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:10.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.564+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8c5c01f020 con 0x7f8c6c105400 2024-04-21T08:43:10.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.752+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8c6c062b50 con 0x7f8c6c105400 2024-04-21T08:43:10.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.753+0000 7f8c627fc640 1 -- 172.21.15.57:0/347866628 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f8c5c00d040 con 0x7f8c6c105400 2024-04-21T08:43:10.752 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:43:10.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8c580a3bb0 msgr2=0x7f8c580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8c580a3bb0 0x7f8c580a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8c68000c00 tx=0x7f8c6802c800 comp rx=0 tx=0).stop 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 msgr2=0x7f8c6c1988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 0x7f8c6c1988d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00b450 tx=0x7f8c5c00b910 comp rx=0 tx=0).stop 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 shutdown_connections 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8c580a3bb0 0x7f8c580a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c6c107d30 0x7f8c6c19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8c6c105400 0x7f8c6c1988d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 --2- 172.21.15.57:0/347866628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c6c102ad0 0x7f8c6c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 >> 172.21.15.57:0/347866628 conn(0x7f8c6c0fc440 msgr2=0x7f8c6c104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 shutdown_connections 2024-04-21T08:43:10.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.755+0000 7f8c73293640 1 -- 172.21.15.57:0/347866628 wait complete. 2024-04-21T08:43:10.765 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 -- 172.21.15.57:0/3703747285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 msgr2=0x7f9ea0103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/3703747285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0103620 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009900 tx=0x7f9e9002bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 -- 172.21.15.57:0/3703747285 shutdown_connections 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/3703747285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 0x7f9ea010a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/3703747285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0103620 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/3703747285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 0x7f9ea01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 -- 172.21.15.57:0/3703747285 >> 172.21.15.57:0/3703747285 conn(0x7f9ea00fd520 msgr2=0x7f9ea00ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 -- 172.21.15.57:0/3703747285 shutdown_connections 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.870+0000 7f9ea72a5640 1 -- 172.21.15.57:0/3703747285 wait complete. 2024-04-21T08:43:10.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 Processor -- start 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 -- start start 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 0x7f9ea0075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 0x7f9ea0074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea010db40 con 0x7f9ea01031a0 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea010d9c0 con 0x7f9ea0101fb0 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea72a5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea010dcc0 con 0x7f9ea0103b60 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0074340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59166/0 (socket says 172.21.15.57:59166) 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 -- 172.21.15.57:0/237863066 learned_addr learned my addr 172.21.15.57:0/237863066 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea5041640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 0x7f9ea0075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea5842640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 0x7f9ea0074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 -- 172.21.15.57:0/237863066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 msgr2=0x7f9ea0074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 0x7f9ea0074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 -- 172.21.15.57:0/237863066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 msgr2=0x7f9ea0075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 0x7f9ea0075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 -- 172.21.15.57:0/237863066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea0075110 con 0x7f9ea01031a0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea5041640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 0x7f9ea0075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea5842640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 0x7f9ea0074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.871+0000 7f9ea4840640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0074340 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9e90000c00 tx=0x7f9e900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.872+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90043020 con 0x7f9ea01031a0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.872+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e900054c0 con 0x7f9ea01031a0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.872+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea01a22e0 con 0x7f9ea01031a0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.872+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90003db0 con 0x7f9ea01031a0 2024-04-21T08:43:10.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.872+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea01a2880 con 0x7f9ea01031a0 2024-04-21T08:43:10.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.873+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea01092d0 con 0x7f9ea01031a0 2024-04-21T08:43:10.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.874+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9e90046020 con 0x7f9ea01031a0 2024-04-21T08:43:10.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.874+0000 7f9e967fc640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9e8c0a3c30 0x7f9e8c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:10.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.874+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9e900f2ae0 con 0x7f9ea01031a0 2024-04-21T08:43:10.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.874+0000 7f9ea5041640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9e8c0a3c30 0x7f9e8c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:10.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.875+0000 7f9ea5041640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9e8c0a3c30 0x7f9e8c0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0058c0 tx=0x7f9e9c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:10.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:10.879+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e9004d020 con 0x7f9ea01031a0 2024-04-21T08:43:11.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.064+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9ea0062b50 con 0x7f9ea01031a0 2024-04-21T08:43:11.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.064+0000 7f9e967fc640 1 -- 172.21.15.57:0/237863066 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f9e900bf280 con 0x7f9ea01031a0 2024-04-21T08:43:11.064 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:43:11.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.066+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9e8c0a3c30 msgr2=0x7f9e8c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.066+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9e8c0a3c30 0x7f9e8c0a60f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0058c0 tx=0x7f9e9c005850 comp rx=0 tx=0).stop 2024-04-21T08:43:11.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 msgr2=0x7f9ea0074340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0074340 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9e90000c00 tx=0x7f9e900042f0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 shutdown_connections 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9e8c0a3c30 0x7f9e8c0a60f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ea0103b60 0x7f9ea0074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea01031a0 0x7f9ea0074340 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 --2- 172.21.15.57:0/237863066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ea0101fb0 0x7f9ea0075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 >> 172.21.15.57:0/237863066 conn(0x7f9ea00fd520 msgr2=0x7f9ea01064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 shutdown_connections 2024-04-21T08:43:11.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.067+0000 7f9ea72a5640 1 -- 172.21.15.57:0/237863066 wait complete. 2024-04-21T08:43:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:10 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1231599450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:10 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2980193782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:43:11.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:10 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2155944065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:43:11.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:10 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1231599450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:11.075 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:10 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2980193782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:43:11.076 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:10 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2155944065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:43:11.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:43:11.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:10 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1231599450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:11.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:10 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2980193782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:43:11.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:10 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2155944065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:43:11.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 -- 172.21.15.57:0/1993444229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064105360 msgr2=0x7f8064107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 --2- 172.21.15.57:0/1993444229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064105360 0x7f8064107750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8058009900 tx=0x7f805802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 -- 172.21.15.57:0/1993444229 shutdown_connections 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 --2- 172.21.15.57:0/1993444229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8064107c90 0x7f806410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 --2- 172.21.15.57:0/1993444229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064105360 0x7f8064107750 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 --2- 172.21.15.57:0/1993444229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064102a30 0x7f8064104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 -- 172.21.15.57:0/1993444229 >> 172.21.15.57:0/1993444229 conn(0x7f80640fc440 msgr2=0x7f80640fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 -- 172.21.15.57:0/1993444229 shutdown_connections 2024-04-21T08:43:11.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.180+0000 7f806b8f4640 1 -- 172.21.15.57:0/1993444229 wait complete. 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 Processor -- start 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 -- start start 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 0x7f8064198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8064105360 0x7f8064198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064107c90 0x7f806419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f806410daf0 con 0x7f8064102a30 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f806410d970 con 0x7f8064107c90 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f806b8f4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f806410dc70 con 0x7f8064105360 2024-04-21T08:43:11.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 0x7f8064198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 0x7f8064198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59176/0 (socket says 172.21.15.57:59176) 2024-04-21T08:43:11.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 -- 172.21.15.57:0/3430164394 learned_addr learned my addr 172.21.15.57:0/3430164394 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:11.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8068e8f640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8064105360 0x7f8064198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069e91640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064107c90 0x7f806419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 -- 172.21.15.57:0/3430164394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8064105360 msgr2=0x7f8064198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8064105360 0x7f8064198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 -- 172.21.15.57:0/3430164394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064107c90 msgr2=0x7f806419f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064107c90 0x7f806419f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 -- 172.21.15.57:0/3430164394 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806419fe60 con 0x7f8064102a30 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069e91640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064107c90 0x7f806419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.181+0000 7f8069690640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 0x7f8064198240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f805400eab0 tx=0x7f805400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.182+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805400ce40 con 0x7f8064102a30 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.182+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8054011920 con 0x7f8064102a30 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.182+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054010a30 con 0x7f8064102a30 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.182+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80641a0150 con 0x7f8064102a30 2024-04-21T08:43:11.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.182+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80641a0670 con 0x7f8064102a30 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.183+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80640fdf30 con 0x7f8064102a30 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.184+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8054010bd0 con 0x7f8064102a30 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.184+0000 7f80627fc640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f804c0a3bb0 0x7f804c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.185+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f80540d0c80 con 0x7f8064102a30 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.185+0000 7f8068e8f640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f804c0a3bb0 0x7f804c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.185+0000 7f8068e8f640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f804c0a3bb0 0x7f804c0a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8058005ab0 tx=0x7f8058005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:11.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.189+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8054016030 con 0x7f8064102a30 2024-04-21T08:43:11.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.387+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f806402d5c0 con 0x7f8064102a30 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.387+0000 7f80627fc640 1 -- 172.21.15.57:0/3430164394 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f805409d420 con 0x7f8064102a30 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3363889148 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/316357528 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/2928520021 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/2928520021 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2818769380 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/898712319 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2910544903 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:11.387 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/3383203302 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3448624546 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:11.388 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/3383203302 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:11.388 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f804c0a3bb0 msgr2=0x7f804c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f804c0a3bb0 0x7f804c0a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8058005ab0 tx=0x7f8058005fd0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 msgr2=0x7f8064198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 0x7f8064198240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f805400eab0 tx=0x7f805400ef70 comp rx=0 tx=0).stop 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 shutdown_connections 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f804c0a3bb0 0x7f804c0a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8064107c90 0x7f806419f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8064105360 0x7f8064198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 --2- 172.21.15.57:0/3430164394 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8064102a30 0x7f8064198240 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 >> 172.21.15.57:0/3430164394 conn(0x7f80640fc440 msgr2=0x7f80641048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:11.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 shutdown_connections 2024-04-21T08:43:11.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.389+0000 7f806b8f4640 1 -- 172.21.15.57:0/3430164394 wait complete. 2024-04-21T08:43:11.399 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:43:11.399 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:43:11.416 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:11.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 -- 172.21.15.57:0/1728961124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8102ae0 msgr2=0x7f7ef8104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/1728961124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8102ae0 0x7f7ef8104ed0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009900 tx=0x7f7ef402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:11.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 -- 172.21.15.57:0/1728961124 shutdown_connections 2024-04-21T08:43:11.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/1728961124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef8107d40 0x7f7ef810a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/1728961124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef8105410 0x7f7ef8107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/1728961124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8102ae0 0x7f7ef8104ed0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 -- 172.21.15.57:0/1728961124 >> 172.21.15.57:0/1728961124 conn(0x7f7ef80fc470 msgr2=0x7f7ef80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 -- 172.21.15.57:0/1728961124 shutdown_connections 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 -- 172.21.15.57:0/1728961124 wait complete. 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 Processor -- start 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.559+0000 7f7eff7d0640 1 -- start start 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eff7d0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef8102ae0 0x7f7ef8102450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eff7d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 0x7f7ef8100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eff7d0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef8107d40 0x7f7ef8100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eff7d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef810db20 con 0x7f7ef8105410 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eff7d0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef810d9a0 con 0x7f7ef8102ae0 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eff7d0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef810dca0 con 0x7f7ef8107d40 2024-04-21T08:43:11.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 0x7f7ef8100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 0x7f7ef8100a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59188/0 (socket says 172.21.15.57:59188) 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 -- 172.21.15.57:0/3450262283 learned_addr learned my addr 172.21.15.57:0/3450262283 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efd56c640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef8102ae0 0x7f7ef8102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efdd6d640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef8107d40 0x7f7ef8100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 -- 172.21.15.57:0/3450262283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef8107d40 msgr2=0x7f7ef8100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef8107d40 0x7f7ef8100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 -- 172.21.15.57:0/3450262283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef8102ae0 msgr2=0x7f7ef8102450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef8102ae0 0x7f7ef8102450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 -- 172.21.15.57:0/3450262283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef8101860 con 0x7f7ef8105410 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7efcd6b640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 0x7f7ef8100a90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800d770 tx=0x7f7ee800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8017070 con 0x7f7ef8105410 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.560+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ee8009c30 con 0x7f7ef8105410 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.561+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef81a22f0 con 0x7f7ef8105410 2024-04-21T08:43:11.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.561+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee800f620 con 0x7f7ef8105410 2024-04-21T08:43:11.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.561+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef81a2890 con 0x7f7ef8105410 2024-04-21T08:43:11.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.562+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef80fdf60 con 0x7f7ef8105410 2024-04-21T08:43:11.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.563+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7ee801a070 con 0x7f7ef8105410 2024-04-21T08:43:11.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.563+0000 7f7eee7fc640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ee40a3c30 0x7f7ee40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.564+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7ee80cffe0 con 0x7f7ef8105410 2024-04-21T08:43:11.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.564+0000 7f7efd56c640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ee40a3c30 0x7f7ee40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.564+0000 7f7efd56c640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ee40a3c30 0x7f7ee40a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009230 tx=0x7f7ef4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:11.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.567+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ee8009de0 con 0x7f7ef8105410 2024-04-21T08:43:11.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.756+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ef802d810 con 0x7f7ef8105410 2024-04-21T08:43:11.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.758+0000 7f7eee7fc640 1 -- 172.21.15.57:0/3450262283 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7f7ee809c1b0 con 0x7f7ef8105410 2024-04-21T08:43:11.757 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:11.758 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:11.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ee40a3c30 msgr2=0x7f7ee40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ee40a3c30 0x7f7ee40a60f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009230 tx=0x7f7ef4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 msgr2=0x7f7ef8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 0x7f7ef8100a90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800d770 tx=0x7f7ee800dc30 comp rx=0 tx=0).stop 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 shutdown_connections 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7ee40a3c30 0x7f7ee40a60f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ef8107d40 0x7f7ef8100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef8105410 0x7f7ef8100a90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 --2- 172.21.15.57:0/3450262283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ef8102ae0 0x7f7ef8102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 >> 172.21.15.57:0/3450262283 conn(0x7f7ef80fc470 msgr2=0x7f7ef8103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 shutdown_connections 2024-04-21T08:43:11.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.760+0000 7f7eff7d0640 1 -- 172.21.15.57:0/3450262283 wait complete. 2024-04-21T08:43:11.770 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:43:11.771 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:43:11.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 -- 172.21.15.57:0/3696298198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c101fe0 msgr2=0x7fb48c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 --2- 172.21.15.57:0/3696298198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c101fe0 0x7fb48c102460 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb480009900 tx=0x7fb48002bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:11.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 -- 172.21.15.57:0/3696298198 shutdown_connections 2024-04-21T08:43:11.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 --2- 172.21.15.57:0/3696298198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb48c1029a0 0x7fb48c1092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 --2- 172.21.15.57:0/3696298198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c101fe0 0x7fb48c102460 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 --2- 172.21.15.57:0/3696298198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb48c100e90 0x7fb48c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.875+0000 7fb493b54640 1 -- 172.21.15.57:0/3696298198 >> 172.21.15.57:0/3696298198 conn(0x7fb48c0fc400 msgr2=0x7fb48c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 -- 172.21.15.57:0/3696298198 shutdown_connections 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 -- 172.21.15.57:0/3696298198 wait complete. 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 Processor -- start 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 -- start start 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb48c100e90 0x7fb48c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb48c101fe0 0x7fb48c198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 0x7fb48c19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb48c10db30 con 0x7fb48c1029a0 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb48c10d9b0 con 0x7fb48c101fe0 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb493b54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb48c10dcb0 con 0x7fb48c100e90 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb4920f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 0x7fb48c19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb4920f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 0x7fb48c19f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59214/0 (socket says 172.21.15.57:59214) 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.876+0000 7fb4920f1640 1 -- 172.21.15.57:0/1062211748 learned_addr learned my addr 172.21.15.57:0/1062211748 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4910ef640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb48c101fe0 0x7fb48c198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4918f0640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb48c100e90 0x7fb48c1981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4920f1640 1 -- 172.21.15.57:0/1062211748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb48c100e90 msgr2=0x7fb48c1981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4920f1640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb48c100e90 0x7fb48c1981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4920f1640 1 -- 172.21.15.57:0/1062211748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb48c101fe0 msgr2=0x7fb48c198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4920f1640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb48c101fe0 0x7fb48c198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4920f1640 1 -- 172.21.15.57:0/1062211748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb48c19fe10 con 0x7fb48c1029a0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb4920f1640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 0x7fb48c19f7a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb48400c970 tx=0x7fb48400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484007bb0 con 0x7fb48c1029a0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb484007d50 con 0x7fb48c1029a0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb48c1a0100 con 0x7fb48c1029a0 2024-04-21T08:43:11.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484018770 con 0x7fb48c1029a0 2024-04-21T08:43:11.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.877+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb48c1a0620 con 0x7fb48c1029a0 2024-04-21T08:43:11.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.878+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb48c108270 con 0x7fb48c1029a0 2024-04-21T08:43:11.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.879+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb48400f990 con 0x7fb48c1029a0 2024-04-21T08:43:11.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.880+0000 7fb48affd640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb4740a3c30 0x7fb4740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:11.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.880+0000 7fb4918f0640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb4740a3c30 0x7fb4740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:11.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.880+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb4840d06d0 con 0x7fb48c1029a0 2024-04-21T08:43:11.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.881+0000 7fb4918f0640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb4740a3c30 0x7fb4740a60f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb47c005fd0 tx=0x7fb47c009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:11.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:11.885+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb484014030 con 0x7fb48c1029a0 2024-04-21T08:43:12.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:11 smithi129 ceph-mon[31272]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:43:12.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:11 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/347866628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:43:12.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:11 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/237863066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:43:12.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:11 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3430164394' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:43:12.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.074+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb48c062b50 con 0x7fb48c1029a0 2024-04-21T08:43:12.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.074+0000 7fb48affd640 1 -- 172.21.15.57:0/1062211748 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fb48409ce70 con 0x7fb48c1029a0 2024-04-21T08:43:12.075 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:43:12.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb4740a3c30 msgr2=0x7fb4740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb4740a3c30 0x7fb4740a60f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb47c005fd0 tx=0x7fb47c009870 comp rx=0 tx=0).stop 2024-04-21T08:43:12.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 msgr2=0x7fb48c19f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 0x7fb48c19f7a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb48400c970 tx=0x7fb48400ce30 comp rx=0 tx=0).stop 2024-04-21T08:43:12.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 shutdown_connections 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb4740a3c30 0x7fb4740a60f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb48c1029a0 0x7fb48c19f7a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb48c101fe0 0x7fb48c198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 --2- 172.21.15.57:0/1062211748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb48c100e90 0x7fb48c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 >> 172.21.15.57:0/1062211748 conn(0x7fb48c0fc400 msgr2=0x7fb48c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 shutdown_connections 2024-04-21T08:43:12.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.076+0000 7fb493b54640 1 -- 172.21.15.57:0/1062211748 wait complete. 2024-04-21T08:43:12.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:11 smithi057 ceph-mon[27968]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:43:12.085 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:11 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/347866628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:43:12.086 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:11 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/237863066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:43:12.086 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:11 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3430164394' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:43:12.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:43:12.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:11 smithi136 ceph-mon[31791]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:43:12.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:11 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/347866628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:43:12.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:11 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/237863066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:43:12.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:11 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3430164394' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:43:12.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.193+0000 7fd8c84a5640 1 -- 172.21.15.57:0/631788110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0102a30 msgr2=0x7fd8c0104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.193+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/631788110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0102a30 0x7fd8c0104e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009900 tx=0x7fd8b802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:12.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- 172.21.15.57:0/631788110 shutdown_connections 2024-04-21T08:43:12.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/631788110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c0107c90 0x7fd8c010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/631788110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8c0105360 0x7fd8c0107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/631788110 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0102a30 0x7fd8c0104e20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- 172.21.15.57:0/631788110 >> 172.21.15.57:0/631788110 conn(0x7fd8c00fc440 msgr2=0x7fd8c00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- 172.21.15.57:0/631788110 shutdown_connections 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- 172.21.15.57:0/631788110 wait complete. 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 Processor -- start 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- start start 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8c0102a30 0x7fd8c0198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c0105360 0x7fd8c01987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 0x7fd8c019f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c010daf0 con 0x7fd8c0107c90 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.194+0000 7fd8c84a5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c010d970 con 0x7fd8c0105360 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c84a5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c010dc70 con 0x7fd8c0102a30 2024-04-21T08:43:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 0x7fd8c019f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 0x7fd8c019f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59240/0 (socket says 172.21.15.57:59240) 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 -- 172.21.15.57:0/1272753231 learned_addr learned my addr 172.21.15.57:0/1272753231 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c5a40640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c0105360 0x7fd8c01987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6241640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8c0102a30 0x7fd8c0198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 -- 172.21.15.57:0/1272753231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8c0102a30 msgr2=0x7fd8c0198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8c0102a30 0x7fd8c0198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 -- 172.21.15.57:0/1272753231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c0105360 msgr2=0x7fd8c01987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c0105360 0x7fd8c01987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 -- 172.21.15.57:0/1272753231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c019fe90 con 0x7fd8c0107c90 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c6a42640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 0x7fd8c019f820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009fd0 tx=0x7fd8b000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0017070 con 0x7fd8c0107c90 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b0003e20 con 0x7fd8c0107c90 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c01a0180 con 0x7fd8c0107c90 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0010460 con 0x7fd8c0107c90 2024-04-21T08:43:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.195+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c01a0680 con 0x7fd8c0107c90 2024-04-21T08:43:12.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.196+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c00fdf30 con 0x7fd8c0107c90 2024-04-21T08:43:12.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.198+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd8b0003970 con 0x7fd8c0107c90 2024-04-21T08:43:12.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.198+0000 7fd8bf7fe640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd8a80a3bb0 0x7fd8a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.198+0000 7fd8c6241640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd8a80a3bb0 0x7fd8a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.198+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd8b00d0370 con 0x7fd8c0107c90 2024-04-21T08:43:12.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.199+0000 7fd8c6241640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd8a80a3bb0 0x7fd8a80a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8000c00 tx=0x7fd8b8005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:12.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.202+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8b009cb10 con 0x7fd8c0107c90 2024-04-21T08:43:12.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.387+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd8c002d5c0 con 0x7fd8c0107c90 2024-04-21T08:43:12.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.387+0000 7fd8bf7fe640 1 -- 172.21.15.57:0/1272753231 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd8b009c410 con 0x7fd8c0107c90 2024-04-21T08:43:12.387 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd8a80a3bb0 msgr2=0x7fd8a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd8a80a3bb0 0x7fd8a80a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8000c00 tx=0x7fd8b8005be0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 msgr2=0x7fd8c019f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 0x7fd8c019f820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009fd0 tx=0x7fd8b000b260 comp rx=0 tx=0).stop 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 shutdown_connections 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd8a80a3bb0 0x7fd8a80a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8c0107c90 0x7fd8c019f820 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c0105360 0x7fd8c01987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 --2- 172.21.15.57:0/1272753231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8c0102a30 0x7fd8c0198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.389+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 >> 172.21.15.57:0/1272753231 conn(0x7fd8c00fc440 msgr2=0x7fd8c0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.390+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 shutdown_connections 2024-04-21T08:43:12.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.390+0000 7fd8c84a5640 1 -- 172.21.15.57:0/1272753231 wait complete. 2024-04-21T08:43:12.400 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:43:12.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 -- 172.21.15.57:0/10118419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 msgr2=0x7fb3a01092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/10118419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a01092c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb398009ca0 tx=0x7fb39802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 -- 172.21.15.57:0/10118419 shutdown_connections 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/10118419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a01092c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/10118419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3a0101fe0 0x7fb3a0102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/10118419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a0100e90 0x7fb3a01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 -- 172.21.15.57:0/10118419 >> 172.21.15.57:0/10118419 conn(0x7fb3a00fc400 msgr2=0x7fb3a00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 -- 172.21.15.57:0/10118419 shutdown_connections 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 -- 172.21.15.57:0/10118419 wait complete. 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 Processor -- start 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.505+0000 7fb3a5d85640 1 -- start start 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3a0100e90 0x7fb3a0069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a0101fe0 0x7fb3a019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a010c9d0 con 0x7fb3a01029a0 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a010c850 con 0x7fb3a0101fe0 2024-04-21T08:43:12.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a010cb50 con 0x7fb3a0100e90 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a019f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59260/0 (socket says 172.21.15.57:59260) 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 -- 172.21.15.57:0/3605340471 learned_addr learned my addr 172.21.15.57:0/3605340471 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39f7fe640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3a0100e90 0x7fb3a0069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 -- 172.21.15.57:0/3605340471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3a0100e90 msgr2=0x7fb3a0069c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39effd640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a0101fe0 0x7fb3a019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3a0100e90 0x7fb3a0069c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 -- 172.21.15.57:0/3605340471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a0101fe0 msgr2=0x7fb3a019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a0101fe0 0x7fb3a019cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 -- 172.21.15.57:0/3605340471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a019fdc0 con 0x7fb3a01029a0 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39ffff640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a019f500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb39802c700 tx=0x7fb398035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:12.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398039040 con 0x7fb3a01029a0 2024-04-21T08:43:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb398044070 con 0x7fb3a01029a0 2024-04-21T08:43:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.506+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a01a0050 con 0x7fb3a01029a0 2024-04-21T08:43:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.507+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398004440 con 0x7fb3a01029a0 2024-04-21T08:43:12.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.507+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a01a0590 con 0x7fb3a01029a0 2024-04-21T08:43:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.507+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a0108270 con 0x7fb3a01029a0 2024-04-21T08:43:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.509+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb3980045e0 con 0x7fb3a01029a0 2024-04-21T08:43:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.509+0000 7fb39cff9640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb3880a3c30 0x7fb3880a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.509+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb3980f2cb0 con 0x7fb3a01029a0 2024-04-21T08:43:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.509+0000 7fb39f7fe640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb3880a3c30 0x7fb3880a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.510+0000 7fb39f7fe640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb3880a3c30 0x7fb3880a60f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3900058c0 tx=0x7fb39000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:12.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.514+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3980bf450 con 0x7fb3a01029a0 2024-04-21T08:43:12.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.703+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb3a0062b50 con 0x7fb3a01029a0 2024-04-21T08:43:12.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.704+0000 7fb39cff9640 1 -- 172.21.15.57:0/3605340471 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fb3980bed50 con 0x7fb3a01029a0 2024-04-21T08:43:12.703 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb3880a3c30 msgr2=0x7fb3880a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb3880a3c30 0x7fb3880a60f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3900058c0 tx=0x7fb39000a4c0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 msgr2=0x7fb3a019f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a019f500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb39802c700 tx=0x7fb398035d60 comp rx=0 tx=0).stop 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 shutdown_connections 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb3880a3c30 0x7fb3880a60f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3a01029a0 0x7fb3a019f500 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3a0101fe0 0x7fb3a019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 --2- 172.21.15.57:0/3605340471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3a0100e90 0x7fb3a0069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.705+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 >> 172.21.15.57:0/3605340471 conn(0x7fb3a00fc400 msgr2=0x7fb3a0105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.706+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 shutdown_connections 2024-04-21T08:43:12.706 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.706+0000 7fb3a5d85640 1 -- 172.21.15.57:0/3605340471 wait complete. 2024-04-21T08:43:12.716 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 -- 172.21.15.57:0/2280427190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c103b60 msgr2=0x7f902c10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 --2- 172.21.15.57:0/2280427190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c103b60 0x7f902c10a3f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f901c009ca0 tx=0x7f901c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 -- 172.21.15.57:0/2280427190 shutdown_connections 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 --2- 172.21.15.57:0/2280427190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c103b60 0x7f902c10a3f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 --2- 172.21.15.57:0/2280427190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f902c1031a0 0x7f902c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 --2- 172.21.15.57:0/2280427190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f902c101fb0 0x7f902c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 -- 172.21.15.57:0/2280427190 >> 172.21.15.57:0/2280427190 conn(0x7f902c0fd520 msgr2=0x7f902c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:12.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 -- 172.21.15.57:0/2280427190 shutdown_connections 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.821+0000 7f9030d96640 1 -- 172.21.15.57:0/2280427190 wait complete. 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 Processor -- start 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 -- start start 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 0x7f902c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f902c1031a0 0x7f902c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f902c103b60 0x7f902c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10db70 con 0x7f902c101fb0 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10d9f0 con 0x7f902c1031a0 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f9030d96640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f902c10dcf0 con 0x7f902c103b60 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f902a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 0x7f902c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f902a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 0x7f902c075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43674/0 (socket says 172.21.15.57:43674) 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.822+0000 7f902a59c640 1 -- 172.21.15.57:0/873006362 learned_addr learned my addr 172.21.15.57:0/873006362 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f902a59c640 1 -- 172.21.15.57:0/873006362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f902c103b60 msgr2=0x7f902c074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f902a59c640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f902c103b60 0x7f902c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f902a59c640 1 -- 172.21.15.57:0/873006362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f902c1031a0 msgr2=0x7f902c074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f902a59c640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f902c1031a0 0x7f902c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f902a59c640 1 -- 172.21.15.57:0/873006362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c075110 con 0x7f902c101fb0 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f902a59c640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 0x7f902c075d00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f902400b450 tx=0x7f902400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024015020 con 0x7f902c101fb0 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9024011d40 con 0x7f902c101fb0 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902c1a2350 con 0x7f902c101fb0 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902400f340 con 0x7f902c101fb0 2024-04-21T08:43:12.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.823+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902c1a28a0 con 0x7f902c101fb0 2024-04-21T08:43:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.824+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c1092d0 con 0x7f902c101fb0 2024-04-21T08:43:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.825+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9024018020 con 0x7f902c101fb0 2024-04-21T08:43:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.826+0000 7f90237fe640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f90180a3bb0 0x7f90180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.826+0000 7f9029d9b640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f90180a3bb0 0x7f90180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:12.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.826+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f90240d01e0 con 0x7f902c101fb0 2024-04-21T08:43:12.827 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.827+0000 7f9029d9b640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f90180a3bb0 0x7f90180a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9014005fd0 tx=0x7f9014005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:12.830 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:12.831+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f902401f020 con 0x7f902c101fb0 2024-04-21T08:43:13.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.015+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f902c062b50 con 0x7f902c101fb0 2024-04-21T08:43:13.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.015+0000 7f90237fe640 1 -- 172.21.15.57:0/873006362 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f902400d040 con 0x7f902c101fb0 2024-04-21T08:43:13.015 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:43:13.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f90180a3bb0 msgr2=0x7f90180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f90180a3bb0 0x7f90180a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9014005fd0 tx=0x7f9014005850 comp rx=0 tx=0).stop 2024-04-21T08:43:13.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 msgr2=0x7f902c075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 0x7f902c075d00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f902400b450 tx=0x7f902400b910 comp rx=0 tx=0).stop 2024-04-21T08:43:13.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 shutdown_connections 2024-04-21T08:43:13.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f90180a3bb0 0x7f90180a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f902c103b60 0x7f902c074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.017+0000 7f9030d96640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f902c1031a0 0x7f902c074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.018+0000 7f9030d96640 1 --2- 172.21.15.57:0/873006362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c101fb0 0x7f902c075d00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.018+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 >> 172.21.15.57:0/873006362 conn(0x7f902c0fd520 msgr2=0x7f902c1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:13.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.018+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 shutdown_connections 2024-04-21T08:43:13.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.018+0000 7f9030d96640 1 -- 172.21.15.57:0/873006362 wait complete. 2024-04-21T08:43:13.026 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3450262283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:13.027 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1062211748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.027 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1272753231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.027 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3605340471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.028 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:43:13.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:12 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3450262283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:13.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:12 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1062211748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:12 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1272753231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:12 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3605340471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 -- 172.21.15.57:0/2077163113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 msgr2=0x7fd7941092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 --2- 172.21.15.57:0/2077163113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd7941092c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd784009ca0 tx=0x7fd78402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 -- 172.21.15.57:0/2077163113 shutdown_connections 2024-04-21T08:43:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 --2- 172.21.15.57:0/2077163113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd7941092c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 --2- 172.21.15.57:0/2077163113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd794101fe0 0x7fd794102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 --2- 172.21.15.57:0/2077163113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd794100e90 0x7fd7941012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 -- 172.21.15.57:0/2077163113 >> 172.21.15.57:0/2077163113 conn(0x7fd7940fc400 msgr2=0x7fd7940fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 -- 172.21.15.57:0/2077163113 shutdown_connections 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.131+0000 7fd7991fd640 1 -- 172.21.15.57:0/2077163113 wait complete. 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 Processor -- start 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 -- start start 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd794100e90 0x7fd7941970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd794101fe0 0x7fd794197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd79419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd79410ca40 con 0x7fd7941029a0 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd79410c8c0 con 0x7fd794101fe0 2024-04-21T08:43:13.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd7991fd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd79410cbc0 con 0x7fd794100e90 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd79359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd79419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd79359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd79419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43692/0 (socket says 172.21.15.57:43692) 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.132+0000 7fd79359e640 1 -- 172.21.15.57:0/3589580557 learned_addr learned my addr 172.21.15.57:0/3589580557 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79259c640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd794101fe0 0x7fd794197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79359e640 1 -- 172.21.15.57:0/3589580557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd794100e90 msgr2=0x7fd7941970f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79359e640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd794100e90 0x7fd7941970f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79359e640 1 -- 172.21.15.57:0/3589580557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd794101fe0 msgr2=0x7fd794197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79359e640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd794101fe0 0x7fd794197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79359e640 1 -- 172.21.15.57:0/3589580557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79419ed10 con 0x7fd7941029a0 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd79359e640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd79419e6a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd784009dc0 tx=0x7fd784032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784034aa0 con 0x7fd7941029a0 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd784034c40 con 0x7fd7941029a0 2024-04-21T08:43:13.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7940690e0 con 0x7fd7941029a0 2024-04-21T08:43:13.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784003e60 con 0x7fd7941029a0 2024-04-21T08:43:13.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.133+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7940695d0 con 0x7fd7941029a0 2024-04-21T08:43:13.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.134+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd794068d70 con 0x7fd7941029a0 2024-04-21T08:43:13.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:12 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3450262283' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:13.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:12 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1062211748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:12 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1272753231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:12 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3605340471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:43:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.135+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd7840325c0 con 0x7fd7941029a0 2024-04-21T08:43:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.136+0000 7fd78bfff640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7800a3bb0 0x7fd7800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.136+0000 7fd792d9d640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7800a3bb0 0x7fd7800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.136+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd7840f4500 con 0x7fd7941029a0 2024-04-21T08:43:13.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.136+0000 7fd792d9d640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7800a3bb0 0x7fd7800a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0096b0 tx=0x7fd78c006c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:13.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.140+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd784042380 con 0x7fd7941029a0 2024-04-21T08:43:13.337 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.338+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd79419efa0 con 0x7fd7941029a0 2024-04-21T08:43:13.338 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.339+0000 7fd78bfff640 1 -- 172.21.15.57:0/3589580557 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7fd7840c0c40 con 0x7fd7941029a0 2024-04-21T08:43:13.338 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3363889148 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:13.338 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/316357528 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:13.338 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/2928520021 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/2928520021 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2818769380 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/898712319 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2910544903 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/3383203302 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3448624546 2024-04-22T08:35:44.517439+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/3383203302 2024-04-22T08:35:17.446046+0000 2024-04-21T08:43:13.339 INFO:teuthology.orchestra.run.smithi057.stderr:listed 10 entries 2024-04-21T08:43:13.340 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7800a3bb0 msgr2=0x7fd7800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.340 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7800a3bb0 0x7fd7800a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0096b0 tx=0x7fd78c006c20 comp rx=0 tx=0).stop 2024-04-21T08:43:13.340 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 msgr2=0x7fd79419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:13.340 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd79419e6a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd784009dc0 tx=0x7fd784032f90 comp rx=0 tx=0).stop 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 shutdown_connections 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7800a3bb0 0x7fd7800a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7941029a0 0x7fd79419e6a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd794101fe0 0x7fd794197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 --2- 172.21.15.57:0/3589580557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd794100e90 0x7fd7941970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 >> 172.21.15.57:0/3589580557 conn(0x7fd7940fc400 msgr2=0x7fd794105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 shutdown_connections 2024-04-21T08:43:13.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:13.341+0000 7fd7991fd640 1 -- 172.21.15.57:0/3589580557 wait complete. 2024-04-21T08:43:13.351 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi057.front.sepia.ceph.com... 2024-04-21T08:43:13.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:13.351 DEBUG:teuthology.orchestra.run.smithi057:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-21T08:43:13.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi057.stderr:ceph-fuse[72690]: fuse finished with error 0 and tester_r 0 2024-04-21T08:43:13.376 INFO:teuthology.orchestra.run:waiting for 300 2024-04-21T08:43:14.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:13 smithi129 ceph-mon[31272]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T08:43:14.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/873006362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:43:14.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3589580557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:43:14.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:13 smithi057 ceph-mon[27968]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T08:43:14.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/873006362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:43:14.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3589580557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:43:14.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:13 smithi136 ceph-mon[31791]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-04-21T08:43:14.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/873006362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:43:14.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3589580557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:43:16.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:15 smithi129 ceph-mon[31272]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 257 B/s wr, 0 op/s 2024-04-21T08:43:16.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:15 smithi057 ceph-mon[27968]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 257 B/s wr, 0 op/s 2024-04-21T08:43:16.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:15 smithi136 ceph-mon[31791]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 257 B/s wr, 0 op/s 2024-04-21T08:43:18.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:17 smithi129 ceph-mon[31272]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:18.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:17 smithi057 ceph-mon[27968]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:18.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:17 smithi136 ceph-mon[31791]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:19.478 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:43:19.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:43:19.478 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-21T08:43:19.502 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:19.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.657+0000 7f93d9187640 1 -- 172.21.15.57:0/1307712551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4102a40 msgr2=0x7f93d4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 --2- 172.21.15.57:0/1307712551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4102a40 0x7f93d4104e30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009900 tx=0x7f93c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 -- 172.21.15.57:0/1307712551 shutdown_connections 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 --2- 172.21.15.57:0/1307712551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d4107ca0 0x7f93d410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 --2- 172.21.15.57:0/1307712551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d4105370 0x7f93d4107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 --2- 172.21.15.57:0/1307712551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4102a40 0x7f93d4104e30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 -- 172.21.15.57:0/1307712551 >> 172.21.15.57:0/1307712551 conn(0x7f93d40fc470 msgr2=0x7f93d40fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 -- 172.21.15.57:0/1307712551 shutdown_connections 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 -- 172.21.15.57:0/1307712551 wait complete. 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.658+0000 7f93d9187640 1 Processor -- start 2024-04-21T08:43:19.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 -- start start 2024-04-21T08:43:19.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d4102a40 0x7f93d41023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 0x7f93d41009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d4107ca0 0x7f93d4100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f93d410db00 con 0x7f93d4105370 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93d410d980 con 0x7f93d4107ca0 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d9187640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f93d410dc80 con 0x7f93d4102a40 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 0x7f93d41009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 0x7f93d41009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43710/0 (socket says 172.21.15.57:43710) 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 -- 172.21.15.57:0/268833349 learned_addr learned my addr 172.21.15.57:0/268833349 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d359e640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d4107ca0 0x7f93d4100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d2d9d640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d4102a40 0x7f93d41023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 -- 172.21.15.57:0/268833349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d4102a40 msgr2=0x7f93d41023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d4102a40 0x7f93d41023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 -- 172.21.15.57:0/268833349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d4107ca0 msgr2=0x7f93d4100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d4107ca0 0x7f93d4100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 -- 172.21.15.57:0/268833349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d41017c0 con 0x7f93d4105370 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.659+0000 7f93d259c640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 0x7f93d41009f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e9f0 tx=0x7f93c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.660+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c800ccc0 con 0x7f93d4105370 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.660+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d41a2300 con 0x7f93d4105370 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.660+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93c80054d0 con 0x7f93d4105370 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.660+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8010430 con 0x7f93d4105370 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.660+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d41a2800 con 0x7f93d4105370 2024-04-21T08:43:19.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.661+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d40fdf60 con 0x7f93d4105370 2024-04-21T08:43:19.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.662+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f93c80046c0 con 0x7f93d4105370 2024-04-21T08:43:19.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.663+0000 7f93bbfff640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93bc0a3bb0 0x7f93bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.663+0000 7f93d2d9d640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93bc0a3bb0 0x7f93bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.663+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93c80d0a10 con 0x7f93d4105370 2024-04-21T08:43:19.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.663+0000 7f93d2d9d640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93bc0a3bb0 0x7f93bc0a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009230 tx=0x7f93c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.667+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93c809d1b0 con 0x7f93d4105370 2024-04-21T08:43:19.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:19 smithi057 ceph-mon[27968]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:19.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.859+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93d4062b50 con 0x7f93d4105370 2024-04-21T08:43:19.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.861+0000 7f93bbfff640 1 -- 172.21.15.57:0/268833349 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7f93c809cab0 con 0x7f93d4105370 2024-04-21T08:43:19.860 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:19.861 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:19.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93bc0a3bb0 msgr2=0x7f93bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93bc0a3bb0 0x7f93bc0a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f93c4009230 tx=0x7f93c4005fd0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 msgr2=0x7f93d41009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 0x7f93d41009f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f93c800e9f0 tx=0x7f93c800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 shutdown_connections 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93bc0a3bb0 0x7f93bc0a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d4107ca0 0x7f93d4100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93d4105370 0x7f93d41009f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 --2- 172.21.15.57:0/268833349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d4102a40 0x7f93d41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 >> 172.21.15.57:0/268833349 conn(0x7f93d40fc470 msgr2=0x7f93d41033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 shutdown_connections 2024-04-21T08:43:19.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.863+0000 7f93d9187640 1 -- 172.21.15.57:0/268833349 wait complete. 2024-04-21T08:43:19.874 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 -- 172.21.15.57:0/2199078540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103180 msgr2=0x7fd71c103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 --2- 172.21.15.57:0/2199078540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103180 0x7fd71c103600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd704009900 tx=0x7fd70402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 -- 172.21.15.57:0/2199078540 shutdown_connections 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 --2- 172.21.15.57:0/2199078540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c103b40 0x7fd71c10a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 --2- 172.21.15.57:0/2199078540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103180 0x7fd71c103600 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 --2- 172.21.15.57:0/2199078540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd71c101f90 0x7fd71c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 -- 172.21.15.57:0/2199078540 >> 172.21.15.57:0/2199078540 conn(0x7fd71c0fd500 msgr2=0x7fd71c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 -- 172.21.15.57:0/2199078540 shutdown_connections 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.976+0000 7fd7230b2640 1 -- 172.21.15.57:0/2199078540 wait complete. 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 Processor -- start 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 -- start start 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101f90 0x7fd71c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd71c103180 0x7fd71c1986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 0x7fd71c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10db20 con 0x7fd71c103b40 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10d9a0 con 0x7fd71c103180 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd7230b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c10dca0 con 0x7fd71c101f90 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 0x7fd71c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 0x7fd71c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43734/0 (socket says 172.21.15.57:43734) 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 -- 172.21.15.57:0/3134565938 learned_addr learned my addr 172.21.15.57:0/3134565938 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd720e4e640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101f90 0x7fd71c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd71bfff640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd71c103180 0x7fd71c1986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 -- 172.21.15.57:0/3134565938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101f90 msgr2=0x7fd71c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101f90 0x7fd71c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 -- 172.21.15.57:0/3134565938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd71c103180 msgr2=0x7fd71c1986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd71c103180 0x7fd71c1986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 -- 172.21.15.57:0/3134565938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71c19fd90 con 0x7fd71c103b40 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.977+0000 7fd72164f640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 0x7fd71c19f720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd71400eaa0 tx=0x7fd71400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.978+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71400cdb0 con 0x7fd71c103b40 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.978+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd714005020 con 0x7fd71c103b40 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.978+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71c1a0080 con 0x7fd71c103b40 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.978+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7140116d0 con 0x7fd71c103b40 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.978+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd71c1a05d0 con 0x7fd71c103b40 2024-04-21T08:43:19.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.979+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71c1092b0 con 0x7fd71c103b40 2024-04-21T08:43:19.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.980+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd7140051d0 con 0x7fd71c103b40 2024-04-21T08:43:19.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.980+0000 7fd719ffb640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7080a3bb0 0x7fd7080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:19.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.981+0000 7fd720e4e640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7080a3bb0 0x7fd7080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:19.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.981+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd7140d0770 con 0x7fd71c103b40 2024-04-21T08:43:19.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.981+0000 7fd720e4e640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7080a3bb0 0x7fd7080a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd70c005fd0 tx=0x7fd70c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:19.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:19.985+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd714016030 con 0x7fd71c103b40 2024-04-21T08:43:20.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:19 smithi129 ceph-mon[31272]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:20.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:19 smithi136 ceph-mon[31791]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:20.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.177+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd71c062b50 con 0x7fd71c103b40 2024-04-21T08:43:20.178 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.179+0000 7fd719ffb640 1 -- 172.21.15.57:0/3134565938 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7fd71409cf10 con 0x7fd71c103b40 2024-04-21T08:43:20.179 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:20.179 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:20.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7080a3bb0 msgr2=0x7fd7080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7080a3bb0 0x7fd7080a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd70c005fd0 tx=0x7fd70c00b410 comp rx=0 tx=0).stop 2024-04-21T08:43:20.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 msgr2=0x7fd71c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 0x7fd71c19f720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd71400eaa0 tx=0x7fd71400ef60 comp rx=0 tx=0).stop 2024-04-21T08:43:20.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 shutdown_connections 2024-04-21T08:43:20.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd7080a3bb0 0x7fd7080a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd71c103b40 0x7fd71c19f720 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd71c103180 0x7fd71c1986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 --2- 172.21.15.57:0/3134565938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd71c101f90 0x7fd71c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.181+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 >> 172.21.15.57:0/3134565938 conn(0x7fd71c0fd500 msgr2=0x7fd71c106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.182+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 shutdown_connections 2024-04-21T08:43:20.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.182+0000 7fd7230b2640 1 -- 172.21.15.57:0/3134565938 wait complete. 2024-04-21T08:43:20.193 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-21T08:43:20.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 -- 172.21.15.57:0/167637040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968102a10 msgr2=0x7f6968104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 --2- 172.21.15.57:0/167637040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968102a10 0x7f6968104e00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6958009900 tx=0x7f695802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 -- 172.21.15.57:0/167637040 shutdown_connections 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 --2- 172.21.15.57:0/167637040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 0x7f696810a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 --2- 172.21.15.57:0/167637040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968105340 0x7f6968107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 --2- 172.21.15.57:0/167637040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968102a10 0x7f6968104e00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.298+0000 7f696cc0f640 1 -- 172.21.15.57:0/167637040 >> 172.21.15.57:0/167637040 conn(0x7f69680fc400 msgr2=0x7f69680fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 -- 172.21.15.57:0/167637040 shutdown_connections 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 -- 172.21.15.57:0/167637040 wait complete. 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 Processor -- start 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 -- start start 2024-04-21T08:43:20.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968102a10 0x7f69681982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 0x7f6968198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 0x7f696819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f696810db40 con 0x7f6968105340 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f696810d9c0 con 0x7f6968102a10 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.299+0000 7f696cc0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f696810dcc0 con 0x7f6968107c70 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 0x7f6968198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 0x7f6968198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43764/0 (socket says 172.21.15.57:43764) 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 -- 172.21.15.57:0/1056883830 learned_addr learned my addr 172.21.15.57:0/1056883830 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f696659c640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968102a10 0x7f69681982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6966d9d640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 0x7f696819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 -- 172.21.15.57:0/1056883830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 msgr2=0x7f696819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 0x7f696819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 -- 172.21.15.57:0/1056883830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968102a10 msgr2=0x7f69681982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968102a10 0x7f69681982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 -- 172.21.15.57:0/1056883830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696819feb0 con 0x7f6968105340 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6966d9d640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 0x7f696819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f696659c640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968102a10 0x7f69681982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f6965d9b640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 0x7f6968198820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f695c00d770 tx=0x7f695c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c017070 con 0x7f6968105340 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f695c009c30 con 0x7f6968105340 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c00f620 con 0x7f6968105340 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69681a01a0 con 0x7f6968105340 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.300+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69681a07c0 con 0x7f6968105340 2024-04-21T08:43:20.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.301+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69680fdef0 con 0x7f6968105340 2024-04-21T08:43:20.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.303+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f695c01a070 con 0x7f6968105340 2024-04-21T08:43:20.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.303+0000 7f69577fe640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f69500a3c30 0x7f69500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.303+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f695c0d0140 con 0x7f6968105340 2024-04-21T08:43:20.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.303+0000 7f696659c640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f69500a3c30 0x7f69500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.304+0000 7f696659c640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f69500a3c30 0x7f69500a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f69580096c0 tx=0x7f6958000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.307+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f695c009de0 con 0x7f6968105340 2024-04-21T08:43:20.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.534+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f696802d830 con 0x7f6968105340 2024-04-21T08:43:20.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.760+0000 7f69577fe640 1 -- 172.21.15.57:0/1056883830 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f695c09c960 con 0x7f6968105340 2024-04-21T08:43:20.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f69500a3c30 msgr2=0x7f69500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f69500a3c30 0x7f69500a60f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f69580096c0 tx=0x7f6958000bc0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 msgr2=0x7f6968198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 0x7f6968198820 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f695c00d770 tx=0x7f695c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:43:20.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 shutdown_connections 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f69500a3c30 0x7f69500a60f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6968107c70 0x7f696819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6968105340 0x7f6968198820 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 --2- 172.21.15.57:0/1056883830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6968102a10 0x7f69681982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 >> 172.21.15.57:0/1056883830 conn(0x7f69680fc400 msgr2=0x7f6968103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 shutdown_connections 2024-04-21T08:43:20.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.763+0000 7f696cc0f640 1 -- 172.21.15.57:0/1056883830 wait complete. 2024-04-21T08:43:20.774 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-21T08:43:20.774 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-21T08:43:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 -- 172.21.15.57:0/2684790979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 msgr2=0x7f06d4107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 --2- 172.21.15.57:0/2684790979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4107750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f06c4009900 tx=0x7f06c402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 -- 172.21.15.57:0/2684790979 shutdown_connections 2024-04-21T08:43:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 --2- 172.21.15.57:0/2684790979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06d4107c90 0x7f06d410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 --2- 172.21.15.57:0/2684790979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4107750 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 --2- 172.21.15.57:0/2684790979 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 0x7f06d4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 -- 172.21.15.57:0/2684790979 >> 172.21.15.57:0/2684790979 conn(0x7f06d40fc440 msgr2=0x7f06d40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 -- 172.21.15.57:0/2684790979 shutdown_connections 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.879+0000 7f06da4e0640 1 -- 172.21.15.57:0/2684790979 wait complete. 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 Processor -- start 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 -- start start 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 0x7f06d4198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06d4107c90 0x7f06d419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f06d410daf0 con 0x7f06d4105360 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06d410d970 con 0x7f06d4107c90 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06da4e0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f06d410dc70 con 0x7f06d4102a30 2024-04-21T08:43:20.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06d37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43786/0 (socket says 172.21.15.57:43786) 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06d37fe640 1 -- 172.21.15.57:0/911341449 learned_addr learned my addr 172.21.15.57:0/911341449 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06d8a7d640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06d4107c90 0x7f06d419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.880+0000 7f06d3fff640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 0x7f06d4198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d37fe640 1 -- 172.21.15.57:0/911341449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 msgr2=0x7f06d4198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d37fe640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 0x7f06d4198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d37fe640 1 -- 172.21.15.57:0/911341449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06d4107c90 msgr2=0x7f06d419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d37fe640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06d4107c90 0x7f06d419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d37fe640 1 -- 172.21.15.57:0/911341449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d419fe70 con 0x7f06d4105360 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d3fff640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 0x7f06d4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d37fe640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4198790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f06c4000c00 tx=0x7f06c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4043020 con 0x7f06d4105360 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06c4004aa0 con 0x7f06d4105360 2024-04-21T08:43:20.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d41a0100 con 0x7f06d4105360 2024-04-21T08:43:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4003d00 con 0x7f06d4105360 2024-04-21T08:43:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.881+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d41a06a0 con 0x7f06d4105360 2024-04-21T08:43:20.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.882+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d40fdf30 con 0x7f06d4105360 2024-04-21T08:43:20.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.883+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f06c4046020 con 0x7f06d4105360 2024-04-21T08:43:20.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.884+0000 7f06d17fa640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06bc0a3c30 0x7f06bc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:20.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.884+0000 7f06d3fff640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06bc0a3c30 0x7f06bc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:20.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.884+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f06c40f2b80 con 0x7f06d4105360 2024-04-21T08:43:20.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.884+0000 7f06d3fff640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06bc0a3c30 0x7f06bc0a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f06c80058c0 tx=0x7f06c8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:20.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:20.888+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f06c404d020 con 0x7f06d4105360 2024-04-21T08:43:21.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:20 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/268833349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:21.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:20 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3134565938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:21.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:20 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1056883830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:43:21.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:20 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/268833349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:21.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:20 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3134565938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:21.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:20 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1056883830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:43:21.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.120+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f06d4062b50 con 0x7f06d4105360 2024-04-21T08:43:21.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:20 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/268833349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:21.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:20 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3134565938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:21.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:20 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1056883830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-21T08:43:21.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.766+0000 7f06d17fa640 1 -- 172.21.15.57:0/911341449 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f06c40a9020 con 0x7f06d4105360 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06bc0a3c30 msgr2=0x7f06bc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06bc0a3c30 0x7f06bc0a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f06c80058c0 tx=0x7f06c8005850 comp rx=0 tx=0).stop 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 msgr2=0x7f06d4198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4198790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f06c4000c00 tx=0x7f06c4004270 comp rx=0 tx=0).stop 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 shutdown_connections 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06bc0a3c30 0x7f06bc0a60f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06d4107c90 0x7f06d419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06d4105360 0x7f06d4198790 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 --2- 172.21.15.57:0/911341449 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f06d4102a30 0x7f06d4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 >> 172.21.15.57:0/911341449 conn(0x7f06d40fc440 msgr2=0x7f06d41032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 shutdown_connections 2024-04-21T08:43:21.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.769+0000 7f06da4e0640 1 -- 172.21.15.57:0/911341449 wait complete. 2024-04-21T08:43:21.783 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 -- 172.21.15.57:0/2543261287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 msgr2=0x7f030c1092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 --2- 172.21.15.57:0/2543261287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c1092c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0304009ca0 tx=0x7f030402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 -- 172.21.15.57:0/2543261287 shutdown_connections 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 --2- 172.21.15.57:0/2543261287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c1092c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 --2- 172.21.15.57:0/2543261287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030c101fe0 0x7f030c102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 --2- 172.21.15.57:0/2543261287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 0x7f030c1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 -- 172.21.15.57:0/2543261287 >> 172.21.15.57:0/2543261287 conn(0x7f030c0fc400 msgr2=0x7f030c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 -- 172.21.15.57:0/2543261287 shutdown_connections 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.894+0000 7f0311c55640 1 -- 172.21.15.57:0/2543261287 wait complete. 2024-04-21T08:43:21.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 Processor -- start 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 -- start start 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 0x7f030c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030c101fe0 0x7f030c1986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f030c10c9d0 con 0x7f030c1029a0 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f030c10c850 con 0x7f030c100e90 2024-04-21T08:43:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f0311c55640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f030c10cb50 con 0x7f030c101fe0 2024-04-21T08:43:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c19f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43804/0 (socket says 172.21.15.57:43804) 2024-04-21T08:43:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 -- 172.21.15.57:0/4129555024 learned_addr learned my addr 172.21.15.57:0/4129555024 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030b7fe640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 0x7f030c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030affd640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030c101fe0 0x7f030c1986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 -- 172.21.15.57:0/4129555024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030c101fe0 msgr2=0x7f030c1986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030c101fe0 0x7f030c1986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 -- 172.21.15.57:0/4129555024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 msgr2=0x7f030c198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 0x7f030c198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030bfff640 1 -- 172.21.15.57:0/4129555024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030c19fd80 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.895+0000 7f030b7fe640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 0x7f030c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.896+0000 7f030bfff640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c19f710 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f030402c700 tx=0x7f0304034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.896+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304039040 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.896+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0304044070 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.896+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030c1a0010 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.896+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03040042a0 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.896+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030c1a0500 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.897+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f030c1a0820 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.898+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0304003970 con 0x7f030c1029a0 2024-04-21T08:43:21.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.898+0000 7f0308ff9640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f02f40a3bb0 0x7f02f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.899+0000 7f030b7fe640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f02f40a3bb0 0x7f02f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.899+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f03040f2b00 con 0x7f030c1029a0 2024-04-21T08:43:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.899+0000 7f030b7fe640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f02f40a3bb0 0x7f02f40a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0058c0 tx=0x7f02fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:21.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:21.903+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03040bf240 con 0x7f030c1029a0 2024-04-21T08:43:22.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:21 smithi129 ceph-mon[31272]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:21 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1056883830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:43:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:21 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:22.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:21 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/911341449' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:43:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:21 smithi057 ceph-mon[27968]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:21 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1056883830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:43:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:21 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:22.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:21 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/911341449' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:43:22.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.132+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f030c1a0e30 con 0x7f030c1029a0 2024-04-21T08:43:22.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:21 smithi136 ceph-mon[31791]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:22.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:21 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1056883830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-21T08:43:22.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:21 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:22.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:21 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/911341449' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-21T08:43:22.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.785+0000 7f0308ff9640 1 -- 172.21.15.57:0/4129555024 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f03040beb40 con 0x7f030c1029a0 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f02f40a3bb0 msgr2=0x7f02f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f02f40a3bb0 0x7f02f40a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0058c0 tx=0x7f02fc005850 comp rx=0 tx=0).stop 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 msgr2=0x7f030c19f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c19f710 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f030402c700 tx=0x7f0304034760 comp rx=0 tx=0).stop 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 shutdown_connections 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f02f40a3bb0 0x7f02f40a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f030c1029a0 0x7f030c19f710 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030c101fe0 0x7f030c1986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 --2- 172.21.15.57:0/4129555024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030c100e90 0x7f030c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 >> 172.21.15.57:0/4129555024 conn(0x7f030c0fc400 msgr2=0x7f030c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:22.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.788+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 shutdown_connections 2024-04-21T08:43:22.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.789+0000 7f0311c55640 1 -- 172.21.15.57:0/4129555024 wait complete. 2024-04-21T08:43:22.802 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-21T08:43:22.803 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-21T08:43:22.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 -- 172.21.15.57:0/3713043360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4105360 msgr2=0x7fbef4107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 --2- 172.21.15.57:0/3713043360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4105360 0x7fbef4107750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009900 tx=0x7fbee802bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:22.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 -- 172.21.15.57:0/3713043360 shutdown_connections 2024-04-21T08:43:22.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 --2- 172.21.15.57:0/3713043360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef4107c90 0x7fbef410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 --2- 172.21.15.57:0/3713043360 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4105360 0x7fbef4107750 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 --2- 172.21.15.57:0/3713043360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbef4102a30 0x7fbef4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 -- 172.21.15.57:0/3713043360 >> 172.21.15.57:0/3713043360 conn(0x7fbef40fc440 msgr2=0x7fbef40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.914+0000 7fbefa606640 1 -- 172.21.15.57:0/3713043360 shutdown_connections 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 -- 172.21.15.57:0/3713043360 wait complete. 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 Processor -- start 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 -- start start 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbef4102a30 0x7fbef41023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef4105360 0x7fbef41009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 0x7fbef4100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbef410db40 con 0x7fbef4107c90 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbef410d9c0 con 0x7fbef4105360 2024-04-21T08:43:22.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.915+0000 7fbefa606640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbef410dcc0 con 0x7fbef4102a30 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 0x7fbef4100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 0x7fbef4100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35644/0 (socket says 172.21.15.57:35644) 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 -- 172.21.15.57:0/718349303 learned_addr learned my addr 172.21.15.57:0/718349303 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef37fe640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef4105360 0x7fbef41009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef3fff640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbef4102a30 0x7fbef41023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 -- 172.21.15.57:0/718349303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbef4102a30 msgr2=0x7fbef41023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbef4102a30 0x7fbef41023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 -- 172.21.15.57:0/718349303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef4105360 msgr2=0x7fbef41009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef4105360 0x7fbef41009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 -- 172.21.15.57:0/718349303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef41017b0 con 0x7fbef4107c90 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef8ba3640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 0x7fbef4100f20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00eb10 tx=0x7fbeec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004b20 con 0x7fbef4107c90 2024-04-21T08:43:22.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef41a2350 con 0x7fbef4107c90 2024-04-21T08:43:22.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbeec004cc0 con 0x7fbef4107c90 2024-04-21T08:43:22.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec010670 con 0x7fbef4107c90 2024-04-21T08:43:22.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.916+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef41a2820 con 0x7fbef4107c90 2024-04-21T08:43:22.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.917+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef40691c0 con 0x7fbef4107c90 2024-04-21T08:43:22.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.918+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbeec005020 con 0x7fbef4107c90 2024-04-21T08:43:22.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.919+0000 7fbef17fa640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbedc0a3bb0 0x7fbedc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:22.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.919+0000 7fbef3fff640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbedc0a3bb0 0x7fbedc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:22.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.919+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbeec0d0d40 con 0x7fbef4107c90 2024-04-21T08:43:22.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.920+0000 7fbef3fff640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbedc0a3bb0 0x7fbedc0a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009aa0 tx=0x7fbee4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:22.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:22.924+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbeec09d480 con 0x7fbef4107c90 2024-04-21T08:43:23.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:22 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/911341449' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:43:23.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:22 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:23.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:22 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:43:23.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:22 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4129555024' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:43:23.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:22 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/911341449' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:43:23.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:22 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:23.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:22 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:43:23.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:22 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4129555024' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:43:23.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:22 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/911341449' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-21T08:43:23.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:22 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-21T08:43:23.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:22 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:43:23.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:22 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4129555024' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-21T08:43:23.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.137+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbef4062390 con 0x7fbedc0a3bb0 2024-04-21T08:43:23.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.158+0000 7fbef17fa640 1 -- 172.21.15.57:0/718349303 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbef4062390 con 0x7fbedc0a3bb0 2024-04-21T08:43:23.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbedc0a3bb0 msgr2=0x7fbedc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbedc0a3bb0 0x7fbedc0a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009aa0 tx=0x7fbee4005f90 comp rx=0 tx=0).stop 2024-04-21T08:43:23.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 msgr2=0x7fbef4100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 0x7fbef4100f20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00eb10 tx=0x7fbeec00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 shutdown_connections 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbedc0a3bb0 0x7fbedc0a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbef4107c90 0x7fbef4100f20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbef4105360 0x7fbef41009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 --2- 172.21.15.57:0/718349303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbef4102a30 0x7fbef41023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 >> 172.21.15.57:0/718349303 conn(0x7fbef40fc440 msgr2=0x7fbef4103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 shutdown_connections 2024-04-21T08:43:23.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.160+0000 7fbefa606640 1 -- 172.21.15.57:0/718349303 wait complete. 2024-04-21T08:43:23.171 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-21T08:43:23.280 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 -- 172.21.15.57:0/4193262947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394100e60 msgr2=0x7f2394101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 --2- 172.21.15.57:0/4193262947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394100e60 0x7f2394101280 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2384009900 tx=0x7f238402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 -- 172.21.15.57:0/4193262947 shutdown_connections 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 --2- 172.21.15.57:0/4193262947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2394102a00 0x7f2394109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 --2- 172.21.15.57:0/4193262947 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2394102040 0x7f23941024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 --2- 172.21.15.57:0/4193262947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394100e60 0x7f2394101280 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 -- 172.21.15.57:0/4193262947 >> 172.21.15.57:0/4193262947 conn(0x7f23940fc440 msgr2=0x7f23940fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 -- 172.21.15.57:0/4193262947 shutdown_connections 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 -- 172.21.15.57:0/4193262947 wait complete. 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 Processor -- start 2024-04-21T08:43:23.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.278+0000 7f2399421640 1 -- start start 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2394100e60 0x7f239410b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2394102040 0x7f2394109ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 0x7f239410a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f239410dbb0 con 0x7f2394102a00 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f239410da30 con 0x7f2394100e60 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f239410dd30 con 0x7f2394102040 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 0x7f239410a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 0x7f239410a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35694/0 (socket says 172.21.15.57:35694) 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 -- 172.21.15.57:0/1824531264 learned_addr learned my addr 172.21.15.57:0/1824531264 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2392ffd640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2394100e60 0x7f239410b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23927fc640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2394102040 0x7f2394109ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 -- 172.21.15.57:0/1824531264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2394102040 msgr2=0x7f2394109ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2394102040 0x7f2394109ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 -- 172.21.15.57:0/1824531264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2394100e60 msgr2=0x7f239410b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2394100e60 0x7f239410b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 -- 172.21.15.57:0/1824531264 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239410ac70 con 0x7f2394102a00 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f23937fe640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 0x7f239410a3e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f238c00b970 tx=0x7f238c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c019070 con 0x7f2394102a00 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f238c014440 con 0x7f2394102a00 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23941a23a0 con 0x7f2394102a00 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f238c0075d0 con 0x7f2394102a00 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.279+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23941a2970 con 0x7f2394102a00 2024-04-21T08:43:23.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.280+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2394108320 con 0x7f2394102a00 2024-04-21T08:43:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.282+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f238c01c070 con 0x7f2394102a00 2024-04-21T08:43:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.283+0000 7f237bfff640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f237c0a3bb0 0x7f237c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.283+0000 7f2392ffd640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f237c0a3bb0 0x7f237c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.283+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f238c0d0080 con 0x7f2394102a00 2024-04-21T08:43:23.285 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.283+0000 7f2392ffd640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f237c0a3bb0 0x7f237c0a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f238402c240 tx=0x7f2384000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:23.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.287+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f238c09c1d0 con 0x7f2394102a00 2024-04-21T08:43:23.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.502+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2394062390 con 0x7f237c0a3bb0 2024-04-21T08:43:23.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.527+0000 7f237bfff640 1 -- 172.21.15.57:0/1824531264 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2394062390 con 0x7f237c0a3bb0 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f237c0a3bb0 msgr2=0x7f237c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f237c0a3bb0 0x7f237c0a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f238402c240 tx=0x7f2384000bc0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 msgr2=0x7f239410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 0x7f239410a3e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f238c00b970 tx=0x7f238c00be30 comp rx=0 tx=0).stop 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 shutdown_connections 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f237c0a3bb0 0x7f237c0a6070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2394102a00 0x7f239410a3e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2394102040 0x7f2394109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 --2- 172.21.15.57:0/1824531264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2394100e60 0x7f239410b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 >> 172.21.15.57:0/1824531264 conn(0x7f23940fc440 msgr2=0x7f23941054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:23.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 shutdown_connections 2024-04-21T08:43:23.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.529+0000 7f2399421640 1 -- 172.21.15.57:0/1824531264 wait complete. 2024-04-21T08:43:23.540 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:43:23.143+0000 7f1f0c42c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:43:23.540 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:43:23.143+0000 7f1f0c42c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:43:23.540 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:43:23.143+0000 7f1f0c42c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:43:23.540 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:43:23.143+0000 7f1f0c42c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:43:23.540 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:43:23.143+0000 7f1f0c42c700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:43:23.541 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-21T08:43:23.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 -- 172.21.15.57:0/1954074326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e041031a0 msgr2=0x7f7e04103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 --2- 172.21.15.57:0/1954074326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e041031a0 0x7f7e04103620 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7df4009900 tx=0x7f7df402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:23.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 -- 172.21.15.57:0/1954074326 shutdown_connections 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 --2- 172.21.15.57:0/1954074326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e04103b60 0x7f7e0410a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 --2- 172.21.15.57:0/1954074326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e041031a0 0x7f7e04103620 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 --2- 172.21.15.57:0/1954074326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e04101fb0 0x7f7e041023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 -- 172.21.15.57:0/1954074326 >> 172.21.15.57:0/1954074326 conn(0x7f7e040fd540 msgr2=0x7f7e040ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 -- 172.21.15.57:0/1954074326 shutdown_connections 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.649+0000 7f7e0b674640 1 -- 172.21.15.57:0/1954074326 wait complete. 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 Processor -- start 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 -- start start 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 0x7f7e041981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e041031a0 0x7f7e041986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e04103b60 0x7f7e0419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0410daf0 con 0x7f7e04101fb0 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0410d970 con 0x7f7e04103b60 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e0b674640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0410dc70 con 0x7f7e041031a0 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 0x7f7e041981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 0x7f7e041981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35712/0 (socket says 172.21.15.57:35712) 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 -- 172.21.15.57:0/437570997 learned_addr learned my addr 172.21.15.57:0/437570997 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09c11640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e04103b60 0x7f7e0419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e08c0f640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e041031a0 0x7f7e041986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 -- 172.21.15.57:0/437570997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e041031a0 msgr2=0x7f7e041986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e041031a0 0x7f7e041986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 -- 172.21.15.57:0/437570997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e04103b60 msgr2=0x7f7e0419f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e04103b60 0x7f7e0419f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e09410640 1 -- 172.21.15.57:0/437570997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0419fdd0 con 0x7f7e04101fb0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.650+0000 7f7e08c0f640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e041031a0 0x7f7e041986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.651+0000 7f7e09410640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 0x7f7e041981b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000d770 tx=0x7f7e0000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.651+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00017070 con 0x7f7e04101fb0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.651+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e00009c30 con 0x7f7e04101fb0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.651+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e041a00c0 con 0x7f7e04101fb0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.651+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00003cc0 con 0x7f7e04101fb0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.651+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e041a0660 con 0x7f7e04101fb0 2024-04-21T08:43:23.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.652+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e04109370 con 0x7f7e04101fb0 2024-04-21T08:43:23.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.653+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7e00003e60 con 0x7f7e04101fb0 2024-04-21T08:43:23.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.653+0000 7f7dfa7fc640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7df00a3c30 0x7f7df00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:23.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.654+0000 7f7e08c0f640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7df00a3c30 0x7f7df00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:23.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.654+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7e000cff50 con 0x7f7e04101fb0 2024-04-21T08:43:23.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.654+0000 7f7e08c0f640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7df00a3c30 0x7f7df00a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7df40096c0 tx=0x7f7df4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:23.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.658+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e00009de0 con 0x7f7e04101fb0 2024-04-21T08:43:23.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-mon[27968]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:23.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4129555024' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:43:23.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-mon[27968]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:43:23.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:23 smithi057 ceph-mon[27968]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:23.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.875+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f7e04062420 con 0x7f7df00a3c30 2024-04-21T08:43:23.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.880+0000 7f7dfa7fc640 1 -- 172.21.15.57:0/437570997 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7e04062420 con 0x7f7df00a3c30 2024-04-21T08:43:23.880 INFO:teuthology.orchestra.run.smithi057.stdout:/volumes/qa/sv_0/a30e3ecd-03fa-4266-9371-5de8404d7881 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7df00a3c30 msgr2=0x7f7df00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7df00a3c30 0x7f7df00a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7df40096c0 tx=0x7f7df4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 msgr2=0x7f7e041981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 0x7f7e041981b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000d770 tx=0x7f7e0000dc30 comp rx=0 tx=0).stop 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 shutdown_connections 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f7df00a3c30 0x7f7df00a60f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e04103b60 0x7f7e0419f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e041031a0 0x7f7e041986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 --2- 172.21.15.57:0/437570997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e04101fb0 0x7f7e041981b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 >> 172.21.15.57:0/437570997 conn(0x7f7e040fd540 msgr2=0x7f7e04106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 shutdown_connections 2024-04-21T08:43:23.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.882+0000 7f7e0b674640 1 -- 172.21.15.57:0/437570997 wait complete. 2024-04-21T08:43:23.893 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.999+0000 7f5165bbd640 1 -- 172.21.15.57:0/234267402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160101fe0 msgr2=0x7f5160102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:23.999+0000 7f5165bbd640 1 --2- 172.21.15.57:0/234267402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160101fe0 0x7f5160102460 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5154009900 tx=0x7f515402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- 172.21.15.57:0/234267402 shutdown_connections 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 --2- 172.21.15.57:0/234267402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51601029a0 0x7f51601092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 --2- 172.21.15.57:0/234267402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160101fe0 0x7f5160102460 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 --2- 172.21.15.57:0/234267402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5160100e90 0x7f51601012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- 172.21.15.57:0/234267402 >> 172.21.15.57:0/234267402 conn(0x7f51600fc400 msgr2=0x7f51600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- 172.21.15.57:0/234267402 shutdown_connections 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- 172.21.15.57:0/234267402 wait complete. 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 Processor -- start 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- start start 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 0x7f5160069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5160101fe0 0x7f516019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51601029a0 0x7f516019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f516010c9d0 con 0x7f5160100e90 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f516010c850 con 0x7f5160101fe0 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.000+0000 7f5165bbd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f516010cb50 con 0x7f51601029a0 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 0x7f5160069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 0x7f5160069c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35732/0 (socket says 172.21.15.57:35732) 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 -- 172.21.15.57:0/3550417541 learned_addr learned my addr 172.21.15.57:0/3550417541 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:24.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515effd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5160101fe0 0x7f516019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 -- 172.21.15.57:0/3550417541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51601029a0 msgr2=0x7f516019f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51601029a0 0x7f516019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 -- 172.21.15.57:0/3550417541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5160101fe0 msgr2=0x7f516019cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5160101fe0 0x7f516019cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 -- 172.21.15.57:0/3550417541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516019fe80 con 0x7f5160100e90 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515f7fe640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 0x7f5160069c50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f515000d950 tx=0x7f515000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5150017070 con 0x7f5160100e90 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51601a0110 con 0x7f5160100e90 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5150009c30 con 0x7f5160100e90 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515000f620 con 0x7f5160100e90 2024-04-21T08:43:24.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.001+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51601a0630 con 0x7f5160100e90 2024-04-21T08:43:24.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.002+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5160108310 con 0x7f5160100e90 2024-04-21T08:43:24.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.003+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5150010460 con 0x7f5160100e90 2024-04-21T08:43:24.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.004+0000 7f515cff9640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51480a3bb0 0x7f51480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.004+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51500d0270 con 0x7f5160100e90 2024-04-21T08:43:24.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.004+0000 7f515effd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51480a3bb0 0x7f51480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.004+0000 7f515effd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51480a3bb0 0x7f51480a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5154005ab0 tx=0x7f5154005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.008+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f515009ca10 con 0x7f5160100e90 2024-04-21T08:43:24.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:23 smithi129 ceph-mon[31272]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:24.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4129555024' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:43:24.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:23 smithi129 ceph-mon[31272]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:43:24.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:23 smithi129 ceph-mon[31272]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:24.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:23 smithi136 ceph-mon[31791]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-21T08:43:24.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4129555024' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-21T08:43:24.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:23 smithi136 ceph-mon[31791]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-21T08:43:24.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:23 smithi136 ceph-mon[31791]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:24.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.222+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5160062420 con 0x7f51480a3bb0 2024-04-21T08:43:24.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.248+0000 7f515cff9640 1 -- 172.21.15.57:0/3550417541 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5160062420 con 0x7f51480a3bb0 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51480a3bb0 msgr2=0x7f51480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51480a3bb0 0x7f51480a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5154005ab0 tx=0x7f5154005fd0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 msgr2=0x7f5160069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 0x7f5160069c50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f515000d950 tx=0x7f515000de10 comp rx=0 tx=0).stop 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 shutdown_connections 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f51480a3bb0 0x7f51480a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51601029a0 0x7f516019f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5160101fe0 0x7f516019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 --2- 172.21.15.57:0/3550417541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5160100e90 0x7f5160069c50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 >> 172.21.15.57:0/3550417541 conn(0x7f51600fc400 msgr2=0x7f51601054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 shutdown_connections 2024-04-21T08:43:24.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.250+0000 7f5165bbd640 1 -- 172.21.15.57:0/3550417541 wait complete. 2024-04-21T08:43:24.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 -- 172.21.15.57:0/1567677155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 msgr2=0x7feadc107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 --2- 172.21.15.57:0/1567677155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc107750 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fead0009900 tx=0x7fead002bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 -- 172.21.15.57:0/1567677155 shutdown_connections 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 --2- 172.21.15.57:0/1567677155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc107c90 0x7feadc10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 --2- 172.21.15.57:0/1567677155 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc107750 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 --2- 172.21.15.57:0/1567677155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc102a30 0x7feadc104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 -- 172.21.15.57:0/1567677155 >> 172.21.15.57:0/1567677155 conn(0x7feadc0fc440 msgr2=0x7feadc0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 -- 172.21.15.57:0/1567677155 shutdown_connections 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.367+0000 7feae2874640 1 -- 172.21.15.57:0/1567677155 wait complete. 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 Processor -- start 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 -- start start 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc102a30 0x7feadc1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc107c90 0x7feadc100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feadc10db30 con 0x7feadc105360 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feadc10d9b0 con 0x7feadc107c90 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feae2874640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feadc10dcb0 con 0x7feadc102a30 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feadb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feadb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc1009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35762/0 (socket says 172.21.15.57:35762) 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.368+0000 7feadb7fe640 1 -- 172.21.15.57:0/2722872068 learned_addr learned my addr 172.21.15.57:0/2722872068 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:24.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feae0e11640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc107c90 0x7feadc100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadbfff640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc102a30 0x7feadc1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadb7fe640 1 -- 172.21.15.57:0/2722872068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc102a30 msgr2=0x7feadc1023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadb7fe640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc102a30 0x7feadc1023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadb7fe640 1 -- 172.21.15.57:0/2722872068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc107c90 msgr2=0x7feadc100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadb7fe640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc107c90 0x7feadc100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadb7fe640 1 -- 172.21.15.57:0/2722872068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feadc1017b0 con 0x7feadc105360 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadbfff640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc102a30 0x7feadc1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feae0e11640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc107c90 0x7feadc100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feadb7fe640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc1009e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fead0036040 tx=0x7fead00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0043020 con 0x7feadc105360 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fead0004aa0 con 0x7feadc105360 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feadc1a22c0 con 0x7feadc105360 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead0003de0 con 0x7feadc105360 2024-04-21T08:43:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.369+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feadc1a28e0 con 0x7feadc105360 2024-04-21T08:43:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.370+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feadc0fdf30 con 0x7feadc105360 2024-04-21T08:43:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.371+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fead0046020 con 0x7feadc105360 2024-04-21T08:43:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.372+0000 7fead97fa640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feac40a3c30 0x7feac40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.372+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fead00f2b10 con 0x7feadc105360 2024-04-21T08:43:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.372+0000 7feadbfff640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feac40a3c30 0x7feac40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.372+0000 7feadbfff640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feac40a3c30 0x7feac40a60f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feacc00baa0 tx=0x7feacc005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.376+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fead004d020 con 0x7feadc105360 2024-04-21T08:43:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.597+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7feadc062420 con 0x7feac40a3c30 2024-04-21T08:43:24.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.602+0000 7fead97fa640 1 -- 172.21.15.57:0/2722872068 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7feadc062420 con 0x7feac40a3c30 2024-04-21T08:43:24.603 INFO:teuthology.orchestra.run.smithi057.stdout:/volumes/qa/sv_1/0d190491-43b7-4ad2-b994-7f7858bb7cfc 2024-04-21T08:43:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feac40a3c30 msgr2=0x7feac40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feac40a3c30 0x7feac40a60f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feacc00baa0 tx=0x7feacc005d90 comp rx=0 tx=0).stop 2024-04-21T08:43:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 msgr2=0x7feadc1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc1009e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fead0036040 tx=0x7fead00042f0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 shutdown_connections 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7feac40a3c30 0x7feac40a60f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feadc107c90 0x7feadc100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feadc105360 0x7feadc1009e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 --2- 172.21.15.57:0/2722872068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feadc102a30 0x7feadc1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 >> 172.21.15.57:0/2722872068 conn(0x7feadc0fc440 msgr2=0x7feadc103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 shutdown_connections 2024-04-21T08:43:24.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.605+0000 7feae2874640 1 -- 172.21.15.57:0/2722872068 wait complete. 2024-04-21T08:43:24.616 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 -- 172.21.15.57:0/1764709956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 msgr2=0x7fe9a8107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 --2- 172.21.15.57:0/1764709956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a8107760 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009900 tx=0x7fe99c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 -- 172.21.15.57:0/1764709956 shutdown_connections 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 --2- 172.21.15.57:0/1764709956 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 0x7fe9a810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 --2- 172.21.15.57:0/1764709956 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a8107760 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 --2- 172.21.15.57:0/1764709956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9a8102a40 0x7fe9a8104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 -- 172.21.15.57:0/1764709956 >> 172.21.15.57:0/1764709956 conn(0x7fe9a80fc470 msgr2=0x7fe9a80fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 -- 172.21.15.57:0/1764709956 shutdown_connections 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.718+0000 7fe9ade14640 1 -- 172.21.15.57:0/1764709956 wait complete. 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 Processor -- start 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 -- start start 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9a8102a40 0x7fe9a81023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a81009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 0x7fe9a8100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a810dac0 con 0x7fe9a8105370 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a810d940 con 0x7fe9a8107ca0 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9ade14640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a810dc40 con 0x7fe9a8102a40 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a81009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a81009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35786/0 (socket says 172.21.15.57:35786) 2024-04-21T08:43:24.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 -- 172.21.15.57:0/2900243190 learned_addr learned my addr 172.21.15.57:0/2900243190 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a7fff640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 0x7fe9a8100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 -- 172.21.15.57:0/2900243190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9a8102a40 msgr2=0x7fe9a81023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9a8102a40 0x7fe9a81023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 -- 172.21.15.57:0/2900243190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 msgr2=0x7fe9a8100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.719+0000 7fe9a6ffd640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 0x7fe9a8100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9a6ffd640 1 -- 172.21.15.57:0/2900243190 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a81017c0 con 0x7fe9a8105370 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9a7fff640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 0x7fe9a8100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9a6ffd640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a81009f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe99c036040 tx=0x7fe99c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c043020 con 0x7fe9a8105370 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe99c004aa0 con 0x7fe9a8105370 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a81a21f0 con 0x7fe9a8105370 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c003de0 con 0x7fe9a8105370 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.720+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a81a2790 con 0x7fe9a8105370 2024-04-21T08:43:24.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.721+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a80fdf60 con 0x7fe9a8105370 2024-04-21T08:43:24.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.722+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe99c046020 con 0x7fe9a8105370 2024-04-21T08:43:24.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.723+0000 7fe9a4ff9640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9900a3c30 0x7fe9900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:24.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.723+0000 7fe9a77fe640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9900a3c30 0x7fe9900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:24.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.723+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe99c0f2b10 con 0x7fe9a8105370 2024-04-21T08:43:24.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.723+0000 7fe9a77fe640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9900a3c30 0x7fe9900a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe99800a990 tx=0x7fe998005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:24.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.727+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe99c04d020 con 0x7fe9a8105370 2024-04-21T08:43:24.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.804+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe99c0bb1c0 con 0x7fe9a8105370 2024-04-21T08:43:24.968 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.968+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9a802d5c0 con 0x7fe9a8105370 2024-04-21T08:43:24.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.969+0000 7fe9a4ff9640 1 -- 172.21.15.57:0/2900243190 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fe99c0bf250 con 0x7fe9a8105370 2024-04-21T08:43:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:24.969 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:43:24.969 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9900a3c30 msgr2=0x7fe9900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9900a3c30 0x7fe9900a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe99800a990 tx=0x7fe998005c70 comp rx=0 tx=0).stop 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 msgr2=0x7fe9a81009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a81009f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe99c036040 tx=0x7fe99c0042f0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 shutdown_connections 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9900a3c30 0x7fe9900a60f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a8107ca0 0x7fe9a8100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9a8105370 0x7fe9a81009f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 --2- 172.21.15.57:0/2900243190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9a8102a40 0x7fe9a81023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 >> 172.21.15.57:0/2900243190 conn(0x7fe9a80fc470 msgr2=0x7fe9a8103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:24.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 shutdown_connections 2024-04-21T08:43:24.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:24.971+0000 7fe9ade14640 1 -- 172.21.15.57:0/2900243190 wait complete. 2024-04-21T08:43:24.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:24 smithi057 ceph-mon[27968]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:24.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:24 smithi057 ceph-mon[27968]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:24.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:24 smithi057 ceph-mon[27968]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:24.982 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:43:25.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:24 smithi129 ceph-mon[31272]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:24 smithi129 ceph-mon[31272]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:24 smithi129 ceph-mon[31272]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3793465494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8100e70 msgr2=0x7fe8e8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3793465494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8100e70 0x7fe8e8101290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009900 tx=0x7fe8e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3793465494 shutdown_connections 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3793465494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8e8102980 0x7fe8e81092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3793465494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8e8101fc0 0x7fe8e8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3793465494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8100e70 0x7fe8e8101290 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3793465494 >> 172.21.15.57:0/3793465494 conn(0x7fe8e80fc3e0 msgr2=0x7fe8e80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3793465494 shutdown_connections 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3793465494 wait complete. 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.084+0000 7fe8ef3ce640 1 Processor -- start 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 -- start start 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8e8100e70 0x7fe8e81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8e8101fc0 0x7fe8e8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 0x7fe8e819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e810db10 con 0x7fe8e8102980 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e810d990 con 0x7fe8e8100e70 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ef3ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e810dc90 con 0x7fe8e8101fc0 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 0x7fe8e819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 0x7fe8e819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35792/0 (socket says 172.21.15.57:35792) 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 -- 172.21.15.57:0/3356076539 learned_addr learned my addr 172.21.15.57:0/3356076539 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed16a640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8e8100e70 0x7fe8e81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ec969640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8e8101fc0 0x7fe8e8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 -- 172.21.15.57:0/3356076539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8e8101fc0 msgr2=0x7fe8e8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8e8101fc0 0x7fe8e8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 -- 172.21.15.57:0/3356076539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8e8100e70 msgr2=0x7fe8e81981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8e8100e70 0x7fe8e81981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 -- 172.21.15.57:0/3356076539 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e819fdf0 con 0x7fe8e8102980 2024-04-21T08:43:25.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.085+0000 7fe8ed96b640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 0x7fe8e819f780 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000eaa0 tx=0x7fe8e000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.086+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e000cdb0 con 0x7fe8e8102980 2024-04-21T08:43:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.086+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e81a00e0 con 0x7fe8e8102980 2024-04-21T08:43:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.086+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8e0005020 con 0x7fe8e8102980 2024-04-21T08:43:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.086+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00116d0 con 0x7fe8e8102980 2024-04-21T08:43:25.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.086+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e81a05e0 con 0x7fe8e8102980 2024-04-21T08:43:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.086+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e81081f0 con 0x7fe8e8102980 2024-04-21T08:43:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.088+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe8e00051d0 con 0x7fe8e8102980 2024-04-21T08:43:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.088+0000 7fe8de7fc640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe8d40a3c40 0x7fe8d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.088+0000 7fe8ed16a640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe8d40a3c40 0x7fe8d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.088+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe8e00d0a80 con 0x7fe8e8102980 2024-04-21T08:43:25.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.089+0000 7fe8ed16a640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe8d40a3c40 0x7fe8d40a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4000c00 tx=0x7fe8e4005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.093+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8e0016030 con 0x7fe8e8102980 2024-04-21T08:43:25.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:24 smithi136 ceph-mon[31791]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:24 smithi136 ceph-mon[31791]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:24 smithi136 ceph-mon[31791]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.282+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8e8062b50 con 0x7fe8e8102980 2024-04-21T08:43:25.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.283+0000 7fe8de7fc640 1 -- 172.21.15.57:0/3356076539 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7fe8e009d0c0 con 0x7fe8e8102980 2024-04-21T08:43:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:25.284 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe8d40a3c40 msgr2=0x7fe8d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe8d40a3c40 0x7fe8d40a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4000c00 tx=0x7fe8e4005be0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 msgr2=0x7fe8e819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 0x7fe8e819f780 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000eaa0 tx=0x7fe8e000ef60 comp rx=0 tx=0).stop 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 shutdown_connections 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe8d40a3c40 0x7fe8d40a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8e8102980 0x7fe8e819f780 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8e8101fc0 0x7fe8e8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 --2- 172.21.15.57:0/3356076539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8e8100e70 0x7fe8e81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 >> 172.21.15.57:0/3356076539 conn(0x7fe8e80fc3e0 msgr2=0x7fe8e81053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.285+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 shutdown_connections 2024-04-21T08:43:25.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.286+0000 7fe8ef3ce640 1 -- 172.21.15.57:0/3356076539 wait complete. 2024-04-21T08:43:25.297 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-21T08:43:25.297 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:43:25.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 -- 172.21.15.57:0/3451058438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 msgr2=0x7f5a2c104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 --2- 172.21.15.57:0/3451058438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c104ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009900 tx=0x7f5a2002bc30 comp rx=0 tx=0).stop 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 -- 172.21.15.57:0/3451058438 shutdown_connections 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 --2- 172.21.15.57:0/3451058438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a2c107d40 0x7f5a2c10a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 --2- 172.21.15.57:0/3451058438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a2c105410 0x7f5a2c107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 --2- 172.21.15.57:0/3451058438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c104ed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 -- 172.21.15.57:0/3451058438 >> 172.21.15.57:0/3451058438 conn(0x7f5a2c0fc470 msgr2=0x7f5a2c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 -- 172.21.15.57:0/3451058438 shutdown_connections 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 -- 172.21.15.57:0/3451058438 wait complete. 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.399+0000 7f5a30a81640 1 Processor -- start 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 -- start start 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a2c105410 0x7f5a2c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a2c107d40 0x7f5a2c19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2c10db20 con 0x7f5a2c102ae0 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2c10d9a0 con 0x7f5a2c107d40 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a30a81640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2c10dca0 con 0x7f5a2c105410 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35810/0 (socket says 172.21.15.57:35810) 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 -- 172.21.15.57:0/4062210645 learned_addr learned my addr 172.21.15.57:0/4062210645 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 -- 172.21.15.57:0/4062210645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a2c105410 msgr2=0x7f5a2c1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a2c105410 0x7f5a2c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 -- 172.21.15.57:0/4062210645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a2c107d40 msgr2=0x7f5a2c19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a2c107d40 0x7f5a2c19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 -- 172.21.15.57:0/4062210645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a2c19fda0 con 0x7f5a2c102ae0 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a2a59c640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c198180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5a20036040 tx=0x7f5a200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.400+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20043020 con 0x7f5a2c102ae0 2024-04-21T08:43:25.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.401+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a2c1a0030 con 0x7f5a2c102ae0 2024-04-21T08:43:25.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.401+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a200054c0 con 0x7f5a2c102ae0 2024-04-21T08:43:25.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.401+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20003db0 con 0x7f5a2c102ae0 2024-04-21T08:43:25.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.401+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a2c1a05d0 con 0x7f5a2c102ae0 2024-04-21T08:43:25.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.401+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a2c0fdf60 con 0x7f5a2c102ae0 2024-04-21T08:43:25.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.403+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a20046020 con 0x7f5a2c102ae0 2024-04-21T08:43:25.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.403+0000 7f5a1b7fe640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5a140a3c40 0x7f5a140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.403+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5a200f2c60 con 0x7f5a2c102ae0 2024-04-21T08:43:25.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.403+0000 7f5a29d9b640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5a140a3c40 0x7f5a140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.404+0000 7f5a29d9b640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5a140a3c40 0x7f5a140a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c199690 tx=0x7f5a1c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.408+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a2004d020 con 0x7f5a2c102ae0 2024-04-21T08:43:25.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.591+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5a2c062390 con 0x7f5a140a3c40 2024-04-21T08:43:25.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.602+0000 7f5a1b7fe640 1 -- 172.21.15.57:0/4062210645 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340444 (secure 0 0 0) 0x7f5a2c062390 con 0x7f5a140a3c40 2024-04-21T08:43:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:25.609 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5a140a3c40 msgr2=0x7f5a140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5a140a3c40 0x7f5a140a6100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c199690 tx=0x7f5a1c005c70 comp rx=0 tx=0).stop 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 msgr2=0x7f5a2c198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c198180 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5a20036040 tx=0x7f5a200042f0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 shutdown_connections 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5a140a3c40 0x7f5a140a6100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a2c107d40 0x7f5a2c19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a2c105410 0x7f5a2c1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 --2- 172.21.15.57:0/4062210645 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a2c102ae0 0x7f5a2c198180 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 >> 172.21.15.57:0/4062210645 conn(0x7f5a2c0fc470 msgr2=0x7f5a2c103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 shutdown_connections 2024-04-21T08:43:25.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.605+0000 7f5a30a81640 1 -- 172.21.15.57:0/4062210645 wait complete. 2024-04-21T08:43:25.617 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-21T08:43:24.610381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":140,"num_read_kb":123,"num_write":230,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360768,"kb_used_data":11540,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581056,"statfs":{"total":1151940427776,"available":1151571001344,"internally_reserved":0,"allocated":11816960,"data_stored":7530903,"data_compressed":286272,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757483+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:43:02.757483+0000","last_peered":"2024-04-21T08:43:02.757483+0000","last_clean":"2024-04-21T08:43:02.757483+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:43:02.757483+0000","last_undegraded":"2024-04-21T08:43:02.757483+0000","last_fullsized":"2024-04-21T08:43:02.757483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756273+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:43:02.756273+0000","last_peered":"2024-04-21T08:43:02.756273+0000","last_clean":"2024-04-21T08:43:02.756273+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:43:02.756273+0000","last_undegraded":"2024-04-21T08:43:02.756273+0000","last_fullsized":"2024-04-21T08:43:02.756273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756018+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:43:02.756018+0000","last_peered":"2024-04-21T08:43:02.756018+0000","last_clean":"2024-04-21T08:43:02.756018+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:43:02.756018+0000","last_undegraded":"2024-04-21T08:43:02.756018+0000","last_fullsized":"2024-04-21T08:43:02.756018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757831+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:43:02.757831+0000","last_peered":"2024-04-21T08:43:02.757831+0000","last_clean":"2024-04-21T08:43:02.757831+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:43:02.757831+0000","last_undegraded":"2024-04-21T08:43:02.757831+0000","last_fullsized":"2024-04-21T08:43:02.757831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756601+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:43:02.756601+0000","last_peered":"2024-04-21T08:43:02.756601+0000","last_clean":"2024-04-21T08:43:02.756601+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:43:02.756601+0000","last_undegraded":"2024-04-21T08:43:02.756601+0000","last_fullsized":"2024-04-21T08:43:02.756601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675166+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:43:02.675166+0000","last_peered":"2024-04-21T08:43:02.675166+0000","last_clean":"2024-04-21T08:43:02.675166+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:43:02.675166+0000","last_undegraded":"2024-04-21T08:43:02.675166+0000","last_fullsized":"2024-04-21T08:43:02.675166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756655+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:43:02.756655+0000","last_peered":"2024-04-21T08:43:02.756655+0000","last_clean":"2024-04-21T08:43:02.756655+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:43:02.756655+0000","last_undegraded":"2024-04-21T08:43:02.756655+0000","last_fullsized":"2024-04-21T08:43:02.756655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755405+0000","last_change":"2024-04-21T08:42:57.403514+0000","last_active":"2024-04-21T08:43:02.755405+0000","last_peered":"2024-04-21T08:43:02.755405+0000","last_clean":"2024-04-21T08:43:02.755405+0000","last_became_active":"2024-04-21T08:42:57.403233+0000","last_became_peered":"2024-04-21T08:42:57.403233+0000","last_unstale":"2024-04-21T08:43:02.755405+0000","last_undegraded":"2024-04-21T08:43:02.755405+0000","last_fullsized":"2024-04-21T08:43:02.755405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:14:36.542619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756449+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:43:02.756449+0000","last_peered":"2024-04-21T08:43:02.756449+0000","last_clean":"2024-04-21T08:43:02.756449+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:43:02.756449+0000","last_undegraded":"2024-04-21T08:43:02.756449+0000","last_fullsized":"2024-04-21T08:43:02.756449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675575+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:43:02.675575+0000","last_peered":"2024-04-21T08:43:02.675575+0000","last_clean":"2024-04-21T08:43:02.675575+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:43:02.675575+0000","last_undegraded":"2024-04-21T08:43:02.675575+0000","last_fullsized":"2024-04-21T08:43:02.675575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757669+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:43:02.757669+0000","last_peered":"2024-04-21T08:43:02.757669+0000","last_clean":"2024-04-21T08:43:02.757669+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:43:02.757669+0000","last_undegraded":"2024-04-21T08:43:02.757669+0000","last_fullsized":"2024-04-21T08:43:02.757669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757785+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:43:02.757785+0000","last_peered":"2024-04-21T08:43:02.757785+0000","last_clean":"2024-04-21T08:43:02.757785+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:43:02.757785+0000","last_undegraded":"2024-04-21T08:43:02.757785+0000","last_fullsized":"2024-04-21T08:43:02.757785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213717+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:43:03.213717+0000","last_peered":"2024-04-21T08:43:03.213717+0000","last_clean":"2024-04-21T08:43:03.213717+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:43:03.213717+0000","last_undegraded":"2024-04-21T08:43:03.213717+0000","last_fullsized":"2024-04-21T08:43:03.213717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755703+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:43:02.755703+0000","last_peered":"2024-04-21T08:43:02.755703+0000","last_clean":"2024-04-21T08:43:02.755703+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:43:02.755703+0000","last_undegraded":"2024-04-21T08:43:02.755703+0000","last_fullsized":"2024-04-21T08:43:02.755703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755642+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:43:02.755642+0000","last_peered":"2024-04-21T08:43:02.755642+0000","last_clean":"2024-04-21T08:43:02.755642+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:43:02.755642+0000","last_undegraded":"2024-04-21T08:43:02.755642+0000","last_fullsized":"2024-04-21T08:43:02.755642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755783+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:43:02.755783+0000","last_peered":"2024-04-21T08:43:02.755783+0000","last_clean":"2024-04-21T08:43:02.755783+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:43:02.755783+0000","last_undegraded":"2024-04-21T08:43:02.755783+0000","last_fullsized":"2024-04-21T08:43:02.755783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757343+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:43:02.757343+0000","last_peered":"2024-04-21T08:43:02.757343+0000","last_clean":"2024-04-21T08:43:02.757343+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:43:02.757343+0000","last_undegraded":"2024-04-21T08:43:02.757343+0000","last_fullsized":"2024-04-21T08:43:02.757343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676194+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:43:02.676194+0000","last_peered":"2024-04-21T08:43:02.676194+0000","last_clean":"2024-04-21T08:43:02.676194+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:43:02.676194+0000","last_undegraded":"2024-04-21T08:43:02.676194+0000","last_fullsized":"2024-04-21T08:43:02.676194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675786+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:43:02.675786+0000","last_peered":"2024-04-21T08:43:02.675786+0000","last_clean":"2024-04-21T08:43:02.675786+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:43:02.675786+0000","last_undegraded":"2024-04-21T08:43:02.675786+0000","last_fullsized":"2024-04-21T08:43:02.675786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675628+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:43:02.675628+0000","last_peered":"2024-04-21T08:43:02.675628+0000","last_clean":"2024-04-21T08:43:02.675628+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:43:02.675628+0000","last_undegraded":"2024-04-21T08:43:02.675628+0000","last_fullsized":"2024-04-21T08:43:02.675628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676561+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:43:02.676561+0000","last_peered":"2024-04-21T08:43:02.676561+0000","last_clean":"2024-04-21T08:43:02.676561+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:43:02.676561+0000","last_undegraded":"2024-04-21T08:43:02.676561+0000","last_fullsized":"2024-04-21T08:43:02.676561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756792+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:43:02.756792+0000","last_peered":"2024-04-21T08:43:02.756792+0000","last_clean":"2024-04-21T08:43:02.756792+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:43:02.756792+0000","last_undegraded":"2024-04-21T08:43:02.756792+0000","last_fullsized":"2024-04-21T08:43:02.756792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757753+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:43:02.757753+0000","last_peered":"2024-04-21T08:43:02.757753+0000","last_clean":"2024-04-21T08:43:02.757753+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:43:02.757753+0000","last_undegraded":"2024-04-21T08:43:02.757753+0000","last_fullsized":"2024-04-21T08:43:02.757753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757032+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:43:02.757032+0000","last_peered":"2024-04-21T08:43:02.757032+0000","last_clean":"2024-04-21T08:43:02.757032+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:43:02.757032+0000","last_undegraded":"2024-04-21T08:43:02.757032+0000","last_fullsized":"2024-04-21T08:43:02.757032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756721+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:43:02.756721+0000","last_peered":"2024-04-21T08:43:02.756721+0000","last_clean":"2024-04-21T08:43:02.756721+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:43:02.756721+0000","last_undegraded":"2024-04-21T08:43:02.756721+0000","last_fullsized":"2024-04-21T08:43:02.756721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756353+0000","last_change":"2024-04-21T08:43:01.680999+0000","last_active":"2024-04-21T08:43:02.756353+0000","last_peered":"2024-04-21T08:43:02.756353+0000","last_clean":"2024-04-21T08:43:02.756353+0000","last_became_active":"2024-04-21T08:43:01.680116+0000","last_became_peered":"2024-04-21T08:43:01.680116+0000","last_unstale":"2024-04-21T08:43:02.756353+0000","last_undegraded":"2024-04-21T08:43:02.756353+0000","last_fullsized":"2024-04-21T08:43:02.756353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:27.124847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757318+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:43:02.757318+0000","last_peered":"2024-04-21T08:43:02.757318+0000","last_clean":"2024-04-21T08:43:02.757318+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:43:02.757318+0000","last_undegraded":"2024-04-21T08:43:02.757318+0000","last_fullsized":"2024-04-21T08:43:02.757318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756980+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:43:02.756980+0000","last_peered":"2024-04-21T08:43:02.756980+0000","last_clean":"2024-04-21T08:43:02.756980+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:43:02.756980+0000","last_undegraded":"2024-04-21T08:43:02.756980+0000","last_fullsized":"2024-04-21T08:43:02.756980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756886+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:43:02.756886+0000","last_peered":"2024-04-21T08:43:02.756886+0000","last_clean":"2024-04-21T08:43:02.756886+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:43:02.756886+0000","last_undegraded":"2024-04-21T08:43:02.756886+0000","last_fullsized":"2024-04-21T08:43:02.756886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756652+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:43:02.756652+0000","last_peered":"2024-04-21T08:43:02.756652+0000","last_clean":"2024-04-21T08:43:02.756652+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:43:02.756652+0000","last_undegraded":"2024-04-21T08:43:02.756652+0000","last_fullsized":"2024-04-21T08:43:02.756652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713886+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:43:02.713886+0000","last_peered":"2024-04-21T08:43:02.713886+0000","last_clean":"2024-04-21T08:43:02.713886+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:43:02.713886+0000","last_undegraded":"2024-04-21T08:43:02.713886+0000","last_fullsized":"2024-04-21T08:43:02.713886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675896+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:43:02.675896+0000","last_peered":"2024-04-21T08:43:02.675896+0000","last_clean":"2024-04-21T08:43:02.675896+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:43:02.675896+0000","last_undegraded":"2024-04-21T08:43:02.675896+0000","last_fullsized":"2024-04-21T08:43:02.675896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713544+0000","last_change":"2024-04-21T08:42:56.637459+0000","last_active":"2024-04-21T08:43:02.713544+0000","last_peered":"2024-04-21T08:43:02.713544+0000","last_clean":"2024-04-21T08:43:02.713544+0000","last_became_active":"2024-04-21T08:42:56.637129+0000","last_became_peered":"2024-04-21T08:42:56.637129+0000","last_unstale":"2024-04-21T08:43:02.713544+0000","last_undegraded":"2024-04-21T08:43:02.713544+0000","last_fullsized":"2024-04-21T08:43:02.713544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:25:19.406774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756655+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:43:02.756655+0000","last_peered":"2024-04-21T08:43:02.756655+0000","last_clean":"2024-04-21T08:43:02.756655+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:43:02.756655+0000","last_undegraded":"2024-04-21T08:43:02.756655+0000","last_fullsized":"2024-04-21T08:43:02.756655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675765+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:43:02.675765+0000","last_peered":"2024-04-21T08:43:02.675765+0000","last_clean":"2024-04-21T08:43:02.675765+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:43:02.675765+0000","last_undegraded":"2024-04-21T08:43:02.675765+0000","last_fullsized":"2024-04-21T08:43:02.675765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675113+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:43:02.675113+0000","last_peered":"2024-04-21T08:43:02.675113+0000","last_clean":"2024-04-21T08:43:02.675113+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:43:02.675113+0000","last_undegraded":"2024-04-21T08:43:02.675113+0000","last_fullsized":"2024-04-21T08:43:02.675113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713725+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:43:02.713725+0000","last_peered":"2024-04-21T08:43:02.713725+0000","last_clean":"2024-04-21T08:43:02.713725+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:43:02.713725+0000","last_undegraded":"2024-04-21T08:43:02.713725+0000","last_fullsized":"2024-04-21T08:43:02.713725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757327+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:43:02.757327+0000","last_peered":"2024-04-21T08:43:02.757327+0000","last_clean":"2024-04-21T08:43:02.757327+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:43:02.757327+0000","last_undegraded":"2024-04-21T08:43:02.757327+0000","last_fullsized":"2024-04-21T08:43:02.757327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213638+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:43:03.213638+0000","last_peered":"2024-04-21T08:43:03.213638+0000","last_clean":"2024-04-21T08:43:03.213638+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:43:03.213638+0000","last_undegraded":"2024-04-21T08:43:03.213638+0000","last_fullsized":"2024-04-21T08:43:03.213638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755691+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:43:02.755691+0000","last_peered":"2024-04-21T08:43:02.755691+0000","last_clean":"2024-04-21T08:43:02.755691+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:43:02.755691+0000","last_undegraded":"2024-04-21T08:43:02.755691+0000","last_fullsized":"2024-04-21T08:43:02.755691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756334+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:43:02.756334+0000","last_peered":"2024-04-21T08:43:02.756334+0000","last_clean":"2024-04-21T08:43:02.756334+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:43:02.756334+0000","last_undegraded":"2024-04-21T08:43:02.756334+0000","last_fullsized":"2024-04-21T08:43:02.756334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713247+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:43:02.713247+0000","last_peered":"2024-04-21T08:43:02.713247+0000","last_clean":"2024-04-21T08:43:02.713247+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:43:02.713247+0000","last_undegraded":"2024-04-21T08:43:02.713247+0000","last_fullsized":"2024-04-21T08:43:02.713247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675751+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:43:02.675751+0000","last_peered":"2024-04-21T08:43:02.675751+0000","last_clean":"2024-04-21T08:43:02.675751+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:43:02.675751+0000","last_undegraded":"2024-04-21T08:43:02.675751+0000","last_fullsized":"2024-04-21T08:43:02.675751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756155+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:43:02.756155+0000","last_peered":"2024-04-21T08:43:02.756155+0000","last_clean":"2024-04-21T08:43:02.756155+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:43:02.756155+0000","last_undegraded":"2024-04-21T08:43:02.756155+0000","last_fullsized":"2024-04-21T08:43:02.756155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675504+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:43:02.675504+0000","last_peered":"2024-04-21T08:43:02.675504+0000","last_clean":"2024-04-21T08:43:02.675504+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:43:02.675504+0000","last_undegraded":"2024-04-21T08:43:02.675504+0000","last_fullsized":"2024-04-21T08:43:02.675504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756741+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:43:02.756741+0000","last_peered":"2024-04-21T08:43:02.756741+0000","last_clean":"2024-04-21T08:43:02.756741+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:43:02.756741+0000","last_undegraded":"2024-04-21T08:43:02.756741+0000","last_fullsized":"2024-04-21T08:43:02.756741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675643+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:43:02.675643+0000","last_peered":"2024-04-21T08:43:02.675643+0000","last_clean":"2024-04-21T08:43:02.675643+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:43:02.675643+0000","last_undegraded":"2024-04-21T08:43:02.675643+0000","last_fullsized":"2024-04-21T08:43:02.675643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757556+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:43:02.757556+0000","last_peered":"2024-04-21T08:43:02.757556+0000","last_clean":"2024-04-21T08:43:02.757556+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:43:02.757556+0000","last_undegraded":"2024-04-21T08:43:02.757556+0000","last_fullsized":"2024-04-21T08:43:02.757556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756359+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:43:02.756359+0000","last_peered":"2024-04-21T08:43:02.756359+0000","last_clean":"2024-04-21T08:43:02.756359+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:43:02.756359+0000","last_undegraded":"2024-04-21T08:43:02.756359+0000","last_fullsized":"2024-04-21T08:43:02.756359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756766+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:43:02.756766+0000","last_peered":"2024-04-21T08:43:02.756766+0000","last_clean":"2024-04-21T08:43:02.756766+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:43:02.756766+0000","last_undegraded":"2024-04-21T08:43:02.756766+0000","last_fullsized":"2024-04-21T08:43:02.756766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713831+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:43:02.713831+0000","last_peered":"2024-04-21T08:43:02.713831+0000","last_clean":"2024-04-21T08:43:02.713831+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:43:02.713831+0000","last_undegraded":"2024-04-21T08:43:02.713831+0000","last_fullsized":"2024-04-21T08:43:02.713831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757158+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:43:02.757158+0000","last_peered":"2024-04-21T08:43:02.757158+0000","last_clean":"2024-04-21T08:43:02.757158+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:43:02.757158+0000","last_undegraded":"2024-04-21T08:43:02.757158+0000","last_fullsized":"2024-04-21T08:43:02.757158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675901+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:43:02.675901+0000","last_peered":"2024-04-21T08:43:02.675901+0000","last_clean":"2024-04-21T08:43:02.675901+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:43:02.675901+0000","last_undegraded":"2024-04-21T08:43:02.675901+0000","last_fullsized":"2024-04-21T08:43:02.675901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:13.378354+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:43:13.378354+0000","last_peered":"2024-04-21T08:43:13.378354+0000","last_clean":"2024-04-21T08:43:13.378354+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:43:13.378354+0000","last_undegraded":"2024-04-21T08:43:13.378354+0000","last_fullsized":"2024-04-21T08:43:13.378354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757170+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:43:02.757170+0000","last_peered":"2024-04-21T08:43:02.757170+0000","last_clean":"2024-04-21T08:43:02.757170+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:43:02.757170+0000","last_undegraded":"2024-04-21T08:43:02.757170+0000","last_fullsized":"2024-04-21T08:43:02.757170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.714241+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:43:02.714241+0000","last_peered":"2024-04-21T08:43:02.714241+0000","last_clean":"2024-04-21T08:43:02.714241+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:43:02.714241+0000","last_undegraded":"2024-04-21T08:43:02.714241+0000","last_fullsized":"2024-04-21T08:43:02.714241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675645+0000","last_change":"2024-04-21T08:42:56.630433+0000","last_active":"2024-04-21T08:43:02.675645+0000","last_peered":"2024-04-21T08:43:02.675645+0000","last_clean":"2024-04-21T08:43:02.675645+0000","last_became_active":"2024-04-21T08:42:56.630123+0000","last_became_peered":"2024-04-21T08:42:56.630123+0000","last_unstale":"2024-04-21T08:43:02.675645+0000","last_undegraded":"2024-04-21T08:43:02.675645+0000","last_fullsized":"2024-04-21T08:43:02.675645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:42.292961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213717+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:43:03.213717+0000","last_peered":"2024-04-21T08:43:03.213717+0000","last_clean":"2024-04-21T08:43:03.213717+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:43:03.213717+0000","last_undegraded":"2024-04-21T08:43:03.213717+0000","last_fullsized":"2024-04-21T08:43:03.213717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756643+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:43:02.756643+0000","last_peered":"2024-04-21T08:43:02.756643+0000","last_clean":"2024-04-21T08:43:02.756643+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:43:02.756643+0000","last_undegraded":"2024-04-21T08:43:02.756643+0000","last_fullsized":"2024-04-21T08:43:02.756643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713546+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:43:02.713546+0000","last_peered":"2024-04-21T08:43:02.713546+0000","last_clean":"2024-04-21T08:43:02.713546+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:43:02.713546+0000","last_undegraded":"2024-04-21T08:43:02.713546+0000","last_fullsized":"2024-04-21T08:43:02.713546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757291+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:43:02.757291+0000","last_peered":"2024-04-21T08:43:02.757291+0000","last_clean":"2024-04-21T08:43:02.757291+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:43:02.757291+0000","last_undegraded":"2024-04-21T08:43:02.757291+0000","last_fullsized":"2024-04-21T08:43:02.757291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676462+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:43:02.676462+0000","last_peered":"2024-04-21T08:43:02.676462+0000","last_clean":"2024-04-21T08:43:02.676462+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:43:02.676462+0000","last_undegraded":"2024-04-21T08:43:02.676462+0000","last_fullsized":"2024-04-21T08:43:02.676462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756528+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:43:02.756528+0000","last_peered":"2024-04-21T08:43:02.756528+0000","last_clean":"2024-04-21T08:43:02.756528+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:43:02.756528+0000","last_undegraded":"2024-04-21T08:43:02.756528+0000","last_fullsized":"2024-04-21T08:43:02.756528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756200+0000","last_change":"2024-04-21T08:42:57.405704+0000","last_active":"2024-04-21T08:43:02.756200+0000","last_peered":"2024-04-21T08:43:02.756200+0000","last_clean":"2024-04-21T08:43:02.756200+0000","last_became_active":"2024-04-21T08:42:57.403829+0000","last_became_peered":"2024-04-21T08:42:57.403829+0000","last_unstale":"2024-04-21T08:43:02.756200+0000","last_undegraded":"2024-04-21T08:43:02.756200+0000","last_fullsized":"2024-04-21T08:43:02.756200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:34:01.082283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214184+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:43:03.214184+0000","last_peered":"2024-04-21T08:43:03.214184+0000","last_clean":"2024-04-21T08:43:03.214184+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:43:03.214184+0000","last_undegraded":"2024-04-21T08:43:03.214184+0000","last_fullsized":"2024-04-21T08:43:03.214184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756330+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:43:02.756330+0000","last_peered":"2024-04-21T08:43:02.756330+0000","last_clean":"2024-04-21T08:43:02.756330+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:43:02.756330+0000","last_undegraded":"2024-04-21T08:43:02.756330+0000","last_fullsized":"2024-04-21T08:43:02.756330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713508+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:43:02.713508+0000","last_peered":"2024-04-21T08:43:02.713508+0000","last_clean":"2024-04-21T08:43:02.713508+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:43:02.713508+0000","last_undegraded":"2024-04-21T08:43:02.713508+0000","last_fullsized":"2024-04-21T08:43:02.713508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713586+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:43:02.713586+0000","last_peered":"2024-04-21T08:43:02.713586+0000","last_clean":"2024-04-21T08:43:02.713586+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:43:02.713586+0000","last_undegraded":"2024-04-21T08:43:02.713586+0000","last_fullsized":"2024-04-21T08:43:02.713586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757157+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:43:02.757157+0000","last_peered":"2024-04-21T08:43:02.757157+0000","last_clean":"2024-04-21T08:43:02.757157+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:43:02.757157+0000","last_undegraded":"2024-04-21T08:43:02.757157+0000","last_fullsized":"2024-04-21T08:43:02.757157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757181+0000","last_change":"2024-04-21T08:42:57.411284+0000","last_active":"2024-04-21T08:43:02.757181+0000","last_peered":"2024-04-21T08:43:02.757181+0000","last_clean":"2024-04-21T08:43:02.757181+0000","last_became_active":"2024-04-21T08:42:57.409361+0000","last_became_peered":"2024-04-21T08:42:57.409361+0000","last_unstale":"2024-04-21T08:43:02.757181+0000","last_undegraded":"2024-04-21T08:43:02.757181+0000","last_fullsized":"2024-04-21T08:43:02.757181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:55.766263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756629+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:43:02.756629+0000","last_peered":"2024-04-21T08:43:02.756629+0000","last_clean":"2024-04-21T08:43:02.756629+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:43:02.756629+0000","last_undegraded":"2024-04-21T08:43:02.756629+0000","last_fullsized":"2024-04-21T08:43:02.756629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675866+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:43:02.675866+0000","last_peered":"2024-04-21T08:43:02.675866+0000","last_clean":"2024-04-21T08:43:02.675866+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:43:02.675866+0000","last_undegraded":"2024-04-21T08:43:02.675866+0000","last_fullsized":"2024-04-21T08:43:02.675866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675472+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:43:02.675472+0000","last_peered":"2024-04-21T08:43:02.675472+0000","last_clean":"2024-04-21T08:43:02.675472+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:43:02.675472+0000","last_undegraded":"2024-04-21T08:43:02.675472+0000","last_fullsized":"2024-04-21T08:43:02.675472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213959+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:43:03.213959+0000","last_peered":"2024-04-21T08:43:03.213959+0000","last_clean":"2024-04-21T08:43:03.213959+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:43:03.213959+0000","last_undegraded":"2024-04-21T08:43:03.213959+0000","last_fullsized":"2024-04-21T08:43:03.213959+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756100+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:43:02.756100+0000","last_peered":"2024-04-21T08:43:02.756100+0000","last_clean":"2024-04-21T08:43:02.756100+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:43:02.756100+0000","last_undegraded":"2024-04-21T08:43:02.756100+0000","last_fullsized":"2024-04-21T08:43:02.756100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675109+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:43:02.675109+0000","last_peered":"2024-04-21T08:43:02.675109+0000","last_clean":"2024-04-21T08:43:02.675109+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:43:02.675109+0000","last_undegraded":"2024-04-21T08:43:02.675109+0000","last_fullsized":"2024-04-21T08:43:02.675109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214106+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:43:03.214106+0000","last_peered":"2024-04-21T08:43:03.214106+0000","last_clean":"2024-04-21T08:43:03.214106+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:43:03.214106+0000","last_undegraded":"2024-04-21T08:43:03.214106+0000","last_fullsized":"2024-04-21T08:43:03.214106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713429+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:43:02.713429+0000","last_peered":"2024-04-21T08:43:02.713429+0000","last_clean":"2024-04-21T08:43:02.713429+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:43:02.713429+0000","last_undegraded":"2024-04-21T08:43:02.713429+0000","last_fullsized":"2024-04-21T08:43:02.713429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213493+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:43:03.213493+0000","last_peered":"2024-04-21T08:43:03.213493+0000","last_clean":"2024-04-21T08:43:03.213493+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:43:03.213493+0000","last_undegraded":"2024-04-21T08:43:03.213493+0000","last_fullsized":"2024-04-21T08:43:03.213493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757465+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:43:02.757465+0000","last_peered":"2024-04-21T08:43:02.757465+0000","last_clean":"2024-04-21T08:43:02.757465+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:43:02.757465+0000","last_undegraded":"2024-04-21T08:43:02.757465+0000","last_fullsized":"2024-04-21T08:43:02.757465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756498+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:43:02.756498+0000","last_peered":"2024-04-21T08:43:02.756498+0000","last_clean":"2024-04-21T08:43:02.756498+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:43:02.756498+0000","last_undegraded":"2024-04-21T08:43:02.756498+0000","last_fullsized":"2024-04-21T08:43:02.756498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756425+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:43:02.756425+0000","last_peered":"2024-04-21T08:43:02.756425+0000","last_clean":"2024-04-21T08:43:02.756425+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:43:02.756425+0000","last_undegraded":"2024-04-21T08:43:02.756425+0000","last_fullsized":"2024-04-21T08:43:02.756425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756997+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:43:02.756997+0000","last_peered":"2024-04-21T08:43:02.756997+0000","last_clean":"2024-04-21T08:43:02.756997+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:43:02.756997+0000","last_undegraded":"2024-04-21T08:43:02.756997+0000","last_fullsized":"2024-04-21T08:43:02.756997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.714100+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:43:02.714100+0000","last_peered":"2024-04-21T08:43:02.714100+0000","last_clean":"2024-04-21T08:43:02.714100+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:43:02.714100+0000","last_undegraded":"2024-04-21T08:43:02.714100+0000","last_fullsized":"2024-04-21T08:43:02.714100+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213490+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:43:03.213490+0000","last_peered":"2024-04-21T08:43:03.213490+0000","last_clean":"2024-04-21T08:43:03.213490+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:43:03.213490+0000","last_undegraded":"2024-04-21T08:43:03.213490+0000","last_fullsized":"2024-04-21T08:43:03.213490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756080+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:43:02.756080+0000","last_peered":"2024-04-21T08:43:02.756080+0000","last_clean":"2024-04-21T08:43:02.756080+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:43:02.756080+0000","last_undegraded":"2024-04-21T08:43:02.756080+0000","last_fullsized":"2024-04-21T08:43:02.756080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756567+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:43:02.756567+0000","last_peered":"2024-04-21T08:43:02.756567+0000","last_clean":"2024-04-21T08:43:02.756567+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:43:02.756567+0000","last_undegraded":"2024-04-21T08:43:02.756567+0000","last_fullsized":"2024-04-21T08:43:02.756567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757185+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:43:02.757185+0000","last_peered":"2024-04-21T08:43:02.757185+0000","last_clean":"2024-04-21T08:43:02.757185+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:43:02.757185+0000","last_undegraded":"2024-04-21T08:43:02.757185+0000","last_fullsized":"2024-04-21T08:43:02.757185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214056+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:43:03.214056+0000","last_peered":"2024-04-21T08:43:03.214056+0000","last_clean":"2024-04-21T08:43:03.214056+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:43:03.214056+0000","last_undegraded":"2024-04-21T08:43:03.214056+0000","last_fullsized":"2024-04-21T08:43:03.214056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755671+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:43:02.755671+0000","last_peered":"2024-04-21T08:43:02.755671+0000","last_clean":"2024-04-21T08:43:02.755671+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:43:02.755671+0000","last_undegraded":"2024-04-21T08:43:02.755671+0000","last_fullsized":"2024-04-21T08:43:02.755671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676295+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:43:02.676295+0000","last_peered":"2024-04-21T08:43:02.676295+0000","last_clean":"2024-04-21T08:43:02.676295+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:43:02.676295+0000","last_undegraded":"2024-04-21T08:43:02.676295+0000","last_fullsized":"2024-04-21T08:43:02.676295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756792+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:43:02.756792+0000","last_peered":"2024-04-21T08:43:02.756792+0000","last_clean":"2024-04-21T08:43:02.756792+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:43:02.756792+0000","last_undegraded":"2024-04-21T08:43:02.756792+0000","last_fullsized":"2024-04-21T08:43:02.756792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675194+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:43:02.675194+0000","last_peered":"2024-04-21T08:43:02.675194+0000","last_clean":"2024-04-21T08:43:02.675194+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:43:02.675194+0000","last_undegraded":"2024-04-21T08:43:02.675194+0000","last_fullsized":"2024-04-21T08:43:02.675194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756624+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:43:02.756624+0000","last_peered":"2024-04-21T08:43:02.756624+0000","last_clean":"2024-04-21T08:43:02.756624+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:43:02.756624+0000","last_undegraded":"2024-04-21T08:43:02.756624+0000","last_fullsized":"2024-04-21T08:43:02.756624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756375+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:43:02.756375+0000","last_peered":"2024-04-21T08:43:02.756375+0000","last_clean":"2024-04-21T08:43:02.756375+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:43:02.756375+0000","last_undegraded":"2024-04-21T08:43:02.756375+0000","last_fullsized":"2024-04-21T08:43:02.756375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756930+0000","last_change":"2024-04-21T08:43:01.720615+0000","last_active":"2024-04-21T08:43:02.756930+0000","last_peered":"2024-04-21T08:43:02.756930+0000","last_clean":"2024-04-21T08:43:02.756930+0000","last_became_active":"2024-04-21T08:43:01.720142+0000","last_became_peered":"2024-04-21T08:43:01.720142+0000","last_unstale":"2024-04-21T08:43:02.756930+0000","last_undegraded":"2024-04-21T08:43:02.756930+0000","last_fullsized":"2024-04-21T08:43:02.756930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:53:44.879493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755912+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:43:02.755912+0000","last_peered":"2024-04-21T08:43:02.755912+0000","last_clean":"2024-04-21T08:43:02.755912+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:43:02.755912+0000","last_undegraded":"2024-04-21T08:43:02.755912+0000","last_fullsized":"2024-04-21T08:43:02.755912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756245+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:43:02.756245+0000","last_peered":"2024-04-21T08:43:02.756245+0000","last_clean":"2024-04-21T08:43:02.756245+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:43:02.756245+0000","last_undegraded":"2024-04-21T08:43:02.756245+0000","last_fullsized":"2024-04-21T08:43:02.756245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676489+0000","last_change":"2024-04-21T08:43:01.686008+0000","last_active":"2024-04-21T08:43:02.676489+0000","last_peered":"2024-04-21T08:43:02.676489+0000","last_clean":"2024-04-21T08:43:02.676489+0000","last_became_active":"2024-04-21T08:43:01.685442+0000","last_became_peered":"2024-04-21T08:43:01.685442+0000","last_unstale":"2024-04-21T08:43:02.676489+0000","last_undegraded":"2024-04-21T08:43:02.676489+0000","last_fullsized":"2024-04-21T08:43:02.676489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:06.000166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756097+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:43:02.756097+0000","last_peered":"2024-04-21T08:43:02.756097+0000","last_clean":"2024-04-21T08:43:02.756097+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:43:02.756097+0000","last_undegraded":"2024-04-21T08:43:02.756097+0000","last_fullsized":"2024-04-21T08:43:02.756097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756161+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:43:02.756161+0000","last_peered":"2024-04-21T08:43:02.756161+0000","last_clean":"2024-04-21T08:43:02.756161+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:43:02.756161+0000","last_undegraded":"2024-04-21T08:43:02.756161+0000","last_fullsized":"2024-04-21T08:43:02.756161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757635+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:43:02.757635+0000","last_peered":"2024-04-21T08:43:02.757635+0000","last_clean":"2024-04-21T08:43:02.757635+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:43:02.757635+0000","last_undegraded":"2024-04-21T08:43:02.757635+0000","last_fullsized":"2024-04-21T08:43:02.757635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675262+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:43:02.675262+0000","last_peered":"2024-04-21T08:43:02.675262+0000","last_clean":"2024-04-21T08:43:02.675262+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:43:02.675262+0000","last_undegraded":"2024-04-21T08:43:02.675262+0000","last_fullsized":"2024-04-21T08:43:02.675262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756140+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:43:02.756140+0000","last_peered":"2024-04-21T08:43:02.756140+0000","last_clean":"2024-04-21T08:43:02.756140+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:43:02.756140+0000","last_undegraded":"2024-04-21T08:43:02.756140+0000","last_fullsized":"2024-04-21T08:43:02.756140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757346+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:43:02.757346+0000","last_peered":"2024-04-21T08:43:02.757346+0000","last_clean":"2024-04-21T08:43:02.757346+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:43:02.757346+0000","last_undegraded":"2024-04-21T08:43:02.757346+0000","last_fullsized":"2024-04-21T08:43:02.757346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756962+0000","last_change":"2024-04-21T08:42:56.639464+0000","last_active":"2024-04-21T08:43:02.756962+0000","last_peered":"2024-04-21T08:43:02.756962+0000","last_clean":"2024-04-21T08:43:02.756962+0000","last_became_active":"2024-04-21T08:42:56.639142+0000","last_became_peered":"2024-04-21T08:42:56.639142+0000","last_unstale":"2024-04-21T08:43:02.756962+0000","last_undegraded":"2024-04-21T08:43:02.756962+0000","last_fullsized":"2024-04-21T08:43:02.756962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:55.703492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756282+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:43:02.756282+0000","last_peered":"2024-04-21T08:43:02.756282+0000","last_clean":"2024-04-21T08:43:02.756282+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:43:02.756282+0000","last_undegraded":"2024-04-21T08:43:02.756282+0000","last_fullsized":"2024-04-21T08:43:02.756282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756224+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:43:02.756224+0000","last_peered":"2024-04-21T08:43:02.756224+0000","last_clean":"2024-04-21T08:43:02.756224+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:43:02.756224+0000","last_undegraded":"2024-04-21T08:43:02.756224+0000","last_fullsized":"2024-04-21T08:43:02.756224+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675696+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:43:02.675696+0000","last_peered":"2024-04-21T08:43:02.675696+0000","last_clean":"2024-04-21T08:43:02.675696+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:43:02.675696+0000","last_undegraded":"2024-04-21T08:43:02.675696+0000","last_fullsized":"2024-04-21T08:43:02.675696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757776+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:43:02.757776+0000","last_peered":"2024-04-21T08:43:02.757776+0000","last_clean":"2024-04-21T08:43:02.757776+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:43:02.757776+0000","last_undegraded":"2024-04-21T08:43:02.757776+0000","last_fullsized":"2024-04-21T08:43:02.757776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755906+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:43:02.755906+0000","last_peered":"2024-04-21T08:43:02.755906+0000","last_clean":"2024-04-21T08:43:02.755906+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:43:02.755906+0000","last_undegraded":"2024-04-21T08:43:02.755906+0000","last_fullsized":"2024-04-21T08:43:02.755906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713932+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:43:02.713932+0000","last_peered":"2024-04-21T08:43:02.713932+0000","last_clean":"2024-04-21T08:43:02.713932+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:43:02.713932+0000","last_undegraded":"2024-04-21T08:43:02.713932+0000","last_fullsized":"2024-04-21T08:43:02.713932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757532+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:43:02.757532+0000","last_peered":"2024-04-21T08:43:02.757532+0000","last_clean":"2024-04-21T08:43:02.757532+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:43:02.757532+0000","last_undegraded":"2024-04-21T08:43:02.757532+0000","last_fullsized":"2024-04-21T08:43:02.757532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713351+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:43:02.713351+0000","last_peered":"2024-04-21T08:43:02.713351+0000","last_clean":"2024-04-21T08:43:02.713351+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:43:02.713351+0000","last_undegraded":"2024-04-21T08:43:02.713351+0000","last_fullsized":"2024-04-21T08:43:02.713351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214115+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:43:03.214115+0000","last_peered":"2024-04-21T08:43:03.214115+0000","last_clean":"2024-04-21T08:43:03.214115+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:43:03.214115+0000","last_undegraded":"2024-04-21T08:43:03.214115+0000","last_fullsized":"2024-04-21T08:43:03.214115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757671+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:43:02.757671+0000","last_peered":"2024-04-21T08:43:02.757671+0000","last_clean":"2024-04-21T08:43:02.757671+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:43:02.757671+0000","last_undegraded":"2024-04-21T08:43:02.757671+0000","last_fullsized":"2024-04-21T08:43:02.757671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756238+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:43:02.756238+0000","last_peered":"2024-04-21T08:43:02.756238+0000","last_clean":"2024-04-21T08:43:02.756238+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:43:02.756238+0000","last_undegraded":"2024-04-21T08:43:02.756238+0000","last_fullsized":"2024-04-21T08:43:02.756238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675590+0000","last_change":"2024-04-21T08:42:57.408595+0000","last_active":"2024-04-21T08:43:02.675590+0000","last_peered":"2024-04-21T08:43:02.675590+0000","last_clean":"2024-04-21T08:43:02.675590+0000","last_became_active":"2024-04-21T08:42:57.408193+0000","last_became_peered":"2024-04-21T08:42:57.408193+0000","last_unstale":"2024-04-21T08:43:02.675590+0000","last_undegraded":"2024-04-21T08:43:02.675590+0000","last_fullsized":"2024-04-21T08:43:02.675590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:04:57.559111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675169+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:43:02.675169+0000","last_peered":"2024-04-21T08:43:02.675169+0000","last_clean":"2024-04-21T08:43:02.675169+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:43:02.675169+0000","last_undegraded":"2024-04-21T08:43:02.675169+0000","last_fullsized":"2024-04-21T08:43:02.675169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755971+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:43:02.755971+0000","last_peered":"2024-04-21T08:43:02.755971+0000","last_clean":"2024-04-21T08:43:02.755971+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:43:02.755971+0000","last_undegraded":"2024-04-21T08:43:02.755971+0000","last_fullsized":"2024-04-21T08:43:02.755971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213944+0000","last_change":"2024-04-21T08:42:57.408413+0000","last_active":"2024-04-21T08:43:03.213944+0000","last_peered":"2024-04-21T08:43:03.213944+0000","last_clean":"2024-04-21T08:43:03.213944+0000","last_became_active":"2024-04-21T08:42:57.408039+0000","last_became_peered":"2024-04-21T08:42:57.408039+0000","last_unstale":"2024-04-21T08:43:03.213944+0000","last_undegraded":"2024-04-21T08:43:03.213944+0000","last_fullsized":"2024-04-21T08:43:03.213944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:13.072822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756995+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:43:02.756995+0000","last_peered":"2024-04-21T08:43:02.756995+0000","last_clean":"2024-04-21T08:43:02.756995+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:43:02.756995+0000","last_undegraded":"2024-04-21T08:43:02.756995+0000","last_fullsized":"2024-04-21T08:43:02.756995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675985+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:43:02.675985+0000","last_peered":"2024-04-21T08:43:02.675985+0000","last_clean":"2024-04-21T08:43:02.675985+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:43:02.675985+0000","last_undegraded":"2024-04-21T08:43:02.675985+0000","last_fullsized":"2024-04-21T08:43:02.675985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756918+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:43:02.756918+0000","last_peered":"2024-04-21T08:43:02.756918+0000","last_clean":"2024-04-21T08:43:02.756918+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:43:02.756918+0000","last_undegraded":"2024-04-21T08:43:02.756918+0000","last_fullsized":"2024-04-21T08:43:02.756918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713373+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:43:02.713373+0000","last_peered":"2024-04-21T08:43:02.713373+0000","last_clean":"2024-04-21T08:43:02.713373+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:43:02.713373+0000","last_undegraded":"2024-04-21T08:43:02.713373+0000","last_fullsized":"2024-04-21T08:43:02.713373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756236+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:43:02.756236+0000","last_peered":"2024-04-21T08:43:02.756236+0000","last_clean":"2024-04-21T08:43:02.756236+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:43:02.756236+0000","last_undegraded":"2024-04-21T08:43:02.756236+0000","last_fullsized":"2024-04-21T08:43:02.756236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756593+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:43:02.756593+0000","last_peered":"2024-04-21T08:43:02.756593+0000","last_clean":"2024-04-21T08:43:02.756593+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:43:02.756593+0000","last_undegraded":"2024-04-21T08:43:02.756593+0000","last_fullsized":"2024-04-21T08:43:02.756593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213800+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:43:03.213800+0000","last_peered":"2024-04-21T08:43:03.213800+0000","last_clean":"2024-04-21T08:43:03.213800+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:43:03.213800+0000","last_undegraded":"2024-04-21T08:43:03.213800+0000","last_fullsized":"2024-04-21T08:43:03.213800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756713+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:43:02.756713+0000","last_peered":"2024-04-21T08:43:02.756713+0000","last_clean":"2024-04-21T08:43:02.756713+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:43:02.756713+0000","last_undegraded":"2024-04-21T08:43:02.756713+0000","last_fullsized":"2024-04-21T08:43:02.756713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":251796,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":50549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":51360,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":51360,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":500390,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":51360,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":40505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:43:25.623 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:43:25.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 -- 172.21.15.57:0/2344506208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 msgr2=0x7fc564104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 --2- 172.21.15.57:0/2344506208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564104de0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc554009ca0 tx=0x7fc55402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 -- 172.21.15.57:0/2344506208 shutdown_connections 2024-04-21T08:43:25.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 --2- 172.21.15.57:0/2344506208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 0x7fc56410a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 --2- 172.21.15.57:0/2344506208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc564105320 0x7fc564107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 --2- 172.21.15.57:0/2344506208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564104de0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.760+0000 7fc569775640 1 -- 172.21.15.57:0/2344506208 >> 172.21.15.57:0/2344506208 conn(0x7fc5640fc3e0 msgr2=0x7fc5640fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 -- 172.21.15.57:0/2344506208 shutdown_connections 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 -- 172.21.15.57:0/2344506208 wait complete. 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 Processor -- start 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 -- start start 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc564105320 0x7fc564198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 0x7fc56419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc56410db10 con 0x7fc5641029f0 2024-04-21T08:43:25.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc56410d990 con 0x7fc564107c50 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.761+0000 7fc569775640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc56410dc90 con 0x7fc564105320 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35830/0 (socket says 172.21.15.57:35830) 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 -- 172.21.15.57:0/847951813 learned_addr learned my addr 172.21.15.57:0/847951813 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc5637fe640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 0x7fc56419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc5627fc640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc564105320 0x7fc564198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 -- 172.21.15.57:0/847951813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc564105320 msgr2=0x7fc564198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc564105320 0x7fc564198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 -- 172.21.15.57:0/847951813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 msgr2=0x7fc56419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 0x7fc56419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 -- 172.21.15.57:0/847951813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56419fe40 con 0x7fc5641029f0 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc5637fe640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 0x7fc56419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc562ffd640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564198220 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc554005e30 tx=0x7fc554034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554039040 con 0x7fc5641029f0 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5641a00d0 con 0x7fc5641029f0 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5540325c0 con 0x7fc5641029f0 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5540042a0 con 0x7fc5641029f0 2024-04-21T08:43:25.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.762+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5641a0590 con 0x7fc5641029f0 2024-04-21T08:43:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.763+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5640fded0 con 0x7fc5641029f0 2024-04-21T08:43:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.764+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc554046030 con 0x7fc5641029f0 2024-04-21T08:43:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.765+0000 7fc54bfff640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc54c0a3c40 0x7fc54c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.765+0000 7fc5627fc640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc54c0a3c40 0x7fc54c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.765+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc5540f21a0 con 0x7fc5641029f0 2024-04-21T08:43:25.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.766+0000 7fc5627fc640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc54c0a3c40 0x7fc54c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc564199730 tx=0x7fc558008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:25.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.770+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55404d060 con 0x7fc5641029f0 2024-04-21T08:43:25.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.954+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc564062390 con 0x7fc54c0a3c40 2024-04-21T08:43:25.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.963+0000 7fc54bfff640 1 -- 172.21.15.57:0/847951813 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340444 (secure 0 0 0) 0x7fc564062390 con 0x7fc54c0a3c40 2024-04-21T08:43:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:25.969 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc54c0a3c40 msgr2=0x7fc54c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc54c0a3c40 0x7fc54c0a6100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc564199730 tx=0x7fc558008040 comp rx=0 tx=0).stop 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 msgr2=0x7fc564198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564198220 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc554005e30 tx=0x7fc554034760 comp rx=0 tx=0).stop 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 shutdown_connections 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc54c0a3c40 0x7fc54c0a6100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc564107c50 0x7fc56419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc564105320 0x7fc564198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.972 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 --2- 172.21.15.57:0/847951813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5641029f0 0x7fc564198220 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:25.972 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 >> 172.21.15.57:0/847951813 conn(0x7fc5640fc3e0 msgr2=0x7fc5641034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:25.972 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 shutdown_connections 2024-04-21T08:43:25.972 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:25.967+0000 7fc569775640 1 -- 172.21.15.57:0/847951813 wait complete. 2024-04-21T08:43:25.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:25 smithi057 ceph-mon[27968]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:25.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:25 smithi057 ceph-mon[27968]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-04-21T08:43:25.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:25 smithi057 ceph-mon[27968]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:43:25.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2900243190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:25.977 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3356076539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:25.978 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-04-21T08:43:24.610381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":140,"num_read_kb":123,"num_write":230,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":180,"ondisk_log_size":180,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360768,"kb_used_data":11540,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581056,"statfs":{"total":1151940427776,"available":1151571001344,"internally_reserved":0,"allocated":11816960,"data_stored":7530903,"data_compressed":286272,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757483+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:43:02.757483+0000","last_peered":"2024-04-21T08:43:02.757483+0000","last_clean":"2024-04-21T08:43:02.757483+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:43:02.757483+0000","last_undegraded":"2024-04-21T08:43:02.757483+0000","last_fullsized":"2024-04-21T08:43:02.757483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756273+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:43:02.756273+0000","last_peered":"2024-04-21T08:43:02.756273+0000","last_clean":"2024-04-21T08:43:02.756273+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:43:02.756273+0000","last_undegraded":"2024-04-21T08:43:02.756273+0000","last_fullsized":"2024-04-21T08:43:02.756273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756018+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:43:02.756018+0000","last_peered":"2024-04-21T08:43:02.756018+0000","last_clean":"2024-04-21T08:43:02.756018+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:43:02.756018+0000","last_undegraded":"2024-04-21T08:43:02.756018+0000","last_fullsized":"2024-04-21T08:43:02.756018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757831+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:43:02.757831+0000","last_peered":"2024-04-21T08:43:02.757831+0000","last_clean":"2024-04-21T08:43:02.757831+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:43:02.757831+0000","last_undegraded":"2024-04-21T08:43:02.757831+0000","last_fullsized":"2024-04-21T08:43:02.757831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756601+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:43:02.756601+0000","last_peered":"2024-04-21T08:43:02.756601+0000","last_clean":"2024-04-21T08:43:02.756601+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:43:02.756601+0000","last_undegraded":"2024-04-21T08:43:02.756601+0000","last_fullsized":"2024-04-21T08:43:02.756601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675166+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:43:02.675166+0000","last_peered":"2024-04-21T08:43:02.675166+0000","last_clean":"2024-04-21T08:43:02.675166+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:43:02.675166+0000","last_undegraded":"2024-04-21T08:43:02.675166+0000","last_fullsized":"2024-04-21T08:43:02.675166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756655+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:43:02.756655+0000","last_peered":"2024-04-21T08:43:02.756655+0000","last_clean":"2024-04-21T08:43:02.756655+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:43:02.756655+0000","last_undegraded":"2024-04-21T08:43:02.756655+0000","last_fullsized":"2024-04-21T08:43:02.756655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755405+0000","last_change":"2024-04-21T08:42:57.403514+0000","last_active":"2024-04-21T08:43:02.755405+0000","last_peered":"2024-04-21T08:43:02.755405+0000","last_clean":"2024-04-21T08:43:02.755405+0000","last_became_active":"2024-04-21T08:42:57.403233+0000","last_became_peered":"2024-04-21T08:42:57.403233+0000","last_unstale":"2024-04-21T08:43:02.755405+0000","last_undegraded":"2024-04-21T08:43:02.755405+0000","last_fullsized":"2024-04-21T08:43:02.755405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:14:36.542619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756449+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:43:02.756449+0000","last_peered":"2024-04-21T08:43:02.756449+0000","last_clean":"2024-04-21T08:43:02.756449+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:43:02.756449+0000","last_undegraded":"2024-04-21T08:43:02.756449+0000","last_fullsized":"2024-04-21T08:43:02.756449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675575+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:43:02.675575+0000","last_peered":"2024-04-21T08:43:02.675575+0000","last_clean":"2024-04-21T08:43:02.675575+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:43:02.675575+0000","last_undegraded":"2024-04-21T08:43:02.675575+0000","last_fullsized":"2024-04-21T08:43:02.675575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757669+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:43:02.757669+0000","last_peered":"2024-04-21T08:43:02.757669+0000","last_clean":"2024-04-21T08:43:02.757669+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:43:02.757669+0000","last_undegraded":"2024-04-21T08:43:02.757669+0000","last_fullsized":"2024-04-21T08:43:02.757669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757785+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:43:02.757785+0000","last_peered":"2024-04-21T08:43:02.757785+0000","last_clean":"2024-04-21T08:43:02.757785+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:43:02.757785+0000","last_undegraded":"2024-04-21T08:43:02.757785+0000","last_fullsized":"2024-04-21T08:43:02.757785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213717+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:43:03.213717+0000","last_peered":"2024-04-21T08:43:03.213717+0000","last_clean":"2024-04-21T08:43:03.213717+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:43:03.213717+0000","last_undegraded":"2024-04-21T08:43:03.213717+0000","last_fullsized":"2024-04-21T08:43:03.213717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755703+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:43:02.755703+0000","last_peered":"2024-04-21T08:43:02.755703+0000","last_clean":"2024-04-21T08:43:02.755703+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:43:02.755703+0000","last_undegraded":"2024-04-21T08:43:02.755703+0000","last_fullsized":"2024-04-21T08:43:02.755703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755642+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:43:02.755642+0000","last_peered":"2024-04-21T08:43:02.755642+0000","last_clean":"2024-04-21T08:43:02.755642+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:43:02.755642+0000","last_undegraded":"2024-04-21T08:43:02.755642+0000","last_fullsized":"2024-04-21T08:43:02.755642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755783+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:43:02.755783+0000","last_peered":"2024-04-21T08:43:02.755783+0000","last_clean":"2024-04-21T08:43:02.755783+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:43:02.755783+0000","last_undegraded":"2024-04-21T08:43:02.755783+0000","last_fullsized":"2024-04-21T08:43:02.755783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757343+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:43:02.757343+0000","last_peered":"2024-04-21T08:43:02.757343+0000","last_clean":"2024-04-21T08:43:02.757343+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:43:02.757343+0000","last_undegraded":"2024-04-21T08:43:02.757343+0000","last_fullsized":"2024-04-21T08:43:02.757343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676194+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:43:02.676194+0000","last_peered":"2024-04-21T08:43:02.676194+0000","last_clean":"2024-04-21T08:43:02.676194+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:43:02.676194+0000","last_undegraded":"2024-04-21T08:43:02.676194+0000","last_fullsized":"2024-04-21T08:43:02.676194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675786+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:43:02.675786+0000","last_peered":"2024-04-21T08:43:02.675786+0000","last_clean":"2024-04-21T08:43:02.675786+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:43:02.675786+0000","last_undegraded":"2024-04-21T08:43:02.675786+0000","last_fullsized":"2024-04-21T08:43:02.675786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675628+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:43:02.675628+0000","last_peered":"2024-04-21T08:43:02.675628+0000","last_clean":"2024-04-21T08:43:02.675628+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:43:02.675628+0000","last_undegraded":"2024-04-21T08:43:02.675628+0000","last_fullsized":"2024-04-21T08:43:02.675628+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676561+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:43:02.676561+0000","last_peered":"2024-04-21T08:43:02.676561+0000","last_clean":"2024-04-21T08:43:02.676561+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:43:02.676561+0000","last_undegraded":"2024-04-21T08:43:02.676561+0000","last_fullsized":"2024-04-21T08:43:02.676561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756792+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:43:02.756792+0000","last_peered":"2024-04-21T08:43:02.756792+0000","last_clean":"2024-04-21T08:43:02.756792+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:43:02.756792+0000","last_undegraded":"2024-04-21T08:43:02.756792+0000","last_fullsized":"2024-04-21T08:43:02.756792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757753+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:43:02.757753+0000","last_peered":"2024-04-21T08:43:02.757753+0000","last_clean":"2024-04-21T08:43:02.757753+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:43:02.757753+0000","last_undegraded":"2024-04-21T08:43:02.757753+0000","last_fullsized":"2024-04-21T08:43:02.757753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757032+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:43:02.757032+0000","last_peered":"2024-04-21T08:43:02.757032+0000","last_clean":"2024-04-21T08:43:02.757032+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:43:02.757032+0000","last_undegraded":"2024-04-21T08:43:02.757032+0000","last_fullsized":"2024-04-21T08:43:02.757032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756721+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:43:02.756721+0000","last_peered":"2024-04-21T08:43:02.756721+0000","last_clean":"2024-04-21T08:43:02.756721+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:43:02.756721+0000","last_undegraded":"2024-04-21T08:43:02.756721+0000","last_fullsized":"2024-04-21T08:43:02.756721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756353+0000","last_change":"2024-04-21T08:43:01.680999+0000","last_active":"2024-04-21T08:43:02.756353+0000","last_peered":"2024-04-21T08:43:02.756353+0000","last_clean":"2024-04-21T08:43:02.756353+0000","last_became_active":"2024-04-21T08:43:01.680116+0000","last_became_peered":"2024-04-21T08:43:01.680116+0000","last_unstale":"2024-04-21T08:43:02.756353+0000","last_undegraded":"2024-04-21T08:43:02.756353+0000","last_fullsized":"2024-04-21T08:43:02.756353+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:44:27.124847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757318+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:43:02.757318+0000","last_peered":"2024-04-21T08:43:02.757318+0000","last_clean":"2024-04-21T08:43:02.757318+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:43:02.757318+0000","last_undegraded":"2024-04-21T08:43:02.757318+0000","last_fullsized":"2024-04-21T08:43:02.757318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756980+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:43:02.756980+0000","last_peered":"2024-04-21T08:43:02.756980+0000","last_clean":"2024-04-21T08:43:02.756980+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:43:02.756980+0000","last_undegraded":"2024-04-21T08:43:02.756980+0000","last_fullsized":"2024-04-21T08:43:02.756980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756886+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:43:02.756886+0000","last_peered":"2024-04-21T08:43:02.756886+0000","last_clean":"2024-04-21T08:43:02.756886+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:43:02.756886+0000","last_undegraded":"2024-04-21T08:43:02.756886+0000","last_fullsized":"2024-04-21T08:43:02.756886+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756652+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:43:02.756652+0000","last_peered":"2024-04-21T08:43:02.756652+0000","last_clean":"2024-04-21T08:43:02.756652+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:43:02.756652+0000","last_undegraded":"2024-04-21T08:43:02.756652+0000","last_fullsized":"2024-04-21T08:43:02.756652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713886+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:43:02.713886+0000","last_peered":"2024-04-21T08:43:02.713886+0000","last_clean":"2024-04-21T08:43:02.713886+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:43:02.713886+0000","last_undegraded":"2024-04-21T08:43:02.713886+0000","last_fullsized":"2024-04-21T08:43:02.713886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675896+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:43:02.675896+0000","last_peered":"2024-04-21T08:43:02.675896+0000","last_clean":"2024-04-21T08:43:02.675896+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:43:02.675896+0000","last_undegraded":"2024-04-21T08:43:02.675896+0000","last_fullsized":"2024-04-21T08:43:02.675896+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713544+0000","last_change":"2024-04-21T08:42:56.637459+0000","last_active":"2024-04-21T08:43:02.713544+0000","last_peered":"2024-04-21T08:43:02.713544+0000","last_clean":"2024-04-21T08:43:02.713544+0000","last_became_active":"2024-04-21T08:42:56.637129+0000","last_became_peered":"2024-04-21T08:42:56.637129+0000","last_unstale":"2024-04-21T08:43:02.713544+0000","last_undegraded":"2024-04-21T08:43:02.713544+0000","last_fullsized":"2024-04-21T08:43:02.713544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:25:19.406774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756655+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:43:02.756655+0000","last_peered":"2024-04-21T08:43:02.756655+0000","last_clean":"2024-04-21T08:43:02.756655+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:43:02.756655+0000","last_undegraded":"2024-04-21T08:43:02.756655+0000","last_fullsized":"2024-04-21T08:43:02.756655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675765+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:43:02.675765+0000","last_peered":"2024-04-21T08:43:02.675765+0000","last_clean":"2024-04-21T08:43:02.675765+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:43:02.675765+0000","last_undegraded":"2024-04-21T08:43:02.675765+0000","last_fullsized":"2024-04-21T08:43:02.675765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675113+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:43:02.675113+0000","last_peered":"2024-04-21T08:43:02.675113+0000","last_clean":"2024-04-21T08:43:02.675113+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:43:02.675113+0000","last_undegraded":"2024-04-21T08:43:02.675113+0000","last_fullsized":"2024-04-21T08:43:02.675113+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713725+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:43:02.713725+0000","last_peered":"2024-04-21T08:43:02.713725+0000","last_clean":"2024-04-21T08:43:02.713725+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:43:02.713725+0000","last_undegraded":"2024-04-21T08:43:02.713725+0000","last_fullsized":"2024-04-21T08:43:02.713725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757327+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:43:02.757327+0000","last_peered":"2024-04-21T08:43:02.757327+0000","last_clean":"2024-04-21T08:43:02.757327+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:43:02.757327+0000","last_undegraded":"2024-04-21T08:43:02.757327+0000","last_fullsized":"2024-04-21T08:43:02.757327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213638+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:43:03.213638+0000","last_peered":"2024-04-21T08:43:03.213638+0000","last_clean":"2024-04-21T08:43:03.213638+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:43:03.213638+0000","last_undegraded":"2024-04-21T08:43:03.213638+0000","last_fullsized":"2024-04-21T08:43:03.213638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755691+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:43:02.755691+0000","last_peered":"2024-04-21T08:43:02.755691+0000","last_clean":"2024-04-21T08:43:02.755691+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:43:02.755691+0000","last_undegraded":"2024-04-21T08:43:02.755691+0000","last_fullsized":"2024-04-21T08:43:02.755691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756334+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:43:02.756334+0000","last_peered":"2024-04-21T08:43:02.756334+0000","last_clean":"2024-04-21T08:43:02.756334+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:43:02.756334+0000","last_undegraded":"2024-04-21T08:43:02.756334+0000","last_fullsized":"2024-04-21T08:43:02.756334+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713247+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:43:02.713247+0000","last_peered":"2024-04-21T08:43:02.713247+0000","last_clean":"2024-04-21T08:43:02.713247+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:43:02.713247+0000","last_undegraded":"2024-04-21T08:43:02.713247+0000","last_fullsized":"2024-04-21T08:43:02.713247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675751+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:43:02.675751+0000","last_peered":"2024-04-21T08:43:02.675751+0000","last_clean":"2024-04-21T08:43:02.675751+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:43:02.675751+0000","last_undegraded":"2024-04-21T08:43:02.675751+0000","last_fullsized":"2024-04-21T08:43:02.675751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756155+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:43:02.756155+0000","last_peered":"2024-04-21T08:43:02.756155+0000","last_clean":"2024-04-21T08:43:02.756155+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:43:02.756155+0000","last_undegraded":"2024-04-21T08:43:02.756155+0000","last_fullsized":"2024-04-21T08:43:02.756155+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675504+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:43:02.675504+0000","last_peered":"2024-04-21T08:43:02.675504+0000","last_clean":"2024-04-21T08:43:02.675504+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:43:02.675504+0000","last_undegraded":"2024-04-21T08:43:02.675504+0000","last_fullsized":"2024-04-21T08:43:02.675504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756741+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:43:02.756741+0000","last_peered":"2024-04-21T08:43:02.756741+0000","last_clean":"2024-04-21T08:43:02.756741+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:43:02.756741+0000","last_undegraded":"2024-04-21T08:43:02.756741+0000","last_fullsized":"2024-04-21T08:43:02.756741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675643+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:43:02.675643+0000","last_peered":"2024-04-21T08:43:02.675643+0000","last_clean":"2024-04-21T08:43:02.675643+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:43:02.675643+0000","last_undegraded":"2024-04-21T08:43:02.675643+0000","last_fullsized":"2024-04-21T08:43:02.675643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757556+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:43:02.757556+0000","last_peered":"2024-04-21T08:43:02.757556+0000","last_clean":"2024-04-21T08:43:02.757556+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:43:02.757556+0000","last_undegraded":"2024-04-21T08:43:02.757556+0000","last_fullsized":"2024-04-21T08:43:02.757556+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756359+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:43:02.756359+0000","last_peered":"2024-04-21T08:43:02.756359+0000","last_clean":"2024-04-21T08:43:02.756359+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:43:02.756359+0000","last_undegraded":"2024-04-21T08:43:02.756359+0000","last_fullsized":"2024-04-21T08:43:02.756359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756766+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:43:02.756766+0000","last_peered":"2024-04-21T08:43:02.756766+0000","last_clean":"2024-04-21T08:43:02.756766+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:43:02.756766+0000","last_undegraded":"2024-04-21T08:43:02.756766+0000","last_fullsized":"2024-04-21T08:43:02.756766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713831+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:43:02.713831+0000","last_peered":"2024-04-21T08:43:02.713831+0000","last_clean":"2024-04-21T08:43:02.713831+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:43:02.713831+0000","last_undegraded":"2024-04-21T08:43:02.713831+0000","last_fullsized":"2024-04-21T08:43:02.713831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757158+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:43:02.757158+0000","last_peered":"2024-04-21T08:43:02.757158+0000","last_clean":"2024-04-21T08:43:02.757158+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:43:02.757158+0000","last_undegraded":"2024-04-21T08:43:02.757158+0000","last_fullsized":"2024-04-21T08:43:02.757158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675901+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:43:02.675901+0000","last_peered":"2024-04-21T08:43:02.675901+0000","last_clean":"2024-04-21T08:43:02.675901+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:43:02.675901+0000","last_undegraded":"2024-04-21T08:43:02.675901+0000","last_fullsized":"2024-04-21T08:43:02.675901+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:13.378354+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:43:13.378354+0000","last_peered":"2024-04-21T08:43:13.378354+0000","last_clean":"2024-04-21T08:43:13.378354+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:43:13.378354+0000","last_undegraded":"2024-04-21T08:43:13.378354+0000","last_fullsized":"2024-04-21T08:43:13.378354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757170+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:43:02.757170+0000","last_peered":"2024-04-21T08:43:02.757170+0000","last_clean":"2024-04-21T08:43:02.757170+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:43:02.757170+0000","last_undegraded":"2024-04-21T08:43:02.757170+0000","last_fullsized":"2024-04-21T08:43:02.757170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.714241+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:43:02.714241+0000","last_peered":"2024-04-21T08:43:02.714241+0000","last_clean":"2024-04-21T08:43:02.714241+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:43:02.714241+0000","last_undegraded":"2024-04-21T08:43:02.714241+0000","last_fullsized":"2024-04-21T08:43:02.714241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675645+0000","last_change":"2024-04-21T08:42:56.630433+0000","last_active":"2024-04-21T08:43:02.675645+0000","last_peered":"2024-04-21T08:43:02.675645+0000","last_clean":"2024-04-21T08:43:02.675645+0000","last_became_active":"2024-04-21T08:42:56.630123+0000","last_became_peered":"2024-04-21T08:42:56.630123+0000","last_unstale":"2024-04-21T08:43:02.675645+0000","last_undegraded":"2024-04-21T08:43:02.675645+0000","last_fullsized":"2024-04-21T08:43:02.675645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:42.292961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213717+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:43:03.213717+0000","last_peered":"2024-04-21T08:43:03.213717+0000","last_clean":"2024-04-21T08:43:03.213717+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:43:03.213717+0000","last_undegraded":"2024-04-21T08:43:03.213717+0000","last_fullsized":"2024-04-21T08:43:03.213717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756643+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:43:02.756643+0000","last_peered":"2024-04-21T08:43:02.756643+0000","last_clean":"2024-04-21T08:43:02.756643+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:43:02.756643+0000","last_undegraded":"2024-04-21T08:43:02.756643+0000","last_fullsized":"2024-04-21T08:43:02.756643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713546+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:43:02.713546+0000","last_peered":"2024-04-21T08:43:02.713546+0000","last_clean":"2024-04-21T08:43:02.713546+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:43:02.713546+0000","last_undegraded":"2024-04-21T08:43:02.713546+0000","last_fullsized":"2024-04-21T08:43:02.713546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757291+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:43:02.757291+0000","last_peered":"2024-04-21T08:43:02.757291+0000","last_clean":"2024-04-21T08:43:02.757291+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:43:02.757291+0000","last_undegraded":"2024-04-21T08:43:02.757291+0000","last_fullsized":"2024-04-21T08:43:02.757291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676462+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:43:02.676462+0000","last_peered":"2024-04-21T08:43:02.676462+0000","last_clean":"2024-04-21T08:43:02.676462+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:43:02.676462+0000","last_undegraded":"2024-04-21T08:43:02.676462+0000","last_fullsized":"2024-04-21T08:43:02.676462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756528+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:43:02.756528+0000","last_peered":"2024-04-21T08:43:02.756528+0000","last_clean":"2024-04-21T08:43:02.756528+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:43:02.756528+0000","last_undegraded":"2024-04-21T08:43:02.756528+0000","last_fullsized":"2024-04-21T08:43:02.756528+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756200+0000","last_change":"2024-04-21T08:42:57.405704+0000","last_active":"2024-04-21T08:43:02.756200+0000","last_peered":"2024-04-21T08:43:02.756200+0000","last_clean":"2024-04-21T08:43:02.756200+0000","last_became_active":"2024-04-21T08:42:57.403829+0000","last_became_peered":"2024-04-21T08:42:57.403829+0000","last_unstale":"2024-04-21T08:43:02.756200+0000","last_undegraded":"2024-04-21T08:43:02.756200+0000","last_fullsized":"2024-04-21T08:43:02.756200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:34:01.082283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214184+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:43:03.214184+0000","last_peered":"2024-04-21T08:43:03.214184+0000","last_clean":"2024-04-21T08:43:03.214184+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:43:03.214184+0000","last_undegraded":"2024-04-21T08:43:03.214184+0000","last_fullsized":"2024-04-21T08:43:03.214184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756330+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:43:02.756330+0000","last_peered":"2024-04-21T08:43:02.756330+0000","last_clean":"2024-04-21T08:43:02.756330+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:43:02.756330+0000","last_undegraded":"2024-04-21T08:43:02.756330+0000","last_fullsized":"2024-04-21T08:43:02.756330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713508+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:43:02.713508+0000","last_peered":"2024-04-21T08:43:02.713508+0000","last_clean":"2024-04-21T08:43:02.713508+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:43:02.713508+0000","last_undegraded":"2024-04-21T08:43:02.713508+0000","last_fullsized":"2024-04-21T08:43:02.713508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713586+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:43:02.713586+0000","last_peered":"2024-04-21T08:43:02.713586+0000","last_clean":"2024-04-21T08:43:02.713586+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:43:02.713586+0000","last_undegraded":"2024-04-21T08:43:02.713586+0000","last_fullsized":"2024-04-21T08:43:02.713586+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757157+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:43:02.757157+0000","last_peered":"2024-04-21T08:43:02.757157+0000","last_clean":"2024-04-21T08:43:02.757157+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:43:02.757157+0000","last_undegraded":"2024-04-21T08:43:02.757157+0000","last_fullsized":"2024-04-21T08:43:02.757157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757181+0000","last_change":"2024-04-21T08:42:57.411284+0000","last_active":"2024-04-21T08:43:02.757181+0000","last_peered":"2024-04-21T08:43:02.757181+0000","last_clean":"2024-04-21T08:43:02.757181+0000","last_became_active":"2024-04-21T08:42:57.409361+0000","last_became_peered":"2024-04-21T08:42:57.409361+0000","last_unstale":"2024-04-21T08:43:02.757181+0000","last_undegraded":"2024-04-21T08:43:02.757181+0000","last_fullsized":"2024-04-21T08:43:02.757181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:59:55.766263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756629+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:43:02.756629+0000","last_peered":"2024-04-21T08:43:02.756629+0000","last_clean":"2024-04-21T08:43:02.756629+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:43:02.756629+0000","last_undegraded":"2024-04-21T08:43:02.756629+0000","last_fullsized":"2024-04-21T08:43:02.756629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675866+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:43:02.675866+0000","last_peered":"2024-04-21T08:43:02.675866+0000","last_clean":"2024-04-21T08:43:02.675866+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:43:02.675866+0000","last_undegraded":"2024-04-21T08:43:02.675866+0000","last_fullsized":"2024-04-21T08:43:02.675866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675472+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:43:02.675472+0000","last_peered":"2024-04-21T08:43:02.675472+0000","last_clean":"2024-04-21T08:43:02.675472+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:43:02.675472+0000","last_undegraded":"2024-04-21T08:43:02.675472+0000","last_fullsized":"2024-04-21T08:43:02.675472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213959+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:43:03.213959+0000","last_peered":"2024-04-21T08:43:03.213959+0000","last_clean":"2024-04-21T08:43:03.213959+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:43:03.213959+0000","last_undegraded":"2024-04-21T08:43:03.213959+0000","last_fullsized":"2024-04-21T08:43:03.213959+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756100+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:43:02.756100+0000","last_peered":"2024-04-21T08:43:02.756100+0000","last_clean":"2024-04-21T08:43:02.756100+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:43:02.756100+0000","last_undegraded":"2024-04-21T08:43:02.756100+0000","last_fullsized":"2024-04-21T08:43:02.756100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675109+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:43:02.675109+0000","last_peered":"2024-04-21T08:43:02.675109+0000","last_clean":"2024-04-21T08:43:02.675109+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:43:02.675109+0000","last_undegraded":"2024-04-21T08:43:02.675109+0000","last_fullsized":"2024-04-21T08:43:02.675109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214106+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:43:03.214106+0000","last_peered":"2024-04-21T08:43:03.214106+0000","last_clean":"2024-04-21T08:43:03.214106+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:43:03.214106+0000","last_undegraded":"2024-04-21T08:43:03.214106+0000","last_fullsized":"2024-04-21T08:43:03.214106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713429+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:43:02.713429+0000","last_peered":"2024-04-21T08:43:02.713429+0000","last_clean":"2024-04-21T08:43:02.713429+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:43:02.713429+0000","last_undegraded":"2024-04-21T08:43:02.713429+0000","last_fullsized":"2024-04-21T08:43:02.713429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213493+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:43:03.213493+0000","last_peered":"2024-04-21T08:43:03.213493+0000","last_clean":"2024-04-21T08:43:03.213493+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:43:03.213493+0000","last_undegraded":"2024-04-21T08:43:03.213493+0000","last_fullsized":"2024-04-21T08:43:03.213493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757465+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:43:02.757465+0000","last_peered":"2024-04-21T08:43:02.757465+0000","last_clean":"2024-04-21T08:43:02.757465+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:43:02.757465+0000","last_undegraded":"2024-04-21T08:43:02.757465+0000","last_fullsized":"2024-04-21T08:43:02.757465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756498+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:43:02.756498+0000","last_peered":"2024-04-21T08:43:02.756498+0000","last_clean":"2024-04-21T08:43:02.756498+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:43:02.756498+0000","last_undegraded":"2024-04-21T08:43:02.756498+0000","last_fullsized":"2024-04-21T08:43:02.756498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756425+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:43:02.756425+0000","last_peered":"2024-04-21T08:43:02.756425+0000","last_clean":"2024-04-21T08:43:02.756425+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:43:02.756425+0000","last_undegraded":"2024-04-21T08:43:02.756425+0000","last_fullsized":"2024-04-21T08:43:02.756425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756997+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:43:02.756997+0000","last_peered":"2024-04-21T08:43:02.756997+0000","last_clean":"2024-04-21T08:43:02.756997+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:43:02.756997+0000","last_undegraded":"2024-04-21T08:43:02.756997+0000","last_fullsized":"2024-04-21T08:43:02.756997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.714100+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:43:02.714100+0000","last_peered":"2024-04-21T08:43:02.714100+0000","last_clean":"2024-04-21T08:43:02.714100+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:43:02.714100+0000","last_undegraded":"2024-04-21T08:43:02.714100+0000","last_fullsized":"2024-04-21T08:43:02.714100+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213490+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:43:03.213490+0000","last_peered":"2024-04-21T08:43:03.213490+0000","last_clean":"2024-04-21T08:43:03.213490+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:43:03.213490+0000","last_undegraded":"2024-04-21T08:43:03.213490+0000","last_fullsized":"2024-04-21T08:43:03.213490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756080+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:43:02.756080+0000","last_peered":"2024-04-21T08:43:02.756080+0000","last_clean":"2024-04-21T08:43:02.756080+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:43:02.756080+0000","last_undegraded":"2024-04-21T08:43:02.756080+0000","last_fullsized":"2024-04-21T08:43:02.756080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756567+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:43:02.756567+0000","last_peered":"2024-04-21T08:43:02.756567+0000","last_clean":"2024-04-21T08:43:02.756567+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:43:02.756567+0000","last_undegraded":"2024-04-21T08:43:02.756567+0000","last_fullsized":"2024-04-21T08:43:02.756567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757185+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:43:02.757185+0000","last_peered":"2024-04-21T08:43:02.757185+0000","last_clean":"2024-04-21T08:43:02.757185+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:43:02.757185+0000","last_undegraded":"2024-04-21T08:43:02.757185+0000","last_fullsized":"2024-04-21T08:43:02.757185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214056+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:43:03.214056+0000","last_peered":"2024-04-21T08:43:03.214056+0000","last_clean":"2024-04-21T08:43:03.214056+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:43:03.214056+0000","last_undegraded":"2024-04-21T08:43:03.214056+0000","last_fullsized":"2024-04-21T08:43:03.214056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755671+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:43:02.755671+0000","last_peered":"2024-04-21T08:43:02.755671+0000","last_clean":"2024-04-21T08:43:02.755671+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:43:02.755671+0000","last_undegraded":"2024-04-21T08:43:02.755671+0000","last_fullsized":"2024-04-21T08:43:02.755671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676295+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:43:02.676295+0000","last_peered":"2024-04-21T08:43:02.676295+0000","last_clean":"2024-04-21T08:43:02.676295+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:43:02.676295+0000","last_undegraded":"2024-04-21T08:43:02.676295+0000","last_fullsized":"2024-04-21T08:43:02.676295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756792+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:43:02.756792+0000","last_peered":"2024-04-21T08:43:02.756792+0000","last_clean":"2024-04-21T08:43:02.756792+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:43:02.756792+0000","last_undegraded":"2024-04-21T08:43:02.756792+0000","last_fullsized":"2024-04-21T08:43:02.756792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675194+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:43:02.675194+0000","last_peered":"2024-04-21T08:43:02.675194+0000","last_clean":"2024-04-21T08:43:02.675194+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:43:02.675194+0000","last_undegraded":"2024-04-21T08:43:02.675194+0000","last_fullsized":"2024-04-21T08:43:02.675194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756624+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:43:02.756624+0000","last_peered":"2024-04-21T08:43:02.756624+0000","last_clean":"2024-04-21T08:43:02.756624+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:43:02.756624+0000","last_undegraded":"2024-04-21T08:43:02.756624+0000","last_fullsized":"2024-04-21T08:43:02.756624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756375+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:43:02.756375+0000","last_peered":"2024-04-21T08:43:02.756375+0000","last_clean":"2024-04-21T08:43:02.756375+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:43:02.756375+0000","last_undegraded":"2024-04-21T08:43:02.756375+0000","last_fullsized":"2024-04-21T08:43:02.756375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756930+0000","last_change":"2024-04-21T08:43:01.720615+0000","last_active":"2024-04-21T08:43:02.756930+0000","last_peered":"2024-04-21T08:43:02.756930+0000","last_clean":"2024-04-21T08:43:02.756930+0000","last_became_active":"2024-04-21T08:43:01.720142+0000","last_became_peered":"2024-04-21T08:43:01.720142+0000","last_unstale":"2024-04-21T08:43:02.756930+0000","last_undegraded":"2024-04-21T08:43:02.756930+0000","last_fullsized":"2024-04-21T08:43:02.756930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:53:44.879493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755912+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:43:02.755912+0000","last_peered":"2024-04-21T08:43:02.755912+0000","last_clean":"2024-04-21T08:43:02.755912+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:43:02.755912+0000","last_undegraded":"2024-04-21T08:43:02.755912+0000","last_fullsized":"2024-04-21T08:43:02.755912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756245+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:43:02.756245+0000","last_peered":"2024-04-21T08:43:02.756245+0000","last_clean":"2024-04-21T08:43:02.756245+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:43:02.756245+0000","last_undegraded":"2024-04-21T08:43:02.756245+0000","last_fullsized":"2024-04-21T08:43:02.756245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.676489+0000","last_change":"2024-04-21T08:43:01.686008+0000","last_active":"2024-04-21T08:43:02.676489+0000","last_peered":"2024-04-21T08:43:02.676489+0000","last_clean":"2024-04-21T08:43:02.676489+0000","last_became_active":"2024-04-21T08:43:01.685442+0000","last_became_peered":"2024-04-21T08:43:01.685442+0000","last_unstale":"2024-04-21T08:43:02.676489+0000","last_undegraded":"2024-04-21T08:43:02.676489+0000","last_fullsized":"2024-04-21T08:43:02.676489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:06.000166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756097+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:43:02.756097+0000","last_peered":"2024-04-21T08:43:02.756097+0000","last_clean":"2024-04-21T08:43:02.756097+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:43:02.756097+0000","last_undegraded":"2024-04-21T08:43:02.756097+0000","last_fullsized":"2024-04-21T08:43:02.756097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756161+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:43:02.756161+0000","last_peered":"2024-04-21T08:43:02.756161+0000","last_clean":"2024-04-21T08:43:02.756161+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:43:02.756161+0000","last_undegraded":"2024-04-21T08:43:02.756161+0000","last_fullsized":"2024-04-21T08:43:02.756161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757635+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:43:02.757635+0000","last_peered":"2024-04-21T08:43:02.757635+0000","last_clean":"2024-04-21T08:43:02.757635+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:43:02.757635+0000","last_undegraded":"2024-04-21T08:43:02.757635+0000","last_fullsized":"2024-04-21T08:43:02.757635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675262+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:43:02.675262+0000","last_peered":"2024-04-21T08:43:02.675262+0000","last_clean":"2024-04-21T08:43:02.675262+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:43:02.675262+0000","last_undegraded":"2024-04-21T08:43:02.675262+0000","last_fullsized":"2024-04-21T08:43:02.675262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756140+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:43:02.756140+0000","last_peered":"2024-04-21T08:43:02.756140+0000","last_clean":"2024-04-21T08:43:02.756140+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:43:02.756140+0000","last_undegraded":"2024-04-21T08:43:02.756140+0000","last_fullsized":"2024-04-21T08:43:02.756140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757346+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:43:02.757346+0000","last_peered":"2024-04-21T08:43:02.757346+0000","last_clean":"2024-04-21T08:43:02.757346+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:43:02.757346+0000","last_undegraded":"2024-04-21T08:43:02.757346+0000","last_fullsized":"2024-04-21T08:43:02.757346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756962+0000","last_change":"2024-04-21T08:42:56.639464+0000","last_active":"2024-04-21T08:43:02.756962+0000","last_peered":"2024-04-21T08:43:02.756962+0000","last_clean":"2024-04-21T08:43:02.756962+0000","last_became_active":"2024-04-21T08:42:56.639142+0000","last_became_peered":"2024-04-21T08:42:56.639142+0000","last_unstale":"2024-04-21T08:43:02.756962+0000","last_undegraded":"2024-04-21T08:43:02.756962+0000","last_fullsized":"2024-04-21T08:43:02.756962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:11:55.703492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756282+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:43:02.756282+0000","last_peered":"2024-04-21T08:43:02.756282+0000","last_clean":"2024-04-21T08:43:02.756282+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:43:02.756282+0000","last_undegraded":"2024-04-21T08:43:02.756282+0000","last_fullsized":"2024-04-21T08:43:02.756282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756224+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:43:02.756224+0000","last_peered":"2024-04-21T08:43:02.756224+0000","last_clean":"2024-04-21T08:43:02.756224+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:43:02.756224+0000","last_undegraded":"2024-04-21T08:43:02.756224+0000","last_fullsized":"2024-04-21T08:43:02.756224+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675696+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:43:02.675696+0000","last_peered":"2024-04-21T08:43:02.675696+0000","last_clean":"2024-04-21T08:43:02.675696+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:43:02.675696+0000","last_undegraded":"2024-04-21T08:43:02.675696+0000","last_fullsized":"2024-04-21T08:43:02.675696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757776+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:43:02.757776+0000","last_peered":"2024-04-21T08:43:02.757776+0000","last_clean":"2024-04-21T08:43:02.757776+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:43:02.757776+0000","last_undegraded":"2024-04-21T08:43:02.757776+0000","last_fullsized":"2024-04-21T08:43:02.757776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755906+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:43:02.755906+0000","last_peered":"2024-04-21T08:43:02.755906+0000","last_clean":"2024-04-21T08:43:02.755906+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:43:02.755906+0000","last_undegraded":"2024-04-21T08:43:02.755906+0000","last_fullsized":"2024-04-21T08:43:02.755906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713932+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:43:02.713932+0000","last_peered":"2024-04-21T08:43:02.713932+0000","last_clean":"2024-04-21T08:43:02.713932+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:43:02.713932+0000","last_undegraded":"2024-04-21T08:43:02.713932+0000","last_fullsized":"2024-04-21T08:43:02.713932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757532+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:43:02.757532+0000","last_peered":"2024-04-21T08:43:02.757532+0000","last_clean":"2024-04-21T08:43:02.757532+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:43:02.757532+0000","last_undegraded":"2024-04-21T08:43:02.757532+0000","last_fullsized":"2024-04-21T08:43:02.757532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713351+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:43:02.713351+0000","last_peered":"2024-04-21T08:43:02.713351+0000","last_clean":"2024-04-21T08:43:02.713351+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:43:02.713351+0000","last_undegraded":"2024-04-21T08:43:02.713351+0000","last_fullsized":"2024-04-21T08:43:02.713351+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.214115+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:43:03.214115+0000","last_peered":"2024-04-21T08:43:03.214115+0000","last_clean":"2024-04-21T08:43:03.214115+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:43:03.214115+0000","last_undegraded":"2024-04-21T08:43:03.214115+0000","last_fullsized":"2024-04-21T08:43:03.214115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.757671+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:43:02.757671+0000","last_peered":"2024-04-21T08:43:02.757671+0000","last_clean":"2024-04-21T08:43:02.757671+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:43:02.757671+0000","last_undegraded":"2024-04-21T08:43:02.757671+0000","last_fullsized":"2024-04-21T08:43:02.757671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756238+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:43:02.756238+0000","last_peered":"2024-04-21T08:43:02.756238+0000","last_clean":"2024-04-21T08:43:02.756238+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:43:02.756238+0000","last_undegraded":"2024-04-21T08:43:02.756238+0000","last_fullsized":"2024-04-21T08:43:02.756238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675590+0000","last_change":"2024-04-21T08:42:57.408595+0000","last_active":"2024-04-21T08:43:02.675590+0000","last_peered":"2024-04-21T08:43:02.675590+0000","last_clean":"2024-04-21T08:43:02.675590+0000","last_became_active":"2024-04-21T08:42:57.408193+0000","last_became_peered":"2024-04-21T08:42:57.408193+0000","last_unstale":"2024-04-21T08:43:02.675590+0000","last_undegraded":"2024-04-21T08:43:02.675590+0000","last_fullsized":"2024-04-21T08:43:02.675590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:04:57.559111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675169+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:43:02.675169+0000","last_peered":"2024-04-21T08:43:02.675169+0000","last_clean":"2024-04-21T08:43:02.675169+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:43:02.675169+0000","last_undegraded":"2024-04-21T08:43:02.675169+0000","last_fullsized":"2024-04-21T08:43:02.675169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.755971+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:43:02.755971+0000","last_peered":"2024-04-21T08:43:02.755971+0000","last_clean":"2024-04-21T08:43:02.755971+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:43:02.755971+0000","last_undegraded":"2024-04-21T08:43:02.755971+0000","last_fullsized":"2024-04-21T08:43:02.755971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213944+0000","last_change":"2024-04-21T08:42:57.408413+0000","last_active":"2024-04-21T08:43:03.213944+0000","last_peered":"2024-04-21T08:43:03.213944+0000","last_clean":"2024-04-21T08:43:03.213944+0000","last_became_active":"2024-04-21T08:42:57.408039+0000","last_became_peered":"2024-04-21T08:42:57.408039+0000","last_unstale":"2024-04-21T08:43:03.213944+0000","last_undegraded":"2024-04-21T08:43:03.213944+0000","last_fullsized":"2024-04-21T08:43:03.213944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:23:13.072822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756995+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:43:02.756995+0000","last_peered":"2024-04-21T08:43:02.756995+0000","last_clean":"2024-04-21T08:43:02.756995+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:43:02.756995+0000","last_undegraded":"2024-04-21T08:43:02.756995+0000","last_fullsized":"2024-04-21T08:43:02.756995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.675985+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:43:02.675985+0000","last_peered":"2024-04-21T08:43:02.675985+0000","last_clean":"2024-04-21T08:43:02.675985+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:43:02.675985+0000","last_undegraded":"2024-04-21T08:43:02.675985+0000","last_fullsized":"2024-04-21T08:43:02.675985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756918+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:43:02.756918+0000","last_peered":"2024-04-21T08:43:02.756918+0000","last_clean":"2024-04-21T08:43:02.756918+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:43:02.756918+0000","last_undegraded":"2024-04-21T08:43:02.756918+0000","last_fullsized":"2024-04-21T08:43:02.756918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.713373+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:43:02.713373+0000","last_peered":"2024-04-21T08:43:02.713373+0000","last_clean":"2024-04-21T08:43:02.713373+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:43:02.713373+0000","last_undegraded":"2024-04-21T08:43:02.713373+0000","last_fullsized":"2024-04-21T08:43:02.713373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756236+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:43:02.756236+0000","last_peered":"2024-04-21T08:43:02.756236+0000","last_clean":"2024-04-21T08:43:02.756236+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:43:02.756236+0000","last_undegraded":"2024-04-21T08:43:02.756236+0000","last_fullsized":"2024-04-21T08:43:02.756236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756593+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:43:02.756593+0000","last_peered":"2024-04-21T08:43:02.756593+0000","last_clean":"2024-04-21T08:43:02.756593+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:43:02.756593+0000","last_undegraded":"2024-04-21T08:43:02.756593+0000","last_fullsized":"2024-04-21T08:43:02.756593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:03.213800+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:43:03.213800+0000","last_peered":"2024-04-21T08:43:03.213800+0000","last_clean":"2024-04-21T08:43:03.213800+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:43:03.213800+0000","last_undegraded":"2024-04-21T08:43:03.213800+0000","last_fullsized":"2024-04-21T08:43:03.213800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-21T08:43:02.756713+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:43:02.756713+0000","last_peered":"2024-04-21T08:43:02.756713+0000","last_clean":"2024-04-21T08:43:02.756713+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:43:02.756713+0000","last_undegraded":"2024-04-21T08:43:02.756713+0000","last_fullsized":"2024-04-21T08:43:02.756713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":251796,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":50549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":51360,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808860,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":970752,"data_stored":477819,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":51360,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":500390,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14688,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":51360,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":40505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:43:25.984 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-21T08:43:25.985 INFO:tasks.cephadm:Setting up client nodes... 2024-04-21T08:43:25.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-21T08:43:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:25 smithi129 ceph-mon[31272]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:25 smithi129 ceph-mon[31272]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-04-21T08:43:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:25 smithi129 ceph-mon[31272]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:43:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2900243190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3356076539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:26.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:25 smithi136 ceph-mon[31791]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:43:26.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:25 smithi136 ceph-mon[31791]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 170 B/s wr, 0 op/s 2024-04-21T08:43:26.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:25 smithi136 ceph-mon[31791]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-21T08:43:26.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2900243190' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:43:26.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3356076539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:26.267 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:27.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:26 smithi129 ceph-mon[31272]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:27.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:26 smithi129 ceph-mon[31272]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:26 smithi057 ceph-mon[27968]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:27.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:26 smithi057 ceph-mon[27968]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:27.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:26 smithi136 ceph-mon[31791]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:27.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:26 smithi136 ceph-mon[31791]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:43:27.976 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:27 smithi057 ceph-mon[27968]: pgmap v310: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-21T08:43:28.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:27 smithi129 ceph-mon[31272]: pgmap v310: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-21T08:43:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.091+0000 7fec2ca0f700 1 -- 172.21.15.57:0/123211893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec281019e0 msgr2=0x7fec28101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.091+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/123211893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec281019e0 0x7fec28101e00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fec180099c0 tx=0x7fec18009cd0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- 172.21.15.57:0/123211893 shutdown_connections 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/123211893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec28103580 0x7fec28109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/123211893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec281019e0 0x7fec28101e00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/123211893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec28102be0 0x7fec28103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- 172.21.15.57:0/123211893 >> 172.21.15.57:0/123211893 conn(0x7fec280fcfa0 msgr2=0x7fec280ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- 172.21.15.57:0/123211893 shutdown_connections 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- 172.21.15.57:0/123211893 wait complete. 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 Processor -- start 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- start start 2024-04-21T08:43:28.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec281019e0 0x7fec28198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec28102be0 0x7fec28199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 0x7fec281a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fec2810d520 con 0x7fec28103580 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec2810d3a0 con 0x7fec28102be0 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.092+0000 7fec2ca0f700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fec2810d6a0 con 0x7fec281019e0 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 0x7fec281a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 0x7fec281a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35854/0 (socket says 172.21.15.57:35854) 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 -- 172.21.15.57:0/1186424034 learned_addr learned my addr 172.21.15.57:0/1186424034 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec25d9b700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec28102be0 0x7fec28199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec2659c700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec281019e0 0x7fec28198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 -- 172.21.15.57:0/1186424034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec281019e0 msgr2=0x7fec28198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:28.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec281019e0 0x7fec28198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 -- 172.21.15.57:0/1186424034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec28102be0 msgr2=0x7fec28199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec28102be0 0x7fec28199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 -- 172.21.15.57:0/1186424034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec281a07b0 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec26d9d700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 0x7fec281a0110 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fec2000b9a0 tx=0x7fec2000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec20019070 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec20014440 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2000ce60 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec281a0aa0 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.093+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec281a0f90 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.094+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec28050f80 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.096+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fec2001c070 con 0x7fec28103580 2024-04-21T08:43:28.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.096+0000 7fec177fe700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec100a3d80 0x7fec100a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.096+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fec200cffc0 con 0x7fec28103580 2024-04-21T08:43:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.096+0000 7fec2659c700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec100a3d80 0x7fec100a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.097+0000 7fec2659c700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec100a3d80 0x7fec100a6240 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fec1800b580 tx=0x7fec18005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:28.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.100+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec2009c7a0 con 0x7fec28103580 2024-04-21T08:43:28.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:27 smithi136 ceph-mon[31791]: pgmap v310: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-21T08:43:28.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.399+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fec281a13e0 con 0x7fec28103580 2024-04-21T08:43:28.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.401+0000 7fec177fe700 1 -- 172.21.15.57:0/1186424034 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fec2009c0a0 con 0x7fec28103580 2024-04-21T08:43:28.401 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2024-04-21T08:43:28.401 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQCw0SRmPdbqFxAALCWg/QrmMPlJNm4dNK22bg== 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec100a3d80 msgr2=0x7fec100a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec100a3d80 0x7fec100a6240 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fec1800b580 tx=0x7fec18005c00 comp rx=0 tx=0).stop 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 msgr2=0x7fec281a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 0x7fec281a0110 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fec2000b9a0 tx=0x7fec2000bd60 comp rx=0 tx=0).stop 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 shutdown_connections 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fec100a3d80 0x7fec100a6240 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec28103580 0x7fec281a0110 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec281019e0 0x7fec28198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 --2- 172.21.15.57:0/1186424034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec28102be0 0x7fec28199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 >> 172.21.15.57:0/1186424034 conn(0x7fec280fcfa0 msgr2=0x7fec280fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:28.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 shutdown_connections 2024-04-21T08:43:28.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:28.403+0000 7fec2ca0f700 1 -- 172.21.15.57:0/1186424034 wait complete. 2024-04-21T08:43:29.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1186424034' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:43:29.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1186424034' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:43:29.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1186424034' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:43:29.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1186424034' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:43:29.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1186424034' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-21T08:43:29.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1186424034' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-21T08:43:29.339 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:43:29.339 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-21T08:43:29.339 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-21T08:43:29.377 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-21T08:43:29.377 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-21T08:43:29.377 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-21T08:43:29.666 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:30.548 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:30 smithi057 ceph-mon[27968]: pgmap v311: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-04-21T08:43:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:30 smithi129 ceph-mon[31272]: pgmap v311: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-04-21T08:43:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:30 smithi136 ceph-mon[31791]: pgmap v311: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 -- 172.21.15.57:0/2229118644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec108a50 msgr2=0x7fa9ec108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 --2- 172.21.15.57:0/2229118644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec108a50 0x7fa9ec108e30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009990 tx=0x7fa9dc009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 -- 172.21.15.57:0/2229118644 shutdown_connections 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 --2- 172.21.15.57:0/2229118644 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec103450 0x7fa9ec10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 --2- 172.21.15.57:0/2229118644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec108a50 0x7fa9ec108e30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 --2- 172.21.15.57:0/2229118644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec102a90 0x7fa9ec102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 -- 172.21.15.57:0/2229118644 >> 172.21.15.57:0/2229118644 conn(0x7fa9ec079ba0 msgr2=0x7fa9ec079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 -- 172.21.15.57:0/2229118644 shutdown_connections 2024-04-21T08:43:31.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.516+0000 7fa9f1493700 1 -- 172.21.15.57:0/2229118644 wait complete. 2024-04-21T08:43:31.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 Processor -- start 2024-04-21T08:43:31.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 -- start start 2024-04-21T08:43:31.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 0x7fa9ec199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec103450 0x7fa9ec1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec108a50 0x7fa9ec19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec112c60 con 0x7fa9ec102a90 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec112ae0 con 0x7fa9ec108a50 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9f1493700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec112de0 con 0x7fa9ec103450 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9eaffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 0x7fa9ec199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9eaffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 0x7fa9ec199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35860/0 (socket says 172.21.15.57:35860) 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9eaffd700 1 -- 172.21.15.57:0/1882260278 learned_addr learned my addr 172.21.15.57:0/1882260278 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eb7fe700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec108a50 0x7fa9ec19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.517+0000 7fa9ea7fc700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec103450 0x7fa9ec1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eaffd700 1 -- 172.21.15.57:0/1882260278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec103450 msgr2=0x7fa9ec1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eaffd700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec103450 0x7fa9ec1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eaffd700 1 -- 172.21.15.57:0/1882260278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec108a50 msgr2=0x7fa9ec19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:31.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eaffd700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec108a50 0x7fa9ec19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eaffd700 1 -- 172.21.15.57:0/1882260278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec19e180 con 0x7fa9ec102a90 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9ea7fc700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec103450 0x7fa9ec1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9eaffd700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 0x7fa9ec199170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc000c00 tx=0x7fa9dc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc027070 con 0x7fa9ec102a90 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ec19e410 con 0x7fa9ec102a90 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ec1a5cf0 con 0x7fa9ec102a90 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9dc0040e0 con 0x7fa9ec102a90 2024-04-21T08:43:31.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.518+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc022ac0 con 0x7fa9ec102a90 2024-04-21T08:43:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.520+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d8005330 con 0x7fa9ec102a90 2024-04-21T08:43:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.521+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa9dc02a070 con 0x7fa9ec102a90 2024-04-21T08:43:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.521+0000 7fa9d3fff700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9d40a3cd0 0x7fa9d40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.521+0000 7fa9ea7fc700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9d40a3cd0 0x7fa9d40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.522+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa9dc0d1e10 con 0x7fa9ec102a90 2024-04-21T08:43:31.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.522+0000 7fa9ea7fc700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9d40a3cd0 0x7fa9d40a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005fd0 tx=0x7fa9e0005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:31.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.525+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9dc031070 con 0x7fa9ec102a90 2024-04-21T08:43:31.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.795+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa9d80058f0 con 0x7fa9ec102a90 2024-04-21T08:43:31.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.799+0000 7fa9d3fff700 1 -- 172.21.15.57:0/1882260278 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7fa9dc013070 con 0x7fa9ec102a90 2024-04-21T08:43:31.803 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:31.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9d40a3cd0 msgr2=0x7fa9d40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9d40a3cd0 0x7fa9d40a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0005fd0 tx=0x7fa9e0005dc0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 msgr2=0x7fa9ec199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 0x7fa9ec199170 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc000c00 tx=0x7fa9dc0047b0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 shutdown_connections 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9d40a3cd0 0x7fa9d40a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9ec108a50 0x7fa9ec19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ec102a90 0x7fa9ec199170 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 --2- 172.21.15.57:0/1882260278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9ec103450 0x7fa9ec1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 >> 172.21.15.57:0/1882260278 conn(0x7fa9ec079ba0 msgr2=0x7fa9ec101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 shutdown_connections 2024-04-21T08:43:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:31.804+0000 7fa9f1493700 1 -- 172.21.15.57:0/1882260278 wait complete. 2024-04-21T08:43:32.665 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:32 smithi057 ceph-mon[27968]: pgmap v312: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-04-21T08:43:32.665 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:32 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1882260278' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:43:32.666 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":3002879345},{"type":"v1","addr":"172.21.15.57:6801","nonce":3002879345}]},"active_addr":"172.21.15.57:6801/3002879345","active_change":"2024-04-21T08:35:44.517594+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.57:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":1907214317}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":4242179214}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":3327023287}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":608627600}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.57:0","nonce":1525724253}]}]} 2024-04-21T08:43:32.671 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-21T08:43:32.671 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-21T08:43:32.671 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:43:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:32 smithi129 ceph-mon[31272]: pgmap v312: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-04-21T08:43:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:32 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1882260278' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:43:32.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:32 smithi136 ceph-mon[31791]: pgmap v312: 129 pgs: 129 active+clean; 617 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 3.7 KiB/s wr, 3 op/s 2024-04-21T08:43:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:32 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1882260278' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-21T08:43:32.924 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:34.670 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:34 smithi057 ceph-mon[27968]: pgmap v313: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-21T08:43:34.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 -- 172.21.15.57:0/3162738749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4107990 msgr2=0x7f3dc4107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 --2- 172.21.15.57:0/3162738749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4107990 0x7f3dc4107d70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009990 tx=0x7f3db8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 -- 172.21.15.57:0/3162738749 shutdown_connections 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 --2- 172.21.15.57:0/3162738749 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dc4102300 0x7f3dc410e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 --2- 172.21.15.57:0/3162738749 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4107990 0x7f3dc4107d70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 --2- 172.21.15.57:0/3162738749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc4101940 0x7f3dc4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 -- 172.21.15.57:0/3162738749 >> 172.21.15.57:0/3162738749 conn(0x7f3dc40788f0 msgr2=0x7f3dc4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 -- 172.21.15.57:0/3162738749 shutdown_connections 2024-04-21T08:43:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.675+0000 7f3dcab03700 1 -- 172.21.15.57:0/3162738749 wait complete. 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 Processor -- start 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 -- start start 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc4101940 0x7f3dc4199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 0x7f3dc419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dc4107990 0x7f3dc41a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4111ae0 con 0x7f3dc4102300 2024-04-21T08:43:34.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4111960 con 0x7f3dc4101940 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dcab03700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4111c60 con 0x7f3dc4107990 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dc3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 0x7f3dc419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dc3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 0x7f3dc419a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40476/0 (socket says 172.21.15.57:40476) 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dc3fff700 1 -- 172.21.15.57:0/788504818 learned_addr learned my addr 172.21.15.57:0/788504818 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dc889f700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc4101940 0x7f3dc4199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.676+0000 7f3dc90a0700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dc4107990 0x7f3dc41a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc3fff700 1 -- 172.21.15.57:0/788504818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dc4107990 msgr2=0x7f3dc41a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc3fff700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dc4107990 0x7f3dc41a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc3fff700 1 -- 172.21.15.57:0/788504818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc4101940 msgr2=0x7f3dc4199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc3fff700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc4101940 0x7f3dc4199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc3fff700 1 -- 172.21.15.57:0/788504818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc41a5500 con 0x7f3dc4102300 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc3fff700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 0x7f3dc419a480 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3db000c920 tx=0x7f3db000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0004a30 con 0x7f3dc4102300 2024-04-21T08:43:34.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3db00054d0 con 0x7f3dc4102300 2024-04-21T08:43:34.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc41a5730 con 0x7f3dc4102300 2024-04-21T08:43:34.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0007c70 con 0x7f3dc4102300 2024-04-21T08:43:34.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.677+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc41a5c70 con 0x7f3dc4102300 2024-04-21T08:43:34.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.678+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc4109eb0 con 0x7f3dc4102300 2024-04-21T08:43:34.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.680+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3db0004c60 con 0x7f3dc4102300 2024-04-21T08:43:34.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.680+0000 7f3dc1ffb700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3db40a3cd0 0x7f3db40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:34.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.680+0000 7f3dc889f700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3db40a3cd0 0x7f3db40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:34.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.680+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3db00d0680 con 0x7f3dc4102300 2024-04-21T08:43:34.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.681+0000 7f3dc889f700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3db40a3cd0 0x7f3db40a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3db800b5c0 tx=0x7f3db8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:34.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.685+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3db0014030 con 0x7f3dc4102300 2024-04-21T08:43:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:34 smithi129 ceph-mon[31272]: pgmap v313: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-21T08:43:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:34 smithi136 ceph-mon[31791]: pgmap v313: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-21T08:43:34.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.893+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3dc4109890 con 0x7f3dc4102300 2024-04-21T08:43:34.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.895+0000 7f3dc1ffb700 1 -- 172.21.15.57:0/788504818 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7f3db009ccc0 con 0x7f3dc4102300 2024-04-21T08:43:34.895 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:34.895 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3db40a3cd0 msgr2=0x7f3db40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3db40a3cd0 0x7f3db40a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3db800b5c0 tx=0x7f3db8005960 comp rx=0 tx=0).stop 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 msgr2=0x7f3dc419a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 0x7f3dc419a480 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3db000c920 tx=0x7f3db000cc30 comp rx=0 tx=0).stop 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 shutdown_connections 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3db40a3cd0 0x7f3db40a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc4101940 0x7f3dc4199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dc4107990 0x7f3dc41a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 --2- 172.21.15.57:0/788504818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3dc4102300 0x7f3dc419a480 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 >> 172.21.15.57:0/788504818 conn(0x7f3dc40788f0 msgr2=0x7f3dc41007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 shutdown_connections 2024-04-21T08:43:34.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:34.897+0000 7f3dcab03700 1 -- 172.21.15.57:0/788504818 wait complete. 2024-04-21T08:43:35.683 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:35 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/788504818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:35.685 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-21T08:43:35.685 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-21T08:43:35.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:35 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/788504818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:35.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:35 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/788504818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:35.940 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:36.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:36 smithi057 ceph-mon[27968]: pgmap v314: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-21T08:43:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:36 smithi129 ceph-mon[31272]: pgmap v314: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-21T08:43:36.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:36 smithi136 ceph-mon[31791]: pgmap v314: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-21T08:43:37.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 -- 172.21.15.57:0/3959198003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 msgr2=0x7ff22c111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 --2- 172.21.15.57:0/3959198003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c111830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff22400b0a0 tx=0x7ff22400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 -- 172.21.15.57:0/3959198003 shutdown_connections 2024-04-21T08:43:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 --2- 172.21.15.57:0/3959198003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c111830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 --2- 172.21.15.57:0/3959198003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff22c074c70 0x7ff22c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 --2- 172.21.15.57:0/3959198003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff22c075590 0x7ff22c10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 -- 172.21.15.57:0/3959198003 >> 172.21.15.57:0/3959198003 conn(0x7ff22c079ba0 msgr2=0x7ff22c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 -- 172.21.15.57:0/3959198003 shutdown_connections 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.733+0000 7ff230cf3700 1 -- 172.21.15.57:0/3959198003 wait complete. 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 Processor -- start 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 -- start start 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff22c074c70 0x7ff22c19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:37.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff22c075590 0x7ff22c19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c19fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c114e40 con 0x7ff22c10f440 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c114cc0 con 0x7ff22c075590 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff230cf3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c114fc0 con 0x7ff22c074c70 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff22ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c19fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff22ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c19fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40502/0 (socket says 172.21.15.57:40502) 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff22ad9d700 1 -- 172.21.15.57:0/765937196 learned_addr learned my addr 172.21.15.57:0/765937196 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff229d9b700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff22c075590 0x7ff22c19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:37.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.734+0000 7ff22a59c700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff22c074c70 0x7ff22c19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff22ad9d700 1 -- 172.21.15.57:0/765937196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff22c074c70 msgr2=0x7ff22c19b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff22ad9d700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff22c074c70 0x7ff22c19b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff22ad9d700 1 -- 172.21.15.57:0/765937196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff22c075590 msgr2=0x7ff22c19b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff22ad9d700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff22c075590 0x7ff22c19b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff22ad9d700 1 -- 172.21.15.57:0/765937196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1a0280 con 0x7ff22c10f440 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff22ad9d700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c19fb40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff2240062a0 tx=0x7ff224004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224021070 con 0x7ff22c10f440 2024-04-21T08:43:37.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c1a0510 con 0x7ff22c10f440 2024-04-21T08:43:37.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff224007c80 con 0x7ff22c10f440 2024-04-21T08:43:37.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22401d450 con 0x7ff22c10f440 2024-04-21T08:43:37.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.735+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c1a7df0 con 0x7ff22c10f440 2024-04-21T08:43:37.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.736+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22c069730 con 0x7ff22c10f440 2024-04-21T08:43:37.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.737+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff224004020 con 0x7ff22c10f440 2024-04-21T08:43:37.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.738+0000 7ff21b7fe700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2140a3cd0 0x7ff2140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:37.743 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.738+0000 7ff22a59c700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2140a3cd0 0x7ff2140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:37.743 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.738+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff22400f020 con 0x7ff22c10f440 2024-04-21T08:43:37.743 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.738+0000 7ff22a59c700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2140a3cd0 0x7ff2140a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff21c006fd0 tx=0x7ff21c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:37.743 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.743+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff22402a020 con 0x7ff22c10f440 2024-04-21T08:43:37.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.950+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff22c02df80 con 0x7ff22c10f440 2024-04-21T08:43:37.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.952+0000 7ff21b7fe700 1 -- 172.21.15.57:0/765937196 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24104 (secure 0 0 0) 0x7ff22409e550 con 0x7ff22c10f440 2024-04-21T08:43:37.952 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:43:37.952 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":89,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:02.031173+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2140a3cd0 msgr2=0x7ff2140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2140a3cd0 0x7ff2140a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff21c006fd0 tx=0x7ff21c008040 comp rx=0 tx=0).stop 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 msgr2=0x7ff22c19fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c19fb40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff2240062a0 tx=0x7ff224004e20 comp rx=0 tx=0).stop 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 shutdown_connections 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff2140a3cd0 0x7ff2140a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff22c10f440 0x7ff22c19fb40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff22c074c70 0x7ff22c19b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 --2- 172.21.15.57:0/765937196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff22c075590 0x7ff22c19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 >> 172.21.15.57:0/765937196 conn(0x7ff22c079ba0 msgr2=0x7ff22c0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 shutdown_connections 2024-04-21T08:43:37.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:37.954+0000 7ff230cf3700 1 -- 172.21.15.57:0/765937196 wait complete. 2024-04-21T08:43:38.674 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-21T08:43:38.674 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-21T08:43:38.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-21T08:43:38.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-21T08:43:38.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-21T08:43:38.676 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-21T08:43:38.676 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-21T08:43:38.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-21T08:43:38.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-21T08:43:38.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-21T08:43:38.678 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-21T08:43:38.678 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-21T08:43:38.720 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:38 smithi057 ceph-mon[27968]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T08:43:38.720 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:38 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/765937196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:38 smithi129 ceph-mon[31272]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T08:43:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/765937196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:38 smithi136 ceph-mon[31791]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-04-21T08:43:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:38 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/765937196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:43:39.437 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:39.438 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:39.438 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:39.441 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:39.444 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:40.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:40 smithi057 ceph-mon[27968]: pgmap v316: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-21T08:43:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:40 smithi129 ceph-mon[31272]: pgmap v316: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-21T08:43:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:40 smithi136 ceph-mon[31791]: pgmap v316: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-21T08:43:41.664 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:42.761 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:42 smithi057 ceph-mon[27968]: pgmap v317: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-21T08:43:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:42 smithi129 ceph-mon[31272]: pgmap v317: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-21T08:43:42.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:42 smithi136 ceph-mon[31791]: pgmap v317: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-21T08:43:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:43.181 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:43.182 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:43.183 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:43.183 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:44.700 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:44 smithi057 ceph-mon[27968]: pgmap v318: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-21T08:43:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:44 smithi129 ceph-mon[31272]: pgmap v318: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-21T08:43:44.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:44 smithi136 ceph-mon[31791]: pgmap v318: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-21T08:43:45.513 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-21T08:43:45.514 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:45 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8, 5, 10]}]: dispatch 2024-04-21T08:43:45.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-21T08:43:45.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:45 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8, 5, 10]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-21T08:43:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:45 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8, 5, 10]}]: dispatch 2024-04-21T08:43:46.060 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:43:46.730 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: pgmap v319: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8, 5, 10]}]': finished 2024-04-21T08:43:46.731 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:46 smithi057 ceph-mon[27968]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: pgmap v319: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-21T08:43:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T08:43:46.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-21T08:43:46.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-21T08:43:46.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8, 5, 10]}]': finished 2024-04-21T08:43:46.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:46 smithi129 ceph-mon[31272]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: pgmap v319: 129 pgs: 129 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-21T08:43:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-04-21T08:43:46.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-21T08:43:46.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8, 5, 10]}]': finished 2024-04-21T08:43:46.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:46 smithi136 ceph-mon[31791]: osdmap e90: 12 total, 12 up, 12 in 2024-04-21T08:43:47.720 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:47 smithi057 ceph-mon[27968]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:43:47.720 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:47 smithi057 ceph-mon[27968]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:43:47.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:47 smithi129 ceph-mon[31272]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:43:47.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:47 smithi129 ceph-mon[31272]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:43:47.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:47 smithi136 ceph-mon[31791]: osdmap e91: 12 total, 12 up, 12 in 2024-04-21T08:43:47.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:47 smithi136 ceph-mon[31791]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-21T08:43:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 -- 172.21.15.57:0/336863362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 msgr2=0x7f6d2c10f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 --2- 172.21.15.57:0/336863362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c10f900 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400b0a0 tx=0x7f6d2400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 -- 172.21.15.57:0/336863362 shutdown_connections 2024-04-21T08:43:49.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 --2- 172.21.15.57:0/336863362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c10f900 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 --2- 172.21.15.57:0/336863362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d2c108a30 0x7f6d2c108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 --2- 172.21.15.57:0/336863362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d2c06b170 0x7f6d2c06b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 -- 172.21.15.57:0/336863362 >> 172.21.15.57:0/336863362 conn(0x7f6d2c0788f0 msgr2=0x7f6d2c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 -- 172.21.15.57:0/336863362 shutdown_connections 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 -- 172.21.15.57:0/336863362 wait complete. 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 Processor -- start 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.309+0000 7f6d341cd700 1 -- start start 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d341cd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d2c06b170 0x7f6d2c19b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d341cd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c19b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d341cd700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d2c108a30 0x7f6d2c19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d341cd700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112bd0 con 0x7f6d2c06bb10 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d341cd700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112a50 con 0x7f6d2c06b170 2024-04-21T08:43:49.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d341cd700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112d50 con 0x7f6d2c108a30 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c19b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c19b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56018/0 (socket says 172.21.15.57:56018) 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 -- 172.21.15.57:0/4231469428 learned_addr learned my addr 172.21.15.57:0/4231469428 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d3276a700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d2c108a30 0x7f6d2c19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31f69700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d2c06b170 0x7f6d2c19b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 -- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d2c108a30 msgr2=0x7f6d2c19fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d2c108a30 0x7f6d2c19fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 -- 172.21.15.57:0/4231469428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d2c06b170 msgr2=0x7f6d2c19b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d2c06b170 0x7f6d2c19b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 -- 172.21.15.57:0/4231469428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c1a02d0 con 0x7f6d2c06bb10 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31f69700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6d2c06b170 0x7f6d2c19b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.310+0000 7f6d31768700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c19b800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00c9b0 tx=0x7f6d1c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:49.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.311+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c007b00 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.311+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d1c007ca0 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.311+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c018660 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.311+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2c1a05c0 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.311+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2c1a7e40 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.311+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d2c063110 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.313+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6d1c00f450 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.313+0000 7f6d22ffd700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6d180a3d50 0x7f6d180a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.313+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f6d1c0d0c80 con 0x7f6d2c06bb10 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.313+0000 7f6d31f69700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6d180a3d50 0x7f6d180a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.314+0000 7f6d22ffd700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] conn(0x7f6d180aee50 0x7f6d180b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.314+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 --> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d1c00c120 con 0x7f6d180aee50 2024-04-21T08:43:49.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.314+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f6d1c0d10d0 con 0x7f6d2c06bb10 2024-04-21T08:43:49.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.314+0000 7f6d31f69700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6d180a3d50 0x7f6d180a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d28005fd0 tx=0x7f6d28005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:49.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.314+0000 7f6d3276a700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] conn(0x7f6d180aee50 0x7f6d180b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:49.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.314+0000 7f6d3276a700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] conn(0x7f6d180aee50 0x7f6d180b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:49.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.315+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== osd.5 v2:172.21.15.136:6808/564097587 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d1c00c120 con 0x7f6d180aee50 2024-04-21T08:43:49.334 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.332+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 --> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d2c063510 con 0x7f6d180aee50 2024-04-21T08:43:49.334 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.333+0000 7f6d22ffd700 1 -- 172.21.15.57:0/4231469428 <== osd.5 v2:172.21.15.136:6808/564097587 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6d2c063510 con 0x7f6d180aee50 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.333+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] conn(0x7f6d180aee50 msgr2=0x7f6d180b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.333+0000 7f6d341cd700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] conn(0x7f6d180aee50 0x7f6d180b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6d180a3d50 msgr2=0x7f6d180a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6d180a3d50 0x7f6d180a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6d28005fd0 tx=0x7f6d28005d40 comp rx=0 tx=0).stop 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 msgr2=0x7f6d2c19b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 --2- 172.21.15.57:0/4231469428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6d2c06bb10 0x7f6d2c19b800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c00c9b0 tx=0x7f6d1c00ccc0 comp rx=0 tx=0).stop 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d3276a700 1 -- 172.21.15.57:0/4231469428 reap_dead start 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 shutdown_connections 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 >> 172.21.15.57:0/4231469428 conn(0x7f6d2c0788f0 msgr2=0x7f6d2c10de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 shutdown_connections 2024-04-21T08:43:49.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:49.334+0000 7f6d341cd700 1 -- 172.21.15.57:0/4231469428 wait complete. 2024-04-21T08:43:49.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:49 smithi057 ceph-mon[27968]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:50.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:49 smithi129 ceph-mon[31272]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:50.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:49 smithi136 ceph-mon[31791]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 -- 172.21.15.57:0/1044323560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806a750 msgr2=0x7f934806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 --2- 172.21.15.57:0/1044323560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806a750 0x7f934806abb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f933c009990 tx=0x7f933c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 -- 172.21.15.57:0/1044323560 shutdown_connections 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 --2- 172.21.15.57:0/1044323560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9348108a30 0x7f9348108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 --2- 172.21.15.57:0/1044323560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f934806b0f0 0x7f934810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 --2- 172.21.15.57:0/1044323560 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806a750 0x7f934806abb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 -- 172.21.15.57:0/1044323560 >> 172.21.15.57:0/1044323560 conn(0x7f93480788f0 msgr2=0x7f9348078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 -- 172.21.15.57:0/1044323560 shutdown_connections 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.509+0000 7f93502af700 1 -- 172.21.15.57:0/1044323560 wait complete. 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 Processor -- start 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 -- start start 2024-04-21T08:43:51.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f934806a750 0x7f934819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b0f0 0x7f934819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9348108a30 0x7f934819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9348112ca0 con 0x7f934806b0f0 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9348112b20 con 0x7f9348108a30 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f93502af700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9348112e20 con 0x7f934806a750 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f934d84a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b0f0 0x7f934819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f934d84a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b0f0 0x7f934819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56022/0 (socket says 172.21.15.57:56022) 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.510+0000 7f934d84a700 1 -- 172.21.15.57:0/492238073 learned_addr learned my addr 172.21.15.57:0/492238073 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934e04b700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f934806a750 0x7f934819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934e84c700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9348108a30 0x7f934819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934d84a700 1 -- 172.21.15.57:0/492238073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f934806a750 msgr2=0x7f934819b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934d84a700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f934806a750 0x7f934819b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934d84a700 1 -- 172.21.15.57:0/492238073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9348108a30 msgr2=0x7f934819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934d84a700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9348108a30 0x7f934819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934d84a700 1 -- 172.21.15.57:0/492238073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93481a0370 con 0x7f934806b0f0 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934e84c700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9348108a30 0x7f934819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934e04b700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f934806a750 0x7f934819b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f934d84a700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b0f0 0x7f934819b8f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f933c000c00 tx=0x7f933c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c027070 con 0x7f934806b0f0 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f933c003f20 con 0x7f934806b0f0 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93481a0600 con 0x7f934806b0f0 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c022c70 con 0x7f934806b0f0 2024-04-21T08:43:51.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.511+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93481a7ee0 con 0x7f934806b0f0 2024-04-21T08:43:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.512+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9348063190 con 0x7f934806b0f0 2024-04-21T08:43:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.513+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f933c02a070 con 0x7f934806b0f0 2024-04-21T08:43:51.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.514+0000 7f93477fe700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93300a3cd0 0x7f93300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.514+0000 7f934e04b700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93300a3cd0 0x7f93300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.514+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f933c0d22c0 con 0x7f934806b0f0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.514+0000 7f93477fe700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] conn(0x7f93300aedd0 0x7f93300b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.514+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 --> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93300b1920 con 0x7f93300aedd0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.514+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f933c0d2710 con 0x7f934806b0f0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.515+0000 7f934e84c700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] conn(0x7f93300aedd0 0x7f93300b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.515+0000 7f934e04b700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93300a3cd0 0x7f93300a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9338009fa0 tx=0x7f9338009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.515+0000 7f934e84c700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] conn(0x7f93300aedd0 0x7f93300b1250 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:51.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.516+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== osd.3 v2:172.21.15.57:6810/3492557165 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f93300b1920 con 0x7f93300aedd0 2024-04-21T08:43:51.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 --> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9348063610 con 0x7f93300aedd0 2024-04-21T08:43:51.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93477fe700 1 -- 172.21.15.57:0/492238073 <== osd.3 v2:172.21.15.57:6810/3492557165 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9348063610 con 0x7f93300aedd0 2024-04-21T08:43:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] conn(0x7f93300aedd0 msgr2=0x7f93300b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] conn(0x7f93300aedd0 0x7f93300b1250 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93300a3cd0 msgr2=0x7f93300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f93300a3cd0 0x7f93300a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9338009fa0 tx=0x7f9338009570 comp rx=0 tx=0).stop 2024-04-21T08:43:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b0f0 msgr2=0x7f934819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:51.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f93502af700 1 --2- 172.21.15.57:0/492238073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f934806b0f0 0x7f934819b8f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f933c000c00 tx=0x7f933c005ef0 comp rx=0 tx=0).stop 2024-04-21T08:43:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.532+0000 7f934e84c700 1 -- 172.21.15.57:0/492238073 reap_dead start 2024-04-21T08:43:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.533+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 shutdown_connections 2024-04-21T08:43:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.533+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 >> 172.21.15.57:0/492238073 conn(0x7f93480788f0 msgr2=0x7f934810dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.533+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 shutdown_connections 2024-04-21T08:43:51.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:51.533+0000 7f93502af700 1 -- 172.21.15.57:0/492238073 wait complete. 2024-04-21T08:43:52.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:51 smithi129 ceph-mon[31272]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 255 B/s wr, 1 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:52.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:51 smithi057 ceph-mon[27968]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 255 B/s wr, 1 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:52.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 -- 172.21.15.57:0/3644839572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 msgr2=0x7f332c108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 --2- 172.21.15.57:0/3644839572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c108e90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f331c00b0a0 tx=0x7f331c00b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 -- 172.21.15.57:0/3644839572 shutdown_connections 2024-04-21T08:43:52.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 --2- 172.21.15.57:0/3644839572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c103480 0x7f332c10f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 --2- 172.21.15.57:0/3644839572 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c108e90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 --2- 172.21.15.57:0/3644839572 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102ac0 0x7f332c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 -- 172.21.15.57:0/3644839572 >> 172.21.15.57:0/3644839572 conn(0x7f332c079bf0 msgr2=0x7f332c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 -- 172.21.15.57:0/3644839572 shutdown_connections 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.771+0000 7f3333bdb700 1 -- 172.21.15.57:0/3644839572 wait complete. 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.772+0000 7f3333bdb700 1 Processor -- start 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.772+0000 7f3333bdb700 1 -- start start 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3333bdb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102ac0 0x7f332c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3333bdb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c103480 0x7f332c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3333bdb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3333bdb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f332c112d70 con 0x7f332c108ab0 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3333bdb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f332c112bf0 con 0x7f332c103480 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3333bdb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f332c112ef0 con 0x7f332c102ac0 2024-04-21T08:43:52.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c0757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55996/0 (socket says 172.21.15.57:55996) 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 -- 172.21.15.57:0/2373880903 learned_addr learned my addr 172.21.15.57:0/2373880903 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3331977700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102ac0 0x7f332c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 -- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102ac0 msgr2=0x7f332c074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102ac0 0x7f332c074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 -- 172.21.15.57:0/2373880903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c103480 msgr2=0x7f332c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f332c103480 0x7f332c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 -- 172.21.15.57:0/2373880903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c1a77b0 con 0x7f332c108ab0 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3331977700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c102ac0 0x7f332c074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f3332178700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c0757c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f332400b550 tx=0x7f332400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.773+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324015020 con 0x7f332c108ab0 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.774+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3324011ce0 con 0x7f332c108ab0 2024-04-21T08:43:52.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.774+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c1a7aa0 con 0x7f332c108ab0 2024-04-21T08:43:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.774+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324010d40 con 0x7f332c108ab0 2024-04-21T08:43:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.774+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c1a7fe0 con 0x7f332c108ab0 2024-04-21T08:43:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.774+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f332c063110 con 0x7f332c108ab0 2024-04-21T08:43:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.776+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3324018020 con 0x7f332c108ab0 2024-04-21T08:43:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.776+0000 7f332affd700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f33140a3cd0 0x7f33140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.776+0000 7f3331977700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f33140a3cd0 0x7f33140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.776+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f33240d0690 con 0x7f332c108ab0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.777+0000 7f332affd700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] conn(0x7f33140aedd0 0x7f33140b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.777+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 --> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3324018ec0 con 0x7f33140aedd0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.777+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f33240d0ae0 con 0x7f332c108ab0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.777+0000 7f3331977700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f33140a3cd0 0x7f33140a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f331c00b070 tx=0x7f331c0060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.777+0000 7f3331176700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] conn(0x7f33140aedd0 0x7f33140b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.779+0000 7f3331176700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] conn(0x7f33140aedd0 0x7f33140b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.780+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== osd.2 v2:172.21.15.136:6800/1481395579 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3324018ec0 con 0x7f33140aedd0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.797+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 --> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f332c063590 con 0x7f33140aedd0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.798+0000 7f332affd700 1 -- 172.21.15.57:0/2373880903 <== osd.2 v2:172.21.15.136:6800/1481395579 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f332c063590 con 0x7f33140aedd0 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.798+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] conn(0x7f33140aedd0 msgr2=0x7f33140b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.798+0000 7f3333bdb700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] conn(0x7f33140aedd0 0x7f33140b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f33140a3cd0 msgr2=0x7f33140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f33140a3cd0 0x7f33140a6190 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f331c00b070 tx=0x7f331c0060c0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 msgr2=0x7f332c0757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 --2- 172.21.15.57:0/2373880903 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f332c108ab0 0x7f332c0757c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f332400b550 tx=0x7f332400b910 comp rx=0 tx=0).stop 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3332178700 1 -- 172.21.15.57:0/2373880903 reap_dead start 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 shutdown_connections 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 >> 172.21.15.57:0/2373880903 conn(0x7f332c079bf0 msgr2=0x7f332c100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 shutdown_connections 2024-04-21T08:43:52.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:48.799+0000 7f3333bdb700 1 -- 172.21.15.57:0/2373880903 wait complete. 2024-04-21T08:43:52.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:51 smithi136 ceph-mon[31791]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 255 B/s wr, 1 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 -- 172.21.15.57:0/4292941541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558106500 msgr2=0x7f155810e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 --2- 172.21.15.57:0/4292941541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558106500 0x7f155810e040 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f155000b0a0 tx=0x7f155000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 -- 172.21.15.57:0/4292941541 shutdown_connections 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 --2- 172.21.15.57:0/4292941541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1558101660 0x7f1558101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 --2- 172.21.15.57:0/4292941541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558106500 0x7f155810e040 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 --2- 172.21.15.57:0/4292941541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1558102010 0x7f1558105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 -- 172.21.15.57:0/4292941541 >> 172.21.15.57:0/4292941541 conn(0x7f1558076c60 msgr2=0x7f1558077070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.532+0000 7f155eea8700 1 -- 172.21.15.57:0/4292941541 shutdown_connections 2024-04-21T08:43:52.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 -- 172.21.15.57:0/4292941541 wait complete. 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 Processor -- start 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 -- start start 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1558101660 0x7f1558199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558102010 0x7f1558199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1558106500 0x7f155819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1558111bc0 con 0x7f1558102010 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1558111a40 con 0x7f1558101660 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f155eea8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1558111d40 con 0x7f1558106500 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.533+0000 7f1557fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558102010 0x7f1558199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558102010 0x7f1558199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56036/0 (socket says 172.21.15.57:56036) 2024-04-21T08:43:52.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 -- 172.21.15.57:0/2495509770 learned_addr learned my addr 172.21.15.57:0/2495509770 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f155d445700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1558106500 0x7f155819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f155cc44700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1558101660 0x7f1558199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 -- 172.21.15.57:0/2495509770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1558106500 msgr2=0x7f155819d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1558106500 0x7f155819d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 -- 172.21.15.57:0/2495509770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1558101660 msgr2=0x7f1558199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1558101660 0x7f1558199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 -- 172.21.15.57:0/2495509770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155819e050 con 0x7f1558102010 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1557fff700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558102010 0x7f1558199580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f154400b4d0 tx=0x7f154400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544015020 con 0x7f1558102010 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1544011ce0 con 0x7f1558102010 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155819e340 con 0x7f1558102010 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544010cd0 con 0x7f1558102010 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.534+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15581a5bc0 con 0x7f1558102010 2024-04-21T08:43:52.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.535+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15580631d0 con 0x7f1558102010 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.536+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1544018020 con 0x7f1558102010 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f1555ffb700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f15480a3d50 0x7f15480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f155cc44700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f15480a3d50 0x7f15480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f15440cffe0 con 0x7f1558102010 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f1555ffb700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] conn(0x7f15480aee40 0x7f15480b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 --> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15480b1990 con 0x7f15480aee40 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f15440d7050 con 0x7f1558102010 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.537+0000 7f155d445700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] conn(0x7f15480aee40 0x7f15480b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.538+0000 7f155cc44700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f15480a3d50 0x7f15480a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f154c00bbd0 tx=0x7f154c00b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.538+0000 7f155d445700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] conn(0x7f15480aee40 0x7f15480b12c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:52.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.539+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== osd.6 v2:172.21.15.57:6818/961629147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15480b1990 con 0x7f15480aee40 2024-04-21T08:43:52.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.555+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 --> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1558063650 con 0x7f15480aee40 2024-04-21T08:43:52.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.555+0000 7f1555ffb700 1 -- 172.21.15.57:0/2495509770 <== osd.6 v2:172.21.15.57:6818/961629147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1558063650 con 0x7f15480aee40 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] conn(0x7f15480aee40 msgr2=0x7f15480b12c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] conn(0x7f15480aee40 0x7f15480b12c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f15480a3d50 msgr2=0x7f15480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f15480a3d50 0x7f15480a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f154c00bbd0 tx=0x7f154c00b5b0 comp rx=0 tx=0).stop 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558102010 msgr2=0x7f1558199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 --2- 172.21.15.57:0/2495509770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1558102010 0x7f1558199580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f154400b4d0 tx=0x7f154400b890 comp rx=0 tx=0).stop 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155d445700 1 -- 172.21.15.57:0/2495509770 reap_dead start 2024-04-21T08:43:52.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 shutdown_connections 2024-04-21T08:43:52.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 >> 172.21.15.57:0/2495509770 conn(0x7f1558076c60 msgr2=0x7f1558105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:52.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 shutdown_connections 2024-04-21T08:43:52.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:52.556+0000 7f155eea8700 1 -- 172.21.15.57:0/2495509770 wait complete. 2024-04-21T08:43:53.022 INFO:teuthology.orchestra.run.smithi057.stdout:163208757304 2024-04-21T08:43:53.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-21T08:43:53.959 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:53 smithi057 ceph-mon[27968]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:53.987 INFO:teuthology.orchestra.run.smithi057.stdout:77309411399 2024-04-21T08:43:53.987 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-21T08:43:54.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:53 smithi129 ceph-mon[31272]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:54.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:53 smithi136 ceph-mon[31791]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 255 B/s wr, 2 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:54.729 INFO:teuthology.orchestra.run.smithi057.stdout:107374182467 2024-04-21T08:43:54.729 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-21T08:43:55.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.890+0000 7f959ac3d700 1 -- 172.21.15.57:0/2012743403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594101960 msgr2=0x7f9594101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.890+0000 7f959ac3d700 1 --2- 172.21.15.57:0/2012743403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594101960 0x7f9594101de0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9580009990 tx=0x7f9580009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 -- 172.21.15.57:0/2012743403 shutdown_connections 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 --2- 172.21.15.57:0/2012743403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9594102320 0x7f959410e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 --2- 172.21.15.57:0/2012743403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9594107960 0x7f9594107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 --2- 172.21.15.57:0/2012743403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594101960 0x7f9594101de0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 -- 172.21.15.57:0/2012743403 >> 172.21.15.57:0/2012743403 conn(0x7f95940789e0 msgr2=0x7f9594078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 -- 172.21.15.57:0/2012743403 shutdown_connections 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 -- 172.21.15.57:0/2012743403 wait complete. 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 Processor -- start 2024-04-21T08:43:55.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.891+0000 7f959ac3d700 1 -- start start 2024-04-21T08:43:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f959ac3d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9594101960 0x7f95940fe990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f959ac3d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594102320 0x7f95940feed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f959ac3d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9594107960 0x7f95940ff410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f959ac3d700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9594111ba0 con 0x7f9594102320 2024-04-21T08:43:55.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f959ac3d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9594111a20 con 0x7f9594107960 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f959ac3d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9594111d20 con 0x7f9594101960 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594102320 0x7f95940feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594102320 0x7f95940feed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54216/0 (socket says 172.21.15.57:54216) 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 -- 172.21.15.57:0/495006479 learned_addr learned my addr 172.21.15.57:0/495006479 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f95989d9700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9594101960 0x7f95940fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f95991da700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9594107960 0x7f95940ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 -- 172.21.15.57:0/495006479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9594101960 msgr2=0x7f95940fe990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9594101960 0x7f95940fe990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 -- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9594107960 msgr2=0x7f95940ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9594107960 0x7f95940ff410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 -- 172.21.15.57:0/495006479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95941a7740 con 0x7f9594102320 2024-04-21T08:43:55.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f95989d9700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9594101960 0x7f95940fe990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9593fff700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594102320 0x7f95940feed0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9580000c00 tx=0x7f95800047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.892+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580027070 con 0x7f9594102320 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.893+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95941a79d0 con 0x7f9594102320 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.893+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95800041d0 con 0x7f9594102320 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.893+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958000f460 con 0x7f9594102320 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.893+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95941a7e80 con 0x7f9594102320 2024-04-21T08:43:55.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.893+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f95940630d0 con 0x7f9594102320 2024-04-21T08:43:55.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.895+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f958002a070 con 0x7f9594102320 2024-04-21T08:43:55.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.895+0000 7f9591ffb700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f95840a3cd0 0x7f95840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f95989d9700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f95840a3cd0 0x7f95840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f95800d1f80 con 0x7f9594102320 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f9591ffb700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] conn(0x7f95840aedd0 0x7f95840b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 --> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95800043d0 con 0x7f95840aedd0 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f95800d23d0 con 0x7f9594102320 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f95989d9700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f95840a3cd0 0x7f95840a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9588007900 tx=0x7f9588008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.896+0000 7f95991da700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] conn(0x7f95840aedd0 0x7f95840b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.897+0000 7f95991da700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] conn(0x7f95840aedd0 0x7f95840b1250 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:55.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.898+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== osd.7 v2:172.21.15.129:6816/1910048774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f95800043d0 con 0x7f95840aedd0 2024-04-21T08:43:55.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.914+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 --> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f95940634d0 con 0x7f95840aedd0 2024-04-21T08:43:55.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f9591ffb700 1 -- 172.21.15.57:0/495006479 <== osd.7 v2:172.21.15.129:6816/1910048774 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f95940634d0 con 0x7f95840aedd0 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] conn(0x7f95840aedd0 msgr2=0x7f95840b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] conn(0x7f95840aedd0 0x7f95840b1250 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f95840a3cd0 msgr2=0x7f95840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f95840a3cd0 0x7f95840a6190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9588007900 tx=0x7f9588008040 comp rx=0 tx=0).stop 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594102320 msgr2=0x7f95940feed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 --2- 172.21.15.57:0/495006479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9594102320 0x7f95940feed0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9580000c00 tx=0x7f95800047b0 comp rx=0 tx=0).stop 2024-04-21T08:43:55.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f95991da700 1 -- 172.21.15.57:0/495006479 reap_dead start 2024-04-21T08:43:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 shutdown_connections 2024-04-21T08:43:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 >> 172.21.15.57:0/495006479 conn(0x7f95940789e0 msgr2=0x7f959406b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 shutdown_connections 2024-04-21T08:43:55.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:55.915+0000 7f959ac3d700 1 -- 172.21.15.57:0/495006479 wait complete. 2024-04-21T08:43:56.000 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:55 smithi057 ceph-mon[27968]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1006 B/s rd, 223 B/s wr, 1 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:56.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:55 smithi129 ceph-mon[31272]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1006 B/s rd, 223 B/s wr, 1 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:56.134 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:55 smithi136 ceph-mon[31791]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1006 B/s rd, 223 B/s wr, 1 op/s; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-04-21T08:43:57.036 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:56 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:57.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:56 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:57.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:56 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:43:57.241 INFO:teuthology.orchestra.run.smithi057.stdout:219043332141 2024-04-21T08:43:57.242 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-21T08:43:57.981 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:57 smithi057 ceph-mon[27968]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 705 B/s rd, 1 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:43:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:57 smithi129 ceph-mon[31272]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 705 B/s rd, 1 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:43:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:57 smithi136 ceph-mon[31791]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 705 B/s rd, 1 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:43:59.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 -- 172.21.15.57:0/1735187470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 msgr2=0x7f832406b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 --2- 172.21.15.57:0/1735187470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f832406b5d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8314009990 tx=0x7f8314009ca0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 -- 172.21.15.57:0/1735187470 shutdown_connections 2024-04-21T08:43:59.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 --2- 172.21.15.57:0/1735187470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8324108a30 0x7f8324108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 --2- 172.21.15.57:0/1735187470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f832406bb10 0x7f832410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 --2- 172.21.15.57:0/1735187470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f832406b5d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 -- 172.21.15.57:0/1735187470 >> 172.21.15.57:0/1735187470 conn(0x7f83240788f0 msgr2=0x7f8324078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 -- 172.21.15.57:0/1735187470 shutdown_connections 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.092+0000 7f832c2ac700 1 -- 172.21.15.57:0/1735187470 wait complete. 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 Processor -- start 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 -- start start 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f83241990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f832406bb10 0x7f83241995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8324108a30 0x7f832419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8324112bd0 con 0x7f832406b170 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8324112a50 con 0x7f832406bb10 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832c2ac700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8324112d50 con 0x7f8324108a30 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f83241990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f83241990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54244/0 (socket says 172.21.15.57:54244) 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 -- 172.21.15.57:0/3875139881 learned_addr learned my addr 172.21.15.57:0/3875139881 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f8329847700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f832406bb10 0x7f83241995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a849700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8324108a30 0x7f832419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 -- 172.21.15.57:0/3875139881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8324108a30 msgr2=0x7f832419d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8324108a30 0x7f832419d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 -- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f832406bb10 msgr2=0x7f83241995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f832406bb10 0x7f83241995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.093+0000 7f832a048700 1 -- 172.21.15.57:0/3875139881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832419e0b0 con 0x7f832406b170 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.094+0000 7f832a048700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f83241990a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f832000d980 tx=0x7f832000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.094+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320016070 con 0x7f832406b170 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.094+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832419e3a0 con 0x7f832406b170 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.094+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83241a5c20 con 0x7f832406b170 2024-04-21T08:43:59.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.094+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8320004830 con 0x7f832406b170 2024-04-21T08:43:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.094+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200054c0 con 0x7f832406b170 2024-04-21T08:43:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.095+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8308001030 con 0x7f832406b170 2024-04-21T08:43:59.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.097+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8320004220 con 0x7f832406b170 2024-04-21T08:43:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.098+0000 7f831b7fe700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f83100a3cd0 0x7f83100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.098+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f83200d0570 con 0x7f832406b170 2024-04-21T08:43:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f8329847700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f83100a3cd0 0x7f83100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f831b7fe700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] conn(0x7f83100aedd0 0x7f83100b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 --> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83100b1920 con 0x7f83100aedd0 2024-04-21T08:43:59.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f83200d7050 con 0x7f832406b170 2024-04-21T08:43:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f832a849700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] conn(0x7f83100aedd0 0x7f83100b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f8329847700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f83100a3cd0 0x7f83100a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83140098e0 tx=0x7f8314005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.099+0000 7f832a849700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] conn(0x7f83100aedd0 0x7f83100b1250 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:59.099 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.100+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== osd.4 v2:172.21.15.129:6808/757493638 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83100b1920 con 0x7f83100aedd0 2024-04-21T08:43:59.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.117+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 --> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8308002e60 con 0x7f83100aedd0 2024-04-21T08:43:59.117 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f831b7fe700 1 -- 172.21.15.57:0/3875139881 <== osd.4 v2:172.21.15.129:6808/757493638 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8308002e60 con 0x7f83100aedd0 2024-04-21T08:43:59.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] conn(0x7f83100aedd0 msgr2=0x7f83100b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] conn(0x7f83100aedd0 0x7f83100b1250 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f83100a3cd0 msgr2=0x7f83100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f83100a3cd0 0x7f83100a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83140098e0 tx=0x7f8314005960 comp rx=0 tx=0).stop 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 msgr2=0x7f83241990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 --2- 172.21.15.57:0/3875139881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f832406b170 0x7f83241990a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f832000d980 tx=0x7f832000dc90 comp rx=0 tx=0).stop 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832a849700 1 -- 172.21.15.57:0/3875139881 reap_dead start 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 shutdown_connections 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 >> 172.21.15.57:0/3875139881 conn(0x7f83240788f0 msgr2=0x7f83240ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 shutdown_connections 2024-04-21T08:43:59.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.118+0000 7f832c2ac700 1 -- 172.21.15.57:0/3875139881 wait complete. 2024-04-21T08:43:59.197 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.197 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.197 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.198 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:58 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.281 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.281 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.281 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.281 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:58 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:58 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 -- 172.21.15.57:0/774814745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10103480 msgr2=0x7fab1010f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 --2- 172.21.15.57:0/774814745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10103480 0x7fab1010f980 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fab0800b0a0 tx=0x7fab0800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 -- 172.21.15.57:0/774814745 shutdown_connections 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 --2- 172.21.15.57:0/774814745 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10103480 0x7fab1010f980 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 --2- 172.21.15.57:0/774814745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab10108ab0 0x7fab10108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 --2- 172.21.15.57:0/774814745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab10102ac0 0x7fab10102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 -- 172.21.15.57:0/774814745 >> 172.21.15.57:0/774814745 conn(0x7fab10079bf0 msgr2=0x7fab1007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 -- 172.21.15.57:0/774814745 shutdown_connections 2024-04-21T08:43:59.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.469+0000 7fab14a11700 1 -- 172.21.15.57:0/774814745 wait complete. 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 Processor -- start 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 -- start start 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10102ac0 0x7fab10074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab10103480 0x7fab10075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab10108ab0 0x7fab100757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fab10112cf0 con 0x7fab10102ac0 2024-04-21T08:43:59.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab10112b70 con 0x7fab10108ab0 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab14a11700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fab10112e70 con 0x7fab10103480 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab0e59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10102ac0 0x7fab10074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab0e59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10102ac0 0x7fab10074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54280/0 (socket says 172.21.15.57:54280) 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab0e59c700 1 -- 172.21.15.57:0/288471229 learned_addr learned my addr 172.21.15.57:0/288471229 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.470+0000 7fab0dd9b700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab10103480 0x7fab10075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab0e59c700 1 -- 172.21.15.57:0/288471229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab10103480 msgr2=0x7fab10075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab0e59c700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab10103480 0x7fab10075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab0e59c700 1 -- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab10108ab0 msgr2=0x7fab100757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab0e59c700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab10108ab0 0x7fab100757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab0e59c700 1 -- 172.21.15.57:0/288471229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab101a7750 con 0x7fab10102ac0 2024-04-21T08:43:59.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab0e59c700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10102ac0 0x7fab10074d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b530 tx=0x7fab0000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00015020 con 0x7fab10102ac0 2024-04-21T08:43:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab00011ce0 con 0x7fab10102ac0 2024-04-21T08:43:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab101a7a40 con 0x7fab10102ac0 2024-04-21T08:43:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab101a7f80 con 0x7fab10102ac0 2024-04-21T08:43:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.471+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00010d20 con 0x7fab10102ac0 2024-04-21T08:43:59.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.472+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab10063110 con 0x7fab10102ac0 2024-04-21T08:43:59.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.473+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fab00018020 con 0x7fab10102ac0 2024-04-21T08:43:59.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.474+0000 7faaff7fe700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7faaf80a3d50 0x7faaf80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.474+0000 7fab0dd9b700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7faaf80a3d50 0x7faaf80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.474+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fab000d06d0 con 0x7fab10102ac0 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.474+0000 7faaff7fe700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] conn(0x7faaf80aee50 0x7faaf80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.474+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 --> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaf80b19a0 con 0x7faaf80aee50 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.474+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fab000d0b20 con 0x7fab10102ac0 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.475+0000 7fab0ed9d700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] conn(0x7faaf80aee50 0x7faaf80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.475+0000 7fab0dd9b700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7faaf80a3d50 0x7faaf80a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab04005fd0 tx=0x7fab04005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.475+0000 7fab0ed9d700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] conn(0x7faaf80aee50 0x7faaf80b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:43:59.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.476+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== osd.10 v2:172.21.15.129:6824/1559005411 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faaf80b19a0 con 0x7faaf80aee50 2024-04-21T08:43:59.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.492+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 --> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fab10063510 con 0x7faaf80aee50 2024-04-21T08:43:59.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7faaff7fe700 1 -- 172.21.15.57:0/288471229 <== osd.10 v2:172.21.15.129:6824/1559005411 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fab10063510 con 0x7faaf80aee50 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] conn(0x7faaf80aee50 msgr2=0x7faaf80b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] conn(0x7faaf80aee50 0x7faaf80b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7faaf80a3d50 msgr2=0x7faaf80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7faaf80a3d50 0x7faaf80a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab04005fd0 tx=0x7fab04005f60 comp rx=0 tx=0).stop 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10102ac0 msgr2=0x7fab10074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 --2- 172.21.15.57:0/288471229 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab10102ac0 0x7fab10074d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab0000b530 tx=0x7fab0000b8f0 comp rx=0 tx=0).stop 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab0ed9d700 1 -- 172.21.15.57:0/288471229 reap_dead start 2024-04-21T08:43:59.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 shutdown_connections 2024-04-21T08:43:59.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 >> 172.21.15.57:0/288471229 conn(0x7fab10079bf0 msgr2=0x7fab10100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:43:59.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 shutdown_connections 2024-04-21T08:43:59.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.493+0000 7fab14a11700 1 -- 172.21.15.57:0/288471229 wait complete. 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 -- 172.21.15.57:0/3937416361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 msgr2=0x7f5100101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 --2- 172.21.15.57:0/3937416361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f5100101db0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009990 tx=0x7f50f0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 -- 172.21.15.57:0/3937416361 shutdown_connections 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 --2- 172.21.15.57:0/3937416361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51001022f0 0x7f510010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 --2- 172.21.15.57:0/3937416361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5100107980 0x7f5100107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 --2- 172.21.15.57:0/3937416361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f5100101db0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 -- 172.21.15.57:0/3937416361 >> 172.21.15.57:0/3937416361 conn(0x7f51000788b0 msgr2=0x7f5100078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:00.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.333+0000 7f5108627700 1 -- 172.21.15.57:0/3937416361 shutdown_connections 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 -- 172.21.15.57:0/3937416361 wait complete. 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 Processor -- start 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 -- start start 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f510019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51001022f0 0x7f510019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5100107980 0x7f51001a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5100111af0 con 0x7f5100101930 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5100111970 con 0x7f5100107980 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.334+0000 7f5108627700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5100111c70 con 0x7f51001022f0 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f510019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f510019a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54256/0 (socket says 172.21.15.57:54256) 2024-04-21T08:44:00.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 -- 172.21.15.57:0/1273768501 learned_addr learned my addr 172.21.15.57:0/1273768501 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f5106bc4700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5100107980 0x7f51001a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f5105bc2700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51001022f0 0x7f510019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 -- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51001022f0 msgr2=0x7f510019a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51001022f0 0x7f510019a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 -- 172.21.15.57:0/1273768501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5100107980 msgr2=0x7f51001a50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5100107980 0x7f51001a50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 -- 172.21.15.57:0/1273768501 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51001a5620 con 0x7f5100101930 2024-04-21T08:44:00.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f5106bc4700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5100107980 0x7f51001a50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f51063c3700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f510019a060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00d980 tx=0x7f50fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc017070 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50fc005020 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51001a5850 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc00fa10 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.335+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51001a5d10 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.336+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51000630d0 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.338+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f50fc004a30 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.338+0000 7f50f77fe700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f50ec0a3cd0 0x7f50ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f5105bc2700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f50ec0a3cd0 0x7f50ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f50fc0d0c90 con 0x7f5100101930 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f50f77fe700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] conn(0x7f50ec0aedd0 0x7f50ec0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 --> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50ec0b1920 con 0x7f50ec0aedd0 2024-04-21T08:44:00.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f50fc0d1040 con 0x7f5100101930 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f5106bc4700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] conn(0x7f50ec0aedd0 0x7f50ec0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f5105bc2700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f50ec0a3cd0 0x7f50ec0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b5c0 tx=0x7f50f0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.339+0000 7f5106bc4700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] conn(0x7f50ec0aedd0 0x7f50ec0b1250 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.340+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== osd.8 v2:172.21.15.136:6816/94445430 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50ec0b1920 con 0x7f50ec0aedd0 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.358+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 --> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5100063500 con 0x7f50ec0aedd0 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f50f77fe700 1 -- 172.21.15.57:0/1273768501 <== osd.8 v2:172.21.15.136:6816/94445430 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5100063500 con 0x7f50ec0aedd0 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] conn(0x7f50ec0aedd0 msgr2=0x7f50ec0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] conn(0x7f50ec0aedd0 0x7f50ec0b1250 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f50ec0a3cd0 msgr2=0x7f50ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f50ec0a3cd0 0x7f50ec0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f50f000b5c0 tx=0x7f50f0005960 comp rx=0 tx=0).stop 2024-04-21T08:44:00.069 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:43:59 smithi057 ceph-mon[27968]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 msgr2=0x7f510019a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 --2- 172.21.15.57:0/1273768501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5100101930 0x7f510019a060 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00d980 tx=0x7f50fc00dc90 comp rx=0 tx=0).stop 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5106bc4700 1 -- 172.21.15.57:0/1273768501 reap_dead start 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 shutdown_connections 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.359+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 >> 172.21.15.57:0/1273768501 conn(0x7f51000788b0 msgr2=0x7f51000ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.360+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 shutdown_connections 2024-04-21T08:44:00.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:43:59.360+0000 7f5108627700 1 -- 172.21.15.57:0/1273768501 wait complete. 2024-04-21T08:44:00.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:43:59 smithi129 ceph-mon[31272]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:44:00.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:43:59 smithi136 ceph-mon[31791]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-21T08:44:01.089 INFO:teuthology.orchestra.run.smithi057.stdout:309237645343 2024-04-21T08:44:01.089 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-21T08:44:01.275 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 -- 172.21.15.57:0/3955889426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8102d60 msgr2=0x7f04a81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.275 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 --2- 172.21.15.57:0/3955889426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8102d60 0x7f04a81031e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f049c009990 tx=0x7f049c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 -- 172.21.15.57:0/3955889426 shutdown_connections 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 --2- 172.21.15.57:0/3955889426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103720 0x7f04a810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 --2- 172.21.15.57:0/3955889426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04a8108bc0 0x7f04a8108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 --2- 172.21.15.57:0/3955889426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8102d60 0x7f04a81031e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 -- 172.21.15.57:0/3955889426 >> 172.21.15.57:0/3955889426 conn(0x7f04a807a9d0 msgr2=0x7f04a807ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 -- 172.21.15.57:0/3955889426 shutdown_connections 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.275+0000 7f04aea15700 1 -- 172.21.15.57:0/3955889426 wait complete. 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.276+0000 7f04aea15700 1 Processor -- start 2024-04-21T08:44:01.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.276+0000 7f04aea15700 1 -- start start 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04aea15700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04a8102d60 0x7f04a8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04aea15700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103720 0x7f04a8075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04aea15700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8108bc0 0x7f04a80757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04aea15700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f04a8112cc0 con 0x7f04a8108bc0 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04aea15700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04a8112b40 con 0x7f04a8103720 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04aea15700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f04a8112e40 con 0x7f04a8102d60 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8108bc0 0x7f04a80757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8108bc0 0x7f04a80757c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54306/0 (socket says 172.21.15.57:54306) 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 -- 172.21.15.57:0/2172003188 learned_addr learned my addr 172.21.15.57:0/2172003188 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04a77fe700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103720 0x7f04a8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04a7fff700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04a8102d60 0x7f04a8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 -- 172.21.15.57:0/2172003188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04a8102d60 msgr2=0x7f04a8074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04a8102d60 0x7f04a8074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 -- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103720 msgr2=0x7f04a8075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04a8103720 0x7f04a8075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 -- 172.21.15.57:0/2172003188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a81a7640 con 0x7f04a8108bc0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04acfb2700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8108bc0 0x7f04a80757c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f04a000efc0 tx=0x7f04a000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.277+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a00092e0 con 0x7f04a8108bc0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.278+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a81a78d0 con 0x7f04a8108bc0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.278+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04a0004bb0 con 0x7f04a8108bc0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.278+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0004020 con 0x7f04a8108bc0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.278+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a81a7d90 con 0x7f04a8108bc0 2024-04-21T08:44:01.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.278+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04a8063150 con 0x7f04a8108bc0 2024-04-21T08:44:01.280 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.280+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f04a001b030 con 0x7f04a8108bc0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.280+0000 7f04a57fa700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f04900a3cd0 0x7f04900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a7fff700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f04900a3cd0 0x7f04900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f04a00d0c30 con 0x7f04a8108bc0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a57fa700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] conn(0x7f04900aedd0 0x7f04900b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 --> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04a0013ec0 con 0x7f04900aedd0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f04a00d1080 con 0x7f04a8108bc0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a7fff700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f04900a3cd0 0x7f04900a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0498005fd0 tx=0x7f0498005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.281+0000 7f04a77fe700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] conn(0x7f04900aedd0 0x7f04900b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.282+0000 7f04a77fe700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] conn(0x7f04900aedd0 0x7f04900b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.283+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== osd.1 v2:172.21.15.129:6800/3548590037 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04a0013ec0 con 0x7f04900aedd0 2024-04-21T08:44:01.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.299+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 --> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f04a80635d0 con 0x7f04900aedd0 2024-04-21T08:44:01.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04a57fa700 1 -- 172.21.15.57:0/2172003188 <== osd.1 v2:172.21.15.129:6800/3548590037 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f04a80635d0 con 0x7f04900aedd0 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] conn(0x7f04900aedd0 msgr2=0x7f04900b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] conn(0x7f04900aedd0 0x7f04900b1250 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f04900a3cd0 msgr2=0x7f04900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f04900a3cd0 0x7f04900a6190 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0498005fd0 tx=0x7f0498005e50 comp rx=0 tx=0).stop 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8108bc0 msgr2=0x7f04a80757c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 --2- 172.21.15.57:0/2172003188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04a8108bc0 0x7f04a80757c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f04a000efc0 tx=0x7f04a000c3a0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04acfb2700 1 -- 172.21.15.57:0/2172003188 reap_dead start 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 shutdown_connections 2024-04-21T08:44:01.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 >> 172.21.15.57:0/2172003188 conn(0x7f04a807a9d0 msgr2=0x7f04a81019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 shutdown_connections 2024-04-21T08:44:01.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.300+0000 7f04aea15700 1 -- 172.21.15.57:0/2172003188 wait complete. 2024-04-21T08:44:01.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.943+0000 7f3b7b63a700 1 -- 172.21.15.57:0/2826052814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 msgr2=0x7f3b74108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.943+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/2826052814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74108e10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009990 tx=0x7f3b70009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 -- 172.21.15.57:0/2826052814 shutdown_connections 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/2826052814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b7406b0f0 0x7f3b7410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/2826052814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74108e10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/2826052814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b7406a750 0x7f3b7406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 -- 172.21.15.57:0/2826052814 >> 172.21.15.57:0/2826052814 conn(0x7f3b740788f0 msgr2=0x7f3b74078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 -- 172.21.15.57:0/2826052814 shutdown_connections 2024-04-21T08:44:01.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 -- 172.21.15.57:0/2826052814 wait complete. 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 Processor -- start 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.944+0000 7f3b7b63a700 1 -- start start 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b7406a750 0x7f3b74101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b7406b0f0 0x7f3b74101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74112bd0 con 0x7f3b74108a30 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74112a50 con 0x7f3b7406b0f0 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3b74112d50 con 0x7f3b7406a750 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74102490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54328/0 (socket says 172.21.15.57:54328) 2024-04-21T08:44:01.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 -- 172.21.15.57:0/3331691320 learned_addr learned my addr 172.21.15.57:0/3331691320 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b793d6700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b7406a750 0x7f3b74101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b78bd5700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b7406b0f0 0x7f3b74101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 -- 172.21.15.57:0/3331691320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b7406a750 msgr2=0x7f3b74101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b7406a750 0x7f3b74101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 -- 172.21.15.57:0/3331691320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b7406b0f0 msgr2=0x7f3b74101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b7406b0f0 0x7f3b74101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 -- 172.21.15.57:0/3331691320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b741a7650 con 0x7f3b74108a30 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b78bd5700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b7406b0f0 0x7f3b74101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b79bd7700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74102490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007ee0 tx=0x7f3b6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00dbd0 con 0x7f3b74108a30 2024-04-21T08:44:01.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.945+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b741a7940 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.946+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b6c00dd70 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.946+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c005990 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.946+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b741a7e80 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.946+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b74063110 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.948+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3b6c01a020 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.948+0000 7f3b6a7fc700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3b600a3cd0 0x7f3b600a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.948+0000 7f3b793d6700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3b600a3cd0 0x7f3b600a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.948+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f3b6c0d0660 con 0x7f3b74108a30 2024-04-21T08:44:01.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.949+0000 7f3b6a7fc700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] conn(0x7f3b600aedd0 0x7f3b600b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.949+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 --> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b600b1920 con 0x7f3b600aedd0 2024-04-21T08:44:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.949+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f3b6c0d0ab0 con 0x7f3b74108a30 2024-04-21T08:44:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.949+0000 7f3b78bd5700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] conn(0x7f3b600aedd0 0x7f3b600b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.949+0000 7f3b793d6700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3b600a3cd0 0x7f3b600a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3b700098e0 tx=0x7f3b70005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.949+0000 7f3b78bd5700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] conn(0x7f3b600aedd0 0x7f3b600b1250 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:01.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.950+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== osd.0 v2:172.21.15.57:6802/1351441903 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b600b1920 con 0x7f3b600aedd0 2024-04-21T08:44:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.966+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 --> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3b74063590 con 0x7f3b600aedd0 2024-04-21T08:44:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b6a7fc700 1 -- 172.21.15.57:0/3331691320 <== osd.0 v2:172.21.15.57:6802/1351441903 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3b74063590 con 0x7f3b600aedd0 2024-04-21T08:44:01.968 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] conn(0x7f3b600aedd0 msgr2=0x7f3b600b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] conn(0x7f3b600aedd0 0x7f3b600b1250 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3b600a3cd0 msgr2=0x7f3b600a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3b600a3cd0 0x7f3b600a6190 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3b700098e0 tx=0x7f3b70005960 comp rx=0 tx=0).stop 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 msgr2=0x7f3b74102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 --2- 172.21.15.57:0/3331691320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b74108a30 0x7f3b74102490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007ee0 tx=0x7f3b6c00d3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b79bd7700 1 -- 172.21.15.57:0/3331691320 reap_dead start 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 shutdown_connections 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 >> 172.21.15.57:0/3331691320 conn(0x7f3b740788f0 msgr2=0x7f3b740ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 shutdown_connections 2024-04-21T08:44:01.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:01.967+0000 7f3b7b63a700 1 -- 172.21.15.57:0/3331691320 wait complete. 2024-04-21T08:44:02.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:01 smithi057 ceph-mon[27968]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:02.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:01 smithi129 ceph-mon[31272]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:01 smithi136 ceph-mon[31791]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:02.570 INFO:teuthology.orchestra.run.smithi057.stdout:253403070507 2024-04-21T08:44:02.571 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-21T08:44:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 -- 172.21.15.57:0/485456135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806a750 msgr2=0x7fb9c806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 --2- 172.21.15.57:0/485456135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806a750 0x7fb9c806abb0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009990 tx=0x7fb9bc009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 -- 172.21.15.57:0/485456135 shutdown_connections 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 --2- 172.21.15.57:0/485456135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c8108a30 0x7fb9c8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 --2- 172.21.15.57:0/485456135 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806b0f0 0x7fb9c810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 --2- 172.21.15.57:0/485456135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806a750 0x7fb9c806abb0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 -- 172.21.15.57:0/485456135 >> 172.21.15.57:0/485456135 conn(0x7fb9c80788f0 msgr2=0x7fb9c8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 -- 172.21.15.57:0/485456135 shutdown_connections 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.758+0000 7fb9ccdea700 1 -- 172.21.15.57:0/485456135 wait complete. 2024-04-21T08:44:02.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.759+0000 7fb9ccdea700 1 Processor -- start 2024-04-21T08:44:02.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 -- start start 2024-04-21T08:44:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806a750 0x7fb9c8199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806b0f0 0x7fb9c8199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c8108a30 0x7fb9c819da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8112c10 con 0x7fb9c806b0f0 2024-04-21T08:44:02.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8112a90 con 0x7fb9c8108a30 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9ccdea700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c8112d90 con 0x7fb9c806a750 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9c659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806a750 0x7fb9c8199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9c5d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806b0f0 0x7fb9c8199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.761+0000 7fb9c659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806a750 0x7fb9c8199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57494/0 (socket says 172.21.15.57:57494) 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 -- 172.21.15.57:0/3693147970 learned_addr learned my addr 172.21.15.57:0/3693147970 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c6d9d700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c8108a30 0x7fb9c819da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 -- 172.21.15.57:0/3693147970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c8108a30 msgr2=0x7fb9c819da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c8108a30 0x7fb9c819da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 -- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806b0f0 msgr2=0x7fb9c8199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9c806b0f0 0x7fb9c8199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 -- 172.21.15.57:0/3693147970 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c819e150 con 0x7fb9c806a750 2024-04-21T08:44:02.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.762+0000 7fb9c659c700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806a750 0x7fb9c8199140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800d980 tx=0x7fb9b800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.763+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8016070 con 0x7fb9c806a750 2024-04-21T08:44:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.763+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9b8004830 con 0x7fb9c806a750 2024-04-21T08:44:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.763+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8005540 con 0x7fb9c806a750 2024-04-21T08:44:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.763+0000 7fb9ccdea700 1 -- 172.21.15.57:0/3693147970 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c819e440 con 0x7fb9c806a750 2024-04-21T08:44:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.763+0000 7fb9ccdea700 1 -- 172.21.15.57:0/3693147970 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c8101940 con 0x7fb9c806a750 2024-04-21T08:44:02.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.764+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9c80631d0 con 0x7fb9c806a750 2024-04-21T08:44:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.767+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb9b8004220 con 0x7fb9c806a750 2024-04-21T08:44:02.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.767+0000 7fb9b77fe700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb9b00a8160 0x7fb9b00aa620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.767+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fb9b80d0c40 con 0x7fb9c806a750 2024-04-21T08:44:02.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.768+0000 7fb9c5d9b700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb9b00a8160 0x7fb9b00aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.768+0000 7fb9b77fe700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] conn(0x7fb9b00b3260 0x7fb9b00b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.768+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 --> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9b8015ec0 con 0x7fb9b00b3260 2024-04-21T08:44:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.768+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb9b80d1090 con 0x7fb9c806a750 2024-04-21T08:44:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.768+0000 7fb9c5d9b700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb9b00a8160 0x7fb9b00aa620 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00b5c0 tx=0x7fb9bc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.768+0000 7fb9c6d9d700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] conn(0x7fb9b00b3260 0x7fb9b00b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:02.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.769+0000 7fb9c6d9d700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] conn(0x7fb9b00b3260 0x7fb9b00b56e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:02.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.770+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== osd.11 v2:172.21.15.136:6824/4048591529 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9b8015ec0 con 0x7fb9b00b3260 2024-04-21T08:44:02.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.793+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 --> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb9c80636a0 con 0x7fb9b00b3260 2024-04-21T08:44:02.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b77fe700 1 -- 172.21.15.57:0/3693147970 <== osd.11 v2:172.21.15.136:6824/4048591529 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb9c80636a0 con 0x7fb9b00b3260 2024-04-21T08:44:02.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] conn(0x7fb9b00b3260 msgr2=0x7fb9b00b56e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] conn(0x7fb9b00b3260 0x7fb9b00b56e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:02.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb9b00a8160 msgr2=0x7fb9b00aa620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fb9b00a8160 0x7fb9b00aa620 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00b5c0 tx=0x7fb9bc005960 comp rx=0 tx=0).stop 2024-04-21T08:44:02.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806a750 msgr2=0x7fb9c8199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:02.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 --2- 172.21.15.57:0/3693147970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9c806a750 0x7fb9c8199140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800d980 tx=0x7fb9b800dc90 comp rx=0 tx=0).stop 2024-04-21T08:44:02.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9c6d9d700 1 -- 172.21.15.57:0/3693147970 reap_dead start 2024-04-21T08:44:02.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 shutdown_connections 2024-04-21T08:44:02.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 >> 172.21.15.57:0/3693147970 conn(0x7fb9c80788f0 msgr2=0x7fb9c80ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:02.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 shutdown_connections 2024-04-21T08:44:02.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:02.794+0000 7fb9b57fa700 1 -- 172.21.15.57:0/3693147970 wait complete. 2024-04-21T08:44:03.755 INFO:teuthology.orchestra.run.smithi057.stdout:197568495668 2024-04-21T08:44:03.755 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-21T08:44:04.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:03 smithi129 ceph-mon[31272]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:04.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:03 smithi057 ceph-mon[27968]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:04.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:03 smithi136 ceph-mon[31791]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:04.444 INFO:teuthology.orchestra.run.smithi057.stdout:343597383706 2024-04-21T08:44:04.444 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-21T08:44:04.863 INFO:teuthology.orchestra.run.smithi057.stdout:55834574927 2024-04-21T08:44:04.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-21T08:44:05.405 INFO:teuthology.orchestra.run.smithi057.stdout:34359738451 2024-04-21T08:44:05.406 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-21T08:44:05.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 -- 172.21.15.57:0/4222791188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130103450 msgr2=0x7f413010f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 --2- 172.21.15.57:0/4222791188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130103450 0x7f413010f8b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f412800b0a0 tx=0x7f412800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 -- 172.21.15.57:0/4222791188 shutdown_connections 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 --2- 172.21.15.57:0/4222791188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130103450 0x7f413010f8b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 --2- 172.21.15.57:0/4222791188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4130108a50 0x7f4130108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 --2- 172.21.15.57:0/4222791188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130102a90 0x7f4130102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 -- 172.21.15.57:0/4222791188 >> 172.21.15.57:0/4222791188 conn(0x7f4130079ba0 msgr2=0x7f4130079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 -- 172.21.15.57:0/4222791188 shutdown_connections 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.800+0000 7f41353c1700 1 -- 172.21.15.57:0/4222791188 wait complete. 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 Processor -- start 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 -- start start 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130102a90 0x7f41301990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103450 0x7f4130199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4130108a50 0x7f413019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4130112c30 con 0x7f4130102a90 2024-04-21T08:44:05.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4130112ab0 con 0x7f4130103450 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f41353c1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4130112db0 con 0x7f4130108a50 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f412effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130102a90 0x7f41301990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f412effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130102a90 0x7f41301990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33648/0 (socket says 172.21.15.57:33648) 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.801+0000 7f412effd700 1 -- 172.21.15.57:0/4041241833 learned_addr learned my addr 172.21.15.57:0/4041241833 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412f7fe700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4130108a50 0x7f413019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412e7fc700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103450 0x7f4130199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412effd700 1 -- 172.21.15.57:0/4041241833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4130108a50 msgr2=0x7f413019d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412effd700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4130108a50 0x7f413019d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412effd700 1 -- 172.21.15.57:0/4041241833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103450 msgr2=0x7f4130199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412effd700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103450 0x7f4130199600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412effd700 1 -- 172.21.15.57:0/4041241833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413019e0d0 con 0x7f4130102a90 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412e7fc700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4130103450 0x7f4130199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f412effd700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130102a90 0x7f41301990c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f412000d950 tx=0x7f412000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120016070 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4120009df0 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413019e3c0 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412000f3c0 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.802+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41301a5c40 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.803+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4130063150 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.804+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4120019070 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f4117fff700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f41180a3cd0 0x7f41180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f41200d09c0 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f412e7fc700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f41180a3cd0 0x7f41180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f4117fff700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] conn(0x7f41180aedd0 0x7f41180b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 --> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4120013ec0 con 0x7f41180aedd0 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f41200d0d70 con 0x7f4130102a90 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f412f7fe700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] conn(0x7f41180aedd0 0x7f41180b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:05.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.805+0000 7f412e7fc700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f41180a3cd0 0x7f41180a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4124005fd0 tx=0x7f4124005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.806+0000 7f412f7fe700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] conn(0x7f41180aedd0 0x7f41180b1250 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:05.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.806+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== osd.9 v2:172.21.15.57:6826/4116200547 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4120013ec0 con 0x7f41180aedd0 2024-04-21T08:44:05.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.823+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 --> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41300635d0 con 0x7f41180aedd0 2024-04-21T08:44:05.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f4117fff700 1 -- 172.21.15.57:0/4041241833 <== osd.9 v2:172.21.15.57:6826/4116200547 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f41300635d0 con 0x7f41180aedd0 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] conn(0x7f41180aedd0 msgr2=0x7f41180b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] conn(0x7f41180aedd0 0x7f41180b1250 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f41180a3cd0 msgr2=0x7f41180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f41180a3cd0 0x7f41180a6190 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4124005fd0 tx=0x7f4124005e20 comp rx=0 tx=0).stop 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130102a90 msgr2=0x7f41301990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 --2- 172.21.15.57:0/4041241833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4130102a90 0x7f41301990c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f412000d950 tx=0x7f412000dc60 comp rx=0 tx=0).stop 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f412f7fe700 1 -- 172.21.15.57:0/4041241833 reap_dead start 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 shutdown_connections 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 >> 172.21.15.57:0/4041241833 conn(0x7f4130079ba0 msgr2=0x7f41301017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:05.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.824+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 shutdown_connections 2024-04-21T08:44:05.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:05.825+0000 7f41353c1700 1 -- 172.21.15.57:0/4041241833 wait complete. 2024-04-21T08:44:06.242 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:05 smithi057 ceph-mon[27968]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:06.244 INFO:teuthology.orchestra.run.smithi057.stdout:141733920830 2024-04-21T08:44:06.244 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-21T08:44:06.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:05 smithi129 ceph-mon[31272]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:06.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:05 smithi136 ceph-mon[31791]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:06.950 INFO:teuthology.orchestra.run.smithi057.stdout:287762808870 2024-04-21T08:44:06.950 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-21T08:44:06.980 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:06.991 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:07.820 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:07.820 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:07.827 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:08.154 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:08.154 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:08.155 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:08.158 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:08.279 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:07 smithi057 ceph-mon[27968]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:08.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:07 smithi129 ceph-mon[31272]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:08.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:07 smithi136 ceph-mon[31791]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:10.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:10 smithi129 ceph-mon[31272]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:10.326 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:10 smithi057 ceph-mon[27968]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:10.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:10 smithi136 ceph-mon[31791]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:12.215 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:12 smithi057 ceph-mon[27968]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:12.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:12 smithi129 ceph-mon[31272]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:12.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:12 smithi136 ceph-mon[31791]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:14.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:14 smithi129 ceph-mon[31272]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:14.325 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:14 smithi057 ceph-mon[27968]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:14.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:14 smithi136 ceph-mon[31791]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 -- 172.21.15.57:0/1768468369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664074c70 msgr2=0x7fd664075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 --2- 172.21.15.57:0/1768468369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664074c70 0x7fd664075050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd654009990 tx=0x7fd654009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 -- 172.21.15.57:0/1768468369 shutdown_connections 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 --2- 172.21.15.57:0/1768468369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd66410f440 0x7fd664111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 --2- 172.21.15.57:0/1768468369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664074c70 0x7fd664075050 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 --2- 172.21.15.57:0/1768468369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd664075590 0x7fd66410ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 -- 172.21.15.57:0/1768468369 >> 172.21.15.57:0/1768468369 conn(0x7fd664079bf0 msgr2=0x7fd66407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 -- 172.21.15.57:0/1768468369 shutdown_connections 2024-04-21T08:44:15.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.198+0000 7fd6696f8700 1 -- 172.21.15.57:0/1768468369 wait complete. 2024-04-21T08:44:15.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.199+0000 7fd6696f8700 1 Processor -- start 2024-04-21T08:44:15.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.199+0000 7fd6696f8700 1 -- start start 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd664074c70 0x7fd66419b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 0x7fd66419b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66410f440 0x7fd66419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd664114e40 con 0x7fd664075590 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd664114cc0 con 0x7fd664074c70 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd664114fc0 con 0x7fd66410f440 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 0x7fd66419b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 0x7fd66419b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57742/0 (socket says 172.21.15.57:57742) 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 -- 172.21.15.57:0/3031014516 learned_addr learned my addr 172.21.15.57:0/3031014516 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd662ffd700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd664074c70 0x7fd66419b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6637fe700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66410f440 0x7fd66419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 -- 172.21.15.57:0/3031014516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66410f440 msgr2=0x7fd66419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66410f440 0x7fd66419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 -- 172.21.15.57:0/3031014516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd664074c70 msgr2=0x7fd66419b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd664074c70 0x7fd66419b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 -- 172.21.15.57:0/3031014516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6641a0370 con 0x7fd664075590 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd662ffd700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd664074c70 0x7fd66419b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6627fc700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 0x7fd66419b8f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d980 tx=0x7fd65800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658016070 con 0x7fd664075590 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.200+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6641a0660 con 0x7fd664075590 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.201+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6580044d0 con 0x7fd664075590 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.201+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658004d00 con 0x7fd664075590 2024-04-21T08:44:15.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.201+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6641a7ee0 con 0x7fd664075590 2024-04-21T08:44:15.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.202+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd664069730 con 0x7fd664075590 2024-04-21T08:44:15.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.203+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd658004020 con 0x7fd664075590 2024-04-21T08:44:15.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.203+0000 7fd64bfff700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd64c0a3d50 0x7fd64c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:15.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.204+0000 7fd662ffd700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd64c0a3d50 0x7fd64c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:15.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.204+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fd6580d0700 con 0x7fd664075590 2024-04-21T08:44:15.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.204+0000 7fd662ffd700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd64c0a3d50 0x7fd64c0a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd654006010 tx=0x7fd65401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:15.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.208+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd65801f020 con 0x7fd664075590 2024-04-21T08:44:15.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.427+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd66402df80 con 0x7fd664075590 2024-04-21T08:44:15.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.427+0000 7fd64bfff700 1 -- 172.21.15.57:0/3031014516 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd65809cc40 con 0x7fd664075590 2024-04-21T08:44:15.427 INFO:teuthology.orchestra.run.smithi057.stdout:141733920833 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.429+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd64c0a3d50 msgr2=0x7fd64c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.429+0000 7fd6696f8700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd64c0a3d50 0x7fd64c0a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd654006010 tx=0x7fd65401a040 comp rx=0 tx=0).stop 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.429+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 msgr2=0x7fd66419b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.429+0000 7fd6696f8700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 0x7fd66419b8f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd65800d980 tx=0x7fd65800dd40 comp rx=0 tx=0).stop 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.429+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 shutdown_connections 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd64c0a3d50 0x7fd64c0a6210 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd664074c70 0x7fd66419b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66410f440 0x7fd66419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 --2- 172.21.15.57:0/3031014516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd664075590 0x7fd66419b8f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:15.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 >> 172.21.15.57:0/3031014516 conn(0x7fd664079bf0 msgr2=0x7fd6640ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:15.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 shutdown_connections 2024-04-21T08:44:15.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:15.430+0000 7fd6696f8700 1 -- 172.21.15.57:0/3031014516 wait complete. 2024-04-21T08:44:15.669 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:16.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:16 smithi057 ceph-mon[27968]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:16.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:16 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3031014516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:44:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:16 smithi129 ceph-mon[31272]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:16.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:16 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3031014516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:44:16.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:16 smithi136 ceph-mon[31791]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:16 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3031014516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-21T08:44:18.278 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:18 smithi057 ceph-mon[27968]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:18.278 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:18 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/860191996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:44:18.311 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920833 for osd.4 2024-04-21T08:44:18.311 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:18.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:18 smithi129 ceph-mon[31272]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:18.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:18 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/860191996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:44:18.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:18 smithi136 ceph-mon[31791]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:18.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:18 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/860191996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 -- 172.21.15.57:0/2901381920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910108a70 msgr2=0x7f4910108e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 --2- 172.21.15.57:0/2901381920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910108a70 0x7f4910108e50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4900009990 tx=0x7f4900009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 -- 172.21.15.57:0/2901381920 shutdown_connections 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 --2- 172.21.15.57:0/2901381920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910103480 0x7f491010f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 --2- 172.21.15.57:0/2901381920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910108a70 0x7f4910108e50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 --2- 172.21.15.57:0/2901381920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4910102ac0 0x7f4910102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 -- 172.21.15.57:0/2901381920 >> 172.21.15.57:0/2901381920 conn(0x7f4910079bb0 msgr2=0x7f4910079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:19.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 -- 172.21.15.57:0/2901381920 shutdown_connections 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.146+0000 7f49148ce700 1 -- 172.21.15.57:0/2901381920 wait complete. 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 Processor -- start 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 -- start start 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4910102ac0 0x7f49101992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 0x7f49101997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910108a70 0x7f491019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4910112d00 con 0x7f4910103480 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4910112b80 con 0x7f4910102ac0 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f49148ce700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4910112e80 con 0x7f4910108a70 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f490dd9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 0x7f49101997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f490dd9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 0x7f49101997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57758/0 (socket says 172.21.15.57:57758) 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f490dd9b700 1 -- 172.21.15.57:0/860191996 learned_addr learned my addr 172.21.15.57:0/860191996 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:19.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f490e59c700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4910102ac0 0x7f49101992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.147+0000 7f490ed9d700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910108a70 0x7f491019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490dd9b700 1 -- 172.21.15.57:0/860191996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910108a70 msgr2=0x7f491019db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490dd9b700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910108a70 0x7f491019db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490dd9b700 1 -- 172.21.15.57:0/860191996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4910102ac0 msgr2=0x7f49101992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490dd9b700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4910102ac0 0x7f49101992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490dd9b700 1 -- 172.21.15.57:0/860191996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491019e2c0 con 0x7f4910103480 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490ed9d700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910108a70 0x7f491019db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:19.490 INFO:teuthology.orchestra.run.smithi057.stdout:163208757310 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f490dd9b700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 0x7f49101997f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f490400b530 tx=0x7f490400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904015020 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f491019e5b0 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4904011ce0 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4904010e40 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.148+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49101a5e30 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.149+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4910069730 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.150+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4904018020 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.151+0000 7f48ff7fe700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f48f80a3cd0 0x7f48f80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.151+0000 7f490e59c700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f48f80a3cd0 0x7f48f80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.151+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f49040cffe0 con 0x7f4910103480 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.151+0000 7f490e59c700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f48f80a3cd0 0x7f48f80a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f49000098e0 tx=0x7f4900005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:19.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.156+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f490401f020 con 0x7f4910103480 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.369+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f491002df80 con 0x7f4910103480 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.369+0000 7f48ff7fe700 1 -- 172.21.15.57:0/860191996 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f490409c630 con 0x7f4910103480 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f48f80a3cd0 msgr2=0x7f48f80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f48f80a3cd0 0x7f48f80a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f49000098e0 tx=0x7f4900005960 comp rx=0 tx=0).stop 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 msgr2=0x7f49101997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 0x7f49101997f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f490400b530 tx=0x7f490400b840 comp rx=0 tx=0).stop 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 shutdown_connections 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f48f80a3cd0 0x7f48f80a6190 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4910102ac0 0x7f49101992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4910108a70 0x7f491019db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 --2- 172.21.15.57:0/860191996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4910103480 0x7f49101997f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 >> 172.21.15.57:0/860191996 conn(0x7f4910079bb0 msgr2=0x7f4910101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 shutdown_connections 2024-04-21T08:44:19.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:17.372+0000 7f49148ce700 1 -- 172.21.15.57:0/860191996 wait complete. 2024-04-21T08:44:20.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:20 smithi057 ceph-mon[27968]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:20.074 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:20 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1277951871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:44:20.101 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757310 for osd.5 2024-04-21T08:44:20.102 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:20.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:20 smithi129 ceph-mon[31272]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:20.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:20 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1277951871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:44:20.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:20 smithi136 ceph-mon[31791]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:20.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:20 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1277951871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 -- 172.21.15.57:0/2093743852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 msgr2=0x7f97a41052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 --2- 172.21.15.57:0/2093743852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a41052c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f97980099c0 tx=0x7f9798009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 -- 172.21.15.57:0/2093743852 shutdown_connections 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 --2- 172.21.15.57:0/2093743852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4109fc0 0x7f97a4111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 --2- 172.21.15.57:0/2093743852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a41052c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 --2- 172.21.15.57:0/2093743852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4105890 0x7f97a41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 -- 172.21.15.57:0/2093743852 >> 172.21.15.57:0/2093743852 conn(0x7f97a4100770 msgr2=0x7f97a4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 -- 172.21.15.57:0/2093743852 shutdown_connections 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.018+0000 7f97aaf00700 1 -- 172.21.15.57:0/2093743852 wait complete. 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.019+0000 7f97aaf00700 1 Processor -- start 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.019+0000 7f97aaf00700 1 -- start start 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4105890 0x7f97a419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4109fc0 0x7f97a419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4114e50 con 0x7f97a4104ee0 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4114cd0 con 0x7f97a4105890 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f97a4114fd0 con 0x7f97a4109fc0 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57770/0 (socket says 172.21.15.57:57770) 2024-04-21T08:44:21.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 -- 172.21.15.57:0/1277951871 learned_addr learned my addr 172.21.15.57:0/1277951871 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a949d700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4109fc0 0x7f97a419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a3fff700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4105890 0x7f97a419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 -- 172.21.15.57:0/1277951871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4109fc0 msgr2=0x7f97a419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4109fc0 0x7f97a419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 -- 172.21.15.57:0/1277951871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4105890 msgr2=0x7f97a419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4105890 0x7f97a419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 -- 172.21.15.57:0/1277951871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a41a0320 con 0x7f97a4104ee0 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a949d700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4109fc0 0x7f97a419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:21.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a3fff700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4105890 0x7f97a419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stdout:77309411405 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a8c9c700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a419b360 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f979800b580 tx=0x7f97980047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798027070 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a41a05b0 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9798004020 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.020+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798022e30 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.021+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41a7e90 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.022+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a4069730 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.023+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9798022590 con 0x7f97a4104ee0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.023+0000 7f97a1ffb700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f97940a3cd0 0x7f97940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.024+0000 7f97a3fff700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f97940a3cd0 0x7f97940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:21.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.024+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f97980d1570 con 0x7f97a4104ee0 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.024+0000 7f97a3fff700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f97940a3cd0 0x7f97940a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9790005950 tx=0x7f97900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.028+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f979805dff0 con 0x7f97a4104ee0 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.241+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f97a402df80 con 0x7f97a4104ee0 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.241+0000 7f97a1ffb700 1 -- 172.21.15.57:0/1277951871 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9798013070 con 0x7f97a4104ee0 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f97940a3cd0 msgr2=0x7f97940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f97940a3cd0 0x7f97940a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9790005950 tx=0x7f97900058e0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 msgr2=0x7f97a419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a419b360 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f979800b580 tx=0x7f97980047b0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 shutdown_connections 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f97940a3cd0 0x7f97940a6190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97a4109fc0 0x7f97a419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97a4104ee0 0x7f97a419b360 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 --2- 172.21.15.57:0/1277951871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97a4105890 0x7f97a419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:21.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.243+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 >> 172.21.15.57:0/1277951871 conn(0x7f97a4100770 msgr2=0x7f97a4109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:21.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.244+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 shutdown_connections 2024-04-21T08:44:21.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:19.244+0000 7f97aaf00700 1 -- 172.21.15.57:0/1277951871 wait complete. 2024-04-21T08:44:22.138 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411405 for osd.2 2024-04-21T08:44:22.139 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:22.244 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:22 smithi057 ceph-mon[27968]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:22.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:22 smithi129 ceph-mon[31272]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:22.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 -- 172.21.15.57:0/2385111908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa608101d20 msgr2=0x7fa6081021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2385111908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa608101d20 0x7fa6081021a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009990 tx=0x7fa5fc009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 -- 172.21.15.57:0/2385111908 shutdown_connections 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2385111908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608106990 0x7fa608106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2385111908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6081026e0 0x7fa60810e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2385111908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa608101d20 0x7fa6081021a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 -- 172.21.15.57:0/2385111908 >> 172.21.15.57:0/2385111908 conn(0x7fa6080788f0 msgr2=0x7fa608078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 -- 172.21.15.57:0/2385111908 shutdown_connections 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.351+0000 7fa60fd05700 1 -- 172.21.15.57:0/2385111908 wait complete. 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 Processor -- start 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 -- start start 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608101d20 0x7fa608199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 0x7fa608199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa608106990 0x7fa60819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa608111af0 con 0x7fa6081026e0 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa608111970 con 0x7fa608101d20 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60fd05700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa608111c70 con 0x7fa608106990 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 0x7fa608199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 0x7fa608199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57798/0 (socket says 172.21.15.57:57798) 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 -- 172.21.15.57:0/2586300046 learned_addr learned my addr 172.21.15.57:0/2586300046 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60daa1700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608101d20 0x7fa608199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 -- 172.21.15.57:0/2586300046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa608106990 msgr2=0x7fa60819d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa608106990 0x7fa60819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 -- 172.21.15.57:0/2586300046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608101d20 msgr2=0x7fa608199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608101d20 0x7fa608199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60d2a0700 1 -- 172.21.15.57:0/2586300046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60819e050 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.352+0000 7fa60daa1700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608101d20 0x7fa608199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.353+0000 7fa60d2a0700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 0x7fa608199580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc000c00 tx=0x7fa5fc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.353+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc027070 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.353+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60819e2e0 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.353+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5fc003fa0 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.353+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc022db0 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.353+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6081a5bc0 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.354+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa60806a9c0 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.356+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa5fc02a070 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.356+0000 7fa606ffd700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5f00a3cd0 0x7fa5f00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.356+0000 7fa60daa1700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5f00a3cd0 0x7fa5f00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.356+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fa5fc0d1990 con 0x7fa6081026e0 2024-04-21T08:44:22.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.357+0000 7fa60daa1700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5f00a3cd0 0x7fa5f00a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa600005950 tx=0x7fa6000058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:22.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.360+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5fc013070 con 0x7fa6081026e0 2024-04-21T08:44:22.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:22 smithi136 ceph-mon[31791]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:22.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.572+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa60819a3a0 con 0x7fa6081026e0 2024-04-21T08:44:22.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.573+0000 7fa606ffd700 1 -- 172.21.15.57:0/2586300046 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5fc09df60 con 0x7fa6081026e0 2024-04-21T08:44:22.573 INFO:teuthology.orchestra.run.smithi057.stdout:34359738455 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5f00a3cd0 msgr2=0x7fa5f00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5f00a3cd0 0x7fa5f00a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa600005950 tx=0x7fa6000058e0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 msgr2=0x7fa608199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 0x7fa608199580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc000c00 tx=0x7fa5fc0047b0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 shutdown_connections 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5f00a3cd0 0x7fa5f00a6190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa608101d20 0x7fa608199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa608106990 0x7fa60819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 --2- 172.21.15.57:0/2586300046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6081026e0 0x7fa608199580 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 >> 172.21.15.57:0/2586300046 conn(0x7fa6080788f0 msgr2=0x7fa60806b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 shutdown_connections 2024-04-21T08:44:22.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:22.576+0000 7fa60fd05700 1 -- 172.21.15.57:0/2586300046 wait complete. 2024-04-21T08:44:23.322 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2586300046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:44:23.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2586300046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:44:23.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2586300046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-21T08:44:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 -- 172.21.15.57:0/2920633790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4102950 msgr2=0x7f21b4102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 --2- 172.21.15.57:0/2920633790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4102950 0x7f21b4102dd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009990 tx=0x7f21a8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 -- 172.21.15.57:0/2920633790 shutdown_connections 2024-04-21T08:44:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 --2- 172.21.15.57:0/2920633790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21b4103310 0x7f21b410f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 --2- 172.21.15.57:0/2920633790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f21b4108950 0x7f21b4108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 --2- 172.21.15.57:0/2920633790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4102950 0x7f21b4102dd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.505+0000 7f21b8a02700 1 -- 172.21.15.57:0/2920633790 >> 172.21.15.57:0/2920633790 conn(0x7f21b40fe460 msgr2=0x7f21b4100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.506+0000 7f21b8a02700 1 -- 172.21.15.57:0/2920633790 shutdown_connections 2024-04-21T08:44:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.506+0000 7f21b8a02700 1 -- 172.21.15.57:0/2920633790 wait complete. 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.506+0000 7f21b8a02700 1 Processor -- start 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.506+0000 7f21b8a02700 1 -- start start 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b8a02700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21b4102950 0x7f21b4199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b8a02700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f21b4103310 0x7f21b4199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b8a02700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 0x7f21b419d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b8a02700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f21b4112b90 con 0x7f21b4108950 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b8a02700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21b4112a10 con 0x7f21b4102950 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b8a02700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f21b4112d10 con 0x7f21b4103310 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 0x7f21b419d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 0x7f21b419d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46820/0 (socket says 172.21.15.57:46820) 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 -- 172.21.15.57:0/3425220256 learned_addr learned my addr 172.21.15.57:0/3425220256 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:23.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b259c700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f21b4103310 0x7f21b4199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b2d9d700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21b4102950 0x7f21b4199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 -- 172.21.15.57:0/3425220256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f21b4103310 msgr2=0x7f21b4199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f21b4103310 0x7f21b4199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 -- 172.21.15.57:0/3425220256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21b4102950 msgr2=0x7f21b4199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21b4102950 0x7f21b4199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 -- 172.21.15.57:0/3425220256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b419e050 con 0x7f21b4108950 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.507+0000 7f21b359e700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 0x7f21b419d910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00ada0 tx=0x7f21ac00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.508+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac017020 con 0x7f21b4108950 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.508+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b419e340 con 0x7f21b4108950 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.508+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21ac00ce30 con 0x7f21b4108950 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.508+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac0054a0 con 0x7f21b4108950 2024-04-21T08:44:23.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.508+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b41a5bc0 con 0x7f21b4108950 2024-04-21T08:44:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.509+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b410aee0 con 0x7f21b4108950 2024-04-21T08:44:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.510+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f21ac00c980 con 0x7f21b4108950 2024-04-21T08:44:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.510+0000 7f219bfff700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f219c0a3cd0 0x7f219c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.511+0000 7f21b2d9d700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f219c0a3cd0 0x7f219c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.511+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f21ac0d0150 con 0x7f21b4108950 2024-04-21T08:44:23.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.511+0000 7f21b2d9d700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f219c0a3cd0 0x7f219c0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f21a400bbb0 tx=0x7f21a400b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.515+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21ac021020 con 0x7f21b4108950 2024-04-21T08:44:23.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.725+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f21b419a3a0 con 0x7f21b4108950 2024-04-21T08:44:23.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.726+0000 7f219bfff700 1 -- 172.21.15.57:0/3425220256 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f21ac09c6c0 con 0x7f21b4108950 2024-04-21T08:44:23.725 INFO:teuthology.orchestra.run.smithi057.stdout:343597383710 2024-04-21T08:44:23.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f219c0a3cd0 msgr2=0x7f219c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f219c0a3cd0 0x7f219c0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f21a400bbb0 tx=0x7f21a400b590 comp rx=0 tx=0).stop 2024-04-21T08:44:23.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 msgr2=0x7f21b419d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:23.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 0x7f21b419d910 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00ada0 tx=0x7f21ac00c330 comp rx=0 tx=0).stop 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 shutdown_connections 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f219c0a3cd0 0x7f219c0a6190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21b4102950 0x7f21b4199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21b4108950 0x7f21b419d910 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 --2- 172.21.15.57:0/3425220256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f21b4103310 0x7f21b4199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 >> 172.21.15.57:0/3425220256 conn(0x7f21b40fe460 msgr2=0x7f21b40fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 shutdown_connections 2024-04-21T08:44:23.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:23.728+0000 7f21b8a02700 1 -- 172.21.15.57:0/3425220256 wait complete. 2024-04-21T08:44:24.313 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:24 smithi057 ceph-mon[27968]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:24.314 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:24 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3425220256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:44:24.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:24 smithi129 ceph-mon[31272]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:24.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:24 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3425220256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:44:24.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:24 smithi136 ceph-mon[31791]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:24.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:24 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3425220256' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-21T08:44:24.834 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738455 for osd.0 2024-04-21T08:44:24.835 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:25.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 -- 172.21.15.57:0/928000380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006b0f0 msgr2=0x7f19d010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 --2- 172.21.15.57:0/928000380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006b0f0 0x7f19d010f900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f19c800b0a0 tx=0x7f19c800b3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 -- 172.21.15.57:0/928000380 shutdown_connections 2024-04-21T08:44:25.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 --2- 172.21.15.57:0/928000380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19d0108a30 0x7f19d0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 --2- 172.21.15.57:0/928000380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006b0f0 0x7f19d010f900 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 --2- 172.21.15.57:0/928000380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19d006a750 0x7f19d006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 -- 172.21.15.57:0/928000380 >> 172.21.15.57:0/928000380 conn(0x7f19d00788f0 msgr2=0x7f19d0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 -- 172.21.15.57:0/928000380 shutdown_connections 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.504+0000 7f19d68e8700 1 -- 172.21.15.57:0/928000380 wait complete. 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.505+0000 7f19d68e8700 1 Processor -- start 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.505+0000 7f19d68e8700 1 -- start start 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d68e8700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 0x7f19d0101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d68e8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19d006b0f0 0x7f19d0101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d68e8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19d0108a30 0x7f19d0102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d68e8700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f19d0112c20 con 0x7f19d006a750 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d68e8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19d0112aa0 con 0x7f19d0108a30 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d68e8700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f19d0112da0 con 0x7f19d006b0f0 2024-04-21T08:44:25.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 0x7f19d0101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 0x7f19d0101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46842/0 (socket says 172.21.15.57:46842) 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 -- 172.21.15.57:0/1226127373 learned_addr learned my addr 172.21.15.57:0/1226127373 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cf7fe700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19d006b0f0 0x7f19d0101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19d4e85700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19d0108a30 0x7f19d0102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 -- 172.21.15.57:0/1226127373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19d006b0f0 msgr2=0x7f19d0101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19d006b0f0 0x7f19d0101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 -- 172.21.15.57:0/1226127373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19d0108a30 msgr2=0x7f19d0102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19d0108a30 0x7f19d0102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 -- 172.21.15.57:0/1226127373 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d01a7560 con 0x7f19d006a750 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.506+0000 7f19cffff700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 0x7f19d0101a10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c9b0 tx=0x7f19c000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.507+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0007b00 con 0x7f19d006a750 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.507+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d01a7850 con 0x7f19d006a750 2024-04-21T08:44:25.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.507+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19c0007ca0 con 0x7f19d006a750 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.507+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0018660 con 0x7f19d006a750 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.507+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d01a7de0 con 0x7f19d006a750 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.508+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d0069730 con 0x7f19d006a750 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.509+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f19c000f450 con 0x7f19d006a750 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.509+0000 7f19cd7fa700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19bc0a3cd0 0x7f19bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.510+0000 7f19cf7fe700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19bc0a3cd0 0x7f19bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.510+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f19c00d0830 con 0x7f19d006a750 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.510+0000 7f19cf7fe700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19bc0a3cd0 0x7f19bc0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f19d0078e80 tx=0x7f19b800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:25.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.514+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19c0014030 con 0x7f19d006a750 2024-04-21T08:44:25.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.727+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f19d0102f70 con 0x7f19d006a750 2024-04-21T08:44:25.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.727+0000 7f19cd7fa700 1 -- 172.21.15.57:0/1226127373 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f19d0102f70 con 0x7f19d006a750 2024-04-21T08:44:25.727 INFO:teuthology.orchestra.run.smithi057.stdout:309237645348 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.729+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19bc0a3cd0 msgr2=0x7f19bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.729+0000 7f19d68e8700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19bc0a3cd0 0x7f19bc0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f19d0078e80 tx=0x7f19b800a380 comp rx=0 tx=0).stop 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.729+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 msgr2=0x7f19d0101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.729+0000 7f19d68e8700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 0x7f19d0101a10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c9b0 tx=0x7f19c000ccc0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 shutdown_connections 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f19bc0a3cd0 0x7f19bc0a6190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19d0108a30 0x7f19d0102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f19d006a750 0x7f19d0101a10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 --2- 172.21.15.57:0/1226127373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19d006b0f0 0x7f19d0101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:25.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 >> 172.21.15.57:0/1226127373 conn(0x7f19d00788f0 msgr2=0x7f19d00ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:25.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 shutdown_connections 2024-04-21T08:44:25.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:25.730+0000 7f19d68e8700 1 -- 172.21.15.57:0/1226127373 wait complete. 2024-04-21T08:44:26.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:26 smithi129 ceph-mon[31272]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:26.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:26 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1226127373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:44:26.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:26 smithi057 ceph-mon[27968]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:26.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:26 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1226127373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:44:26.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:26 smithi136 ceph-mon[31791]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:26.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:26 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1226127373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-21T08:44:26.631 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645348 for osd.10 2024-04-21T08:44:26.631 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 -- 172.21.15.57:0/54902853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4102a90 msgr2=0x7fd9d4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/54902853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4102a90 0x7fd9d4102f10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009990 tx=0x7fd9c4009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 -- 172.21.15.57:0/54902853 shutdown_connections 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/54902853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9d4103450 0x7fd9d410f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/54902853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d4108a50 0x7fd9d4108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/54902853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4102a90 0x7fd9d4102f10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 -- 172.21.15.57:0/54902853 >> 172.21.15.57:0/54902853 conn(0x7fd9d4079ba0 msgr2=0x7fd9d4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 -- 172.21.15.57:0/54902853 shutdown_connections 2024-04-21T08:44:26.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.844+0000 7fd9d8e17700 1 -- 172.21.15.57:0/54902853 wait complete. 2024-04-21T08:44:26.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 Processor -- start 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 -- start start 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9d4102a90 0x7fd9d4199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d4103450 0x7fd9d41997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 0x7fd9d419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d4112ce0 con 0x7fd9d4108a50 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d4112b60 con 0x7fd9d4102a90 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d8e17700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d4112e60 con 0x7fd9d4103450 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.845+0000 7fd9d2d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 0x7fd9d419db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 0x7fd9d419db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46846/0 (socket says 172.21.15.57:46846) 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 -- 172.21.15.57:0/327415963 learned_addr learned my addr 172.21.15.57:0/327415963 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d259c700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9d4102a90 0x7fd9d4199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d1d9b700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d4103450 0x7fd9d41997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:26.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 -- 172.21.15.57:0/327415963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d4103450 msgr2=0x7fd9d41997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d4103450 0x7fd9d41997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 -- 172.21.15.57:0/327415963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9d4102a90 msgr2=0x7fd9d4199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9d4102a90 0x7fd9d4199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 -- 172.21.15.57:0/327415963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d419e2a0 con 0x7fd9d4108a50 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d2d9d700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 0x7fd9d419db60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00b970 tx=0x7fd9cc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc00d860 con 0x7fd9d4108a50 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9cc00e430 con 0x7fd9d4108a50 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d419e590 con 0x7fd9d4108a50 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc017660 con 0x7fd9d4108a50 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.846+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d41a5e10 con 0x7fd9d4108a50 2024-04-21T08:44:26.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.847+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d4069730 con 0x7fd9d4108a50 2024-04-21T08:44:26.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.848+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd9cc00da00 con 0x7fd9d4108a50 2024-04-21T08:44:26.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.849+0000 7fd9c37fe700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd9bc0a3cd0 0x7fd9bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:26.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.849+0000 7fd9d259c700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd9bc0a3cd0 0x7fd9bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:26.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.849+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fd9cc0d04b0 con 0x7fd9d4108a50 2024-04-21T08:44:26.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.850+0000 7fd9d259c700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd9bc0a3cd0 0x7fd9bc0a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8009a10 tx=0x7fd9c8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:26.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:26.854+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9cc013030 con 0x7fd9d4108a50 2024-04-21T08:44:27.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.066+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd9d419a5f0 con 0x7fd9d4108a50 2024-04-21T08:44:27.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.067+0000 7fd9c37fe700 1 -- 172.21.15.57:0/327415963 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd9cc09c9f0 con 0x7fd9d4108a50 2024-04-21T08:44:27.066 INFO:teuthology.orchestra.run.smithi057.stdout:197568495675 2024-04-21T08:44:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd9bc0a3cd0 msgr2=0x7fd9bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd9bc0a3cd0 0x7fd9bc0a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9c8009a10 tx=0x7fd9c8009190 comp rx=0 tx=0).stop 2024-04-21T08:44:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 msgr2=0x7fd9d419db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 0x7fd9d419db60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc00b970 tx=0x7fd9cc00bc80 comp rx=0 tx=0).stop 2024-04-21T08:44:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 shutdown_connections 2024-04-21T08:44:27.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd9bc0a3cd0 0x7fd9bc0a6190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9d4102a90 0x7fd9d4199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9d4108a50 0x7fd9d419db60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 --2- 172.21.15.57:0/327415963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9d4103450 0x7fd9d41997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 >> 172.21.15.57:0/327415963 conn(0x7fd9d4079ba0 msgr2=0x7fd9d4101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 shutdown_connections 2024-04-21T08:44:27.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.069+0000 7fd9d8e17700 1 -- 172.21.15.57:0/327415963 wait complete. 2024-04-21T08:44:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 -- 172.21.15.57:0/1019454317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450107960 msgr2=0x7f7450107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 --2- 172.21.15.57:0/1019454317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450107960 0x7f7450107d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7440009990 tx=0x7f7440009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 -- 172.21.15.57:0/1019454317 shutdown_connections 2024-04-21T08:44:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 --2- 172.21.15.57:0/1019454317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450102320 0x7f745010e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 --2- 172.21.15.57:0/1019454317 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450107960 0x7f7450107d40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 --2- 172.21.15.57:0/1019454317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7450101960 0x7f7450101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.671+0000 7f7455d26700 1 -- 172.21.15.57:0/1019454317 >> 172.21.15.57:0/1019454317 conn(0x7f74500789e0 msgr2=0x7f7450078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.672+0000 7f7455d26700 1 -- 172.21.15.57:0/1019454317 shutdown_connections 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.672+0000 7f7455d26700 1 -- 172.21.15.57:0/1019454317 wait complete. 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.672+0000 7f7455d26700 1 Processor -- start 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 -- start start 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450101960 0x7f7450199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 0x7f745019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7450107960 0x7f74501a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7450111b70 con 0x7f7450102320 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f74501119f0 con 0x7f7450107960 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f7455d26700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7450111cf0 con 0x7f7450101960 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 0x7f745019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 0x7f745019a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46864/0 (socket says 172.21.15.57:46864) 2024-04-21T08:44:27.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 -- 172.21.15.57:0/2322349014 learned_addr learned my addr 172.21.15.57:0/2322349014 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744f7fe700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450101960 0x7f7450199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744ffff700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7450107960 0x7f74501a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 -- 172.21.15.57:0/2322349014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450101960 msgr2=0x7f7450199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450101960 0x7f7450199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 -- 172.21.15.57:0/2322349014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7450107960 msgr2=0x7f74501a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7450107960 0x7f74501a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 -- 172.21.15.57:0/2322349014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74501a54f0 con 0x7f7450102320 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744f7fe700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450101960 0x7f7450199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.673+0000 7f744effd700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 0x7f745019a470 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f744400d980 tx=0x7f744400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.674+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444016070 con 0x7f7450102320 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.674+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74501a5720 con 0x7f7450102320 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.674+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74440044d0 con 0x7f7450102320 2024-04-21T08:44:27.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.674+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444004d00 con 0x7f7450102320 2024-04-21T08:44:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.674+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74501a5c80 con 0x7f7450102320 2024-04-21T08:44:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.675+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7450109f60 con 0x7f7450102320 2024-04-21T08:44:27.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.676+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7444004020 con 0x7f7450102320 2024-04-21T08:44:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.677+0000 7f744cff9700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f74380a3cd0 0x7f74380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.677+0000 7f744f7fe700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f74380a3cd0 0x7f74380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.678+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f74440d0cc0 con 0x7f7450102320 2024-04-21T08:44:27.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.678+0000 7f744f7fe700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f74380a3cd0 0x7f74380a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7440006010 tx=0x7f744000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:27.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.682+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f744401f020 con 0x7f7450102320 2024-04-21T08:44:27.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.897+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f745006b760 con 0x7f7450102320 2024-04-21T08:44:27.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.897+0000 7f744cff9700 1 -- 172.21.15.57:0/2322349014 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f744409d200 con 0x7f7450102320 2024-04-21T08:44:27.897 INFO:teuthology.orchestra.run.smithi057.stdout:219043332148 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f74380a3cd0 msgr2=0x7f74380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f74380a3cd0 0x7f74380a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7440006010 tx=0x7f744000b540 comp rx=0 tx=0).stop 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 msgr2=0x7f745019a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 0x7f745019a470 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f744400d980 tx=0x7f744400dd40 comp rx=0 tx=0).stop 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 shutdown_connections 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f74380a3cd0 0x7f74380a6190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7450107960 0x7f74501a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7450101960 0x7f7450199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 --2- 172.21.15.57:0/2322349014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7450102320 0x7f745019a470 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.899+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 >> 172.21.15.57:0/2322349014 conn(0x7f74500789e0 msgr2=0x7f7450104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.900+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 shutdown_connections 2024-04-21T08:44:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:27.900+0000 7f7455d26700 1 -- 172.21.15.57:0/2322349014 wait complete. 2024-04-21T08:44:28.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:28 smithi129 ceph-mon[31272]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:28.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/327415963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:44:28.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2322349014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:44:28.330 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:28 smithi057 ceph-mon[27968]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:28.331 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/327415963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:44:28.331 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2322349014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:44:28.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383710 for osd.11 2024-04-21T08:44:28.363 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:28 smithi136 ceph-mon[31791]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/327415963' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-21T08:44:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2322349014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-21T08:44:28.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/3543348856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 msgr2=0x7f8ec8108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/3543348856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec8108e00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009990 tx=0x7f8eb8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/3543348856 shutdown_connections 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/3543348856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec806b980 0x7f8ec810f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/3543348856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec8108e00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/3543348856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 0x7f8ec806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/3543348856 >> 172.21.15.57:0/3543348856 conn(0x7f8ec80788b0 msgr2=0x7f8ec8078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/3543348856 shutdown_connections 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.978+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/3543348856 wait complete. 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 Processor -- start 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 -- start start 2024-04-21T08:44:28.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 0x7f8ec8199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec806b980 0x7f8ec8199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec8112bc0 con 0x7f8ec8108a20 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec8112a40 con 0x7f8ec806afe0 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ecd8d5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec8112d40 con 0x7f8ec806b980 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ec77fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ec77fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec819d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46888/0 (socket says 172.21.15.57:46888) 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.979+0000 7f8ec77fe700 1 -- 172.21.15.57:0/1007856065 learned_addr learned my addr 172.21.15.57:0/1007856065 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec6ffd700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 0x7f8ec8199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec67fc700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec806b980 0x7f8ec8199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec77fe700 1 -- 172.21.15.57:0/1007856065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec806b980 msgr2=0x7f8ec8199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec77fe700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec806b980 0x7f8ec8199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec77fe700 1 -- 172.21.15.57:0/1007856065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 msgr2=0x7f8ec8199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec77fe700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 0x7f8ec8199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec77fe700 1 -- 172.21.15.57:0/1007856065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec819e120 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec6ffd700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 0x7f8ec8199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ec77fe700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec819d9e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0007ee0 tx=0x7f8ec000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000dbd0 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ec000dd70 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec819e410 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0005990 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.980+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec81a5c90 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.981+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec8069730 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.982+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8ec001a020 con 0x7f8ec8108a20 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.983+0000 7f8ecc8d3700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eb00a3d50 0x7f8eb00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:28.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.983+0000 7f8ec6ffd700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eb00a3d50 0x7f8eb00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:28.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.983+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f8ec00d06d0 con 0x7f8ec8108a20 2024-04-21T08:44:28.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.983+0000 7f8ec6ffd700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eb00a3d50 0x7f8eb00a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8006010 tx=0x7f8eb800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:28.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:28.987+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ec0010070 con 0x7f8ec8108a20 2024-04-21T08:44:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.197+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8ec802df80 con 0x7f8ec8108a20 2024-04-21T08:44:29.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.197+0000 7f8ecc8d3700 1 -- 172.21.15.57:0/1007856065 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8ec009cbb0 con 0x7f8ec8108a20 2024-04-21T08:44:29.198 INFO:teuthology.orchestra.run.smithi057.stdout:107374182474 2024-04-21T08:44:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.199+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eb00a3d50 msgr2=0x7f8eb00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.199+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eb00a3d50 0x7f8eb00a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8006010 tx=0x7f8eb800b540 comp rx=0 tx=0).stop 2024-04-21T08:44:29.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.199+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 msgr2=0x7f8ec819d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.199+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec819d9e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0007ee0 tx=0x7f8ec000d3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 shutdown_connections 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8eb00a3d50 0x7f8eb00a6210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ec806afe0 0x7f8ec8199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ec8108a20 0x7f8ec819d9e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 --2- 172.21.15.57:0/1007856065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ec806b980 0x7f8ec8199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 >> 172.21.15.57:0/1007856065 conn(0x7f8ec80788b0 msgr2=0x7f8ec80ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 shutdown_connections 2024-04-21T08:44:29.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:29.200+0000 7f8ecd8d5700 1 -- 172.21.15.57:0/1007856065 wait complete. 2024-04-21T08:44:29.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332141 got 219043332148 for osd.7 2024-04-21T08:44:29.666 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:30.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:30 smithi129 ceph-mon[31272]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:30.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:30 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1007856065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:44:30.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:30 smithi057 ceph-mon[27968]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:30.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:30 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1007856065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:44:30.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:30 smithi136 ceph-mon[31791]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:30.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:30 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1007856065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-21T08:44:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 -- 172.21.15.57:0/1989123473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c108a30 msgr2=0x7eff3c108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 --2- 172.21.15.57:0/1989123473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c108a30 0x7eff3c108e10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7eff38009990 tx=0x7eff38009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 -- 172.21.15.57:0/1989123473 shutdown_connections 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 --2- 172.21.15.57:0/1989123473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff3c06b0f0 0x7eff3c10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 --2- 172.21.15.57:0/1989123473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c108a30 0x7eff3c108e10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 --2- 172.21.15.57:0/1989123473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06a750 0x7eff3c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 -- 172.21.15.57:0/1989123473 >> 172.21.15.57:0/1989123473 conn(0x7eff3c0788f0 msgr2=0x7eff3c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 -- 172.21.15.57:0/1989123473 shutdown_connections 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 -- 172.21.15.57:0/1989123473 wait complete. 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.630+0000 7eff4367c700 1 Processor -- start 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 -- start start 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 0x7eff3c101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06b0f0 0x7eff3c101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff3c108a30 0x7eff3c102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c112c30 con 0x7eff3c06a750 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c112ab0 con 0x7eff3c06b0f0 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff4367c700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c112db0 con 0x7eff3c108a30 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 0x7eff3c101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 0x7eff3c101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46898/0 (socket says 172.21.15.57:46898) 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 -- 172.21.15.57:0/1830813375 learned_addr learned my addr 172.21.15.57:0/1830813375 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41c19700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff3c108a30 0x7eff3c102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff40c17700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06b0f0 0x7eff3c101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 -- 172.21.15.57:0/1830813375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff3c108a30 msgr2=0x7eff3c102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff3c108a30 0x7eff3c102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 -- 172.21.15.57:0/1830813375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06b0f0 msgr2=0x7eff3c101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06b0f0 0x7eff3c101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 -- 172.21.15.57:0/1830813375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c1a7640 con 0x7eff3c06a750 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff40c17700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06b0f0 0x7eff3c101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.631+0000 7eff41418700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 0x7eff3c101a10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7eff380057f0 tx=0x7eff38003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.632+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38027070 con 0x7eff3c06a750 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.632+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff380040a0 con 0x7eff3c06a750 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.632+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3c1a78d0 con 0x7eff3c06a750 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.632+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38022dc0 con 0x7eff3c06a750 2024-04-21T08:44:30.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.632+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3c1a7e00 con 0x7eff3c06a750 2024-04-21T08:44:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.633+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff3c069730 con 0x7eff3c06a750 2024-04-21T08:44:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.634+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7eff3802a070 con 0x7eff3c06a750 2024-04-21T08:44:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.634+0000 7eff327fc700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7eff280a3d50 0x7eff280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.635+0000 7eff40c17700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7eff280a3d50 0x7eff280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.635+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7eff380d16a0 con 0x7eff3c06a750 2024-04-21T08:44:30.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.635+0000 7eff40c17700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7eff280a3d50 0x7eff280a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7eff2c005fd0 tx=0x7eff2c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:30.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.639+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff3805eff0 con 0x7eff3c06a750 2024-04-21T08:44:30.789 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182474 for osd.3 2024-04-21T08:44:30.789 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.856+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7eff3c02e140 con 0x7eff3c06a750 2024-04-21T08:44:30.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.857+0000 7eff327fc700 1 -- 172.21.15.57:0/1830813375 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7eff38013070 con 0x7eff3c06a750 2024-04-21T08:44:30.857 INFO:teuthology.orchestra.run.smithi057.stdout:253403070513 2024-04-21T08:44:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7eff280a3d50 msgr2=0x7eff280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7eff280a3d50 0x7eff280a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7eff2c005fd0 tx=0x7eff2c005e20 comp rx=0 tx=0).stop 2024-04-21T08:44:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 msgr2=0x7eff3c101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 0x7eff3c101a10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7eff380057f0 tx=0x7eff38003680 comp rx=0 tx=0).stop 2024-04-21T08:44:30.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 shutdown_connections 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7eff280a3d50 0x7eff280a6210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff3c108a30 0x7eff3c102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff3c06a750 0x7eff3c101a10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 --2- 172.21.15.57:0/1830813375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff3c06b0f0 0x7eff3c101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 >> 172.21.15.57:0/1830813375 conn(0x7eff3c0788f0 msgr2=0x7eff3c0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 shutdown_connections 2024-04-21T08:44:30.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:30.859+0000 7eff4367c700 1 -- 172.21.15.57:0/1830813375 wait complete. 2024-04-21T08:44:31.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:31 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1830813375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:44:31.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:31 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1830813375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:44:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:31 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1830813375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-21T08:44:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 -- 172.21.15.57:0/2978931562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e8108ad0 msgr2=0x7f77e8108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 --2- 172.21.15.57:0/2978931562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e8108ad0 0x7f77e8108eb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009990 tx=0x7f77d8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 -- 172.21.15.57:0/2978931562 shutdown_connections 2024-04-21T08:44:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 --2- 172.21.15.57:0/2978931562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e81019d0 0x7f77e810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 --2- 172.21.15.57:0/2978931562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e8108ad0 0x7f77e8108eb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 --2- 172.21.15.57:0/2978931562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8101030 0x7f77e8101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 -- 172.21.15.57:0/2978931562 >> 172.21.15.57:0/2978931562 conn(0x7f77e8077fb0 msgr2=0x7f77e80ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 -- 172.21.15.57:0/2978931562 shutdown_connections 2024-04-21T08:44:31.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.533+0000 7f77ec867700 1 -- 172.21.15.57:0/2978931562 wait complete. 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.534+0000 7f77ec867700 1 Processor -- start 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.534+0000 7f77ec867700 1 -- start start 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77ec867700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e8101030 0x7f77e8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77ec867700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 0x7f77e81996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77ec867700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8108ad0 0x7f77e819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77ec867700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8112c60 con 0x7f77e81019d0 2024-04-21T08:44:31.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77ec867700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8112ae0 con 0x7f77e8108ad0 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77ec867700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8112de0 con 0x7f77e8101030 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 0x7f77e81996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 0x7f77e81996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46912/0 (socket says 172.21.15.57:46912) 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 -- 172.21.15.57:0/409638858 learned_addr learned my addr 172.21.15.57:0/409638858 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e659c700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e8101030 0x7f77e8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e6d9d700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8108ad0 0x7f77e819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 -- 172.21.15.57:0/409638858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e8101030 msgr2=0x7f77e8199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e8101030 0x7f77e8199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 -- 172.21.15.57:0/409638858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8108ad0 msgr2=0x7f77e819da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8108ad0 0x7f77e819da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 -- 172.21.15.57:0/409638858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e819e180 con 0x7f77e81019d0 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e6d9d700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8108ad0 0x7f77e819da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:31.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.535+0000 7f77e5d9b700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 0x7f77e81996b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b530 tx=0x7f77dc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.536+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc015020 con 0x7f77e81019d0 2024-04-21T08:44:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.536+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77dc011ce0 con 0x7f77e81019d0 2024-04-21T08:44:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.536+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e819e470 con 0x7f77e81019d0 2024-04-21T08:44:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.536+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77dc010e40 con 0x7f77e81019d0 2024-04-21T08:44:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.536+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e81a5cf0 con 0x7f77e81019d0 2024-04-21T08:44:31.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.537+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e8104350 con 0x7f77e81019d0 2024-04-21T08:44:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.538+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f77dc018020 con 0x7f77e81019d0 2024-04-21T08:44:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.538+0000 7f77d77fe700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f77d00a3cd0 0x7f77d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.539+0000 7f77e659c700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f77d00a3cd0 0x7f77d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.539+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f77dc0d0620 con 0x7f77e81019d0 2024-04-21T08:44:31.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.539+0000 7f77e659c700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f77d00a3cd0 0x7f77d00a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b5c0 tx=0x7f77d8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:31.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.542+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77dc01f020 con 0x7f77e81019d0 2024-04-21T08:44:31.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.757+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f77e802df80 con 0x7f77e81019d0 2024-04-21T08:44:31.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.757+0000 7f77d77fe700 1 -- 172.21.15.57:0/409638858 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f77dc09cc10 con 0x7f77e81019d0 2024-04-21T08:44:31.758 INFO:teuthology.orchestra.run.smithi057.stdout:55834574932 2024-04-21T08:44:31.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f77d00a3cd0 msgr2=0x7f77d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f77d00a3cd0 0x7f77d00a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b5c0 tx=0x7f77d8005960 comp rx=0 tx=0).stop 2024-04-21T08:44:31.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 msgr2=0x7f77e81996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:31.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 0x7f77e81996b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b530 tx=0x7f77dc00b840 comp rx=0 tx=0).stop 2024-04-21T08:44:31.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 shutdown_connections 2024-04-21T08:44:31.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f77d00a3cd0 0x7f77d00a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77e8108ad0 0x7f77e819da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77e8101030 0x7f77e8199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 --2- 172.21.15.57:0/409638858 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77e81019d0 0x7f77e81996b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:31.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 >> 172.21.15.57:0/409638858 conn(0x7f77e8077fb0 msgr2=0x7f77e80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:31.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 shutdown_connections 2024-04-21T08:44:31.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:31.760+0000 7f77ec867700 1 -- 172.21.15.57:0/409638858 wait complete. 2024-04-21T08:44:32.216 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495668 got 197568495675 for osd.6 2024-04-21T08:44:32.217 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:32.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:32 smithi129 ceph-mon[31272]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:32.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:32 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/409638858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:44:32.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:32 smithi057 ceph-mon[27968]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:32.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:32 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/409638858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:44:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:32 smithi136 ceph-mon[31791]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:32 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/409638858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-21T08:44:32.902 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070507 got 253403070513 for osd.8 2024-04-21T08:44:32.903 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:32.957 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:33.399 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574932 for osd.1 2024-04-21T08:44:33.399 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:34 smithi129 ceph-mon[31272]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:34 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:34 smithi057 ceph-mon[27968]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:34 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:34 smithi136 ceph-mon[31791]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:34 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 -- 172.21.15.57:0/530560786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00108a50 msgr2=0x7f9b00108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/530560786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00108a50 0x7f9b00108e30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f9af0009990 tx=0x7f9af0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 -- 172.21.15.57:0/530560786 shutdown_connections 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/530560786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b00103450 0x7f9b0010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/530560786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00108a50 0x7f9b00108e30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/530560786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b00102a90 0x7f9b00102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 -- 172.21.15.57:0/530560786 >> 172.21.15.57:0/530560786 conn(0x7f9b00079ba0 msgr2=0x7f9b00079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:35.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 -- 172.21.15.57:0/530560786 shutdown_connections 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.273+0000 7f9b04dbb700 1 -- 172.21.15.57:0/530560786 wait complete. 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 Processor -- start 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 -- start start 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 0x7f9b0019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b00103450 0x7f9b0019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b00108a50 0x7f9b0019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00112be0 con 0x7f9b00102a90 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00112a60 con 0x7f9b00103450 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.274+0000 7f9b04dbb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00112d60 con 0x7f9b00108a50 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 0x7f9b0019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 0x7f9b0019b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47688/0 (socket says 172.21.15.57:47688) 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 -- 172.21.15.57:0/85567966 learned_addr learned my addr 172.21.15.57:0/85567966 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:35.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afdd9b700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b00103450 0x7f9b0019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afed9d700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b00108a50 0x7f9b0019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 -- 172.21.15.57:0/85567966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b00108a50 msgr2=0x7f9b0019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b00108a50 0x7f9b0019fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 -- 172.21.15.57:0/85567966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b00103450 msgr2=0x7f9b0019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b00103450 0x7f9b0019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 -- 172.21.15.57:0/85567966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b001a02c0 con 0x7f9b00102a90 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9afe59c700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 0x7f9b0019b2b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9af0000c00 tx=0x7f9af00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0027070 con 0x7f9b00102a90 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9af0003f20 con 0x7f9b00102a90 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b001a0550 con 0x7f9b00102a90 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0022cf0 con 0x7f9b00102a90 2024-04-21T08:44:35.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.275+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b001a7e30 con 0x7f9b00102a90 2024-04-21T08:44:35.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.276+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b00104180 con 0x7f9b00102a90 2024-04-21T08:44:35.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.277+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9af002a070 con 0x7f9b00102a90 2024-04-21T08:44:35.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.278+0000 7f9aef7fe700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9ae80a3cd0 0x7f9ae80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:35.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.278+0000 7f9afdd9b700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9ae80a3cd0 0x7f9ae80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:35.280 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.278+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f9af00d1a80 con 0x7f9b00102a90 2024-04-21T08:44:35.280 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.279+0000 7f9afdd9b700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9ae80a3cd0 0x7f9ae80a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9b00101180 tx=0x7f9af400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:35.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.282+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9af0013070 con 0x7f9b00102a90 2024-04-21T08:44:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:35.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:35 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:35.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:35.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:35.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:35 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:44:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:44:35.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:35 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' 2024-04-21T08:44:35.492 INFO:teuthology.orchestra.run.smithi057.stdout:287762808875 2024-04-21T08:44:35.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.491+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f9b001a8000 con 0x7f9b00102a90 2024-04-21T08:44:35.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.492+0000 7f9aef7fe700 1 -- 172.21.15.57:0/85567966 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9af009dff0 con 0x7f9b00102a90 2024-04-21T08:44:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9ae80a3cd0 msgr2=0x7f9ae80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:35.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9ae80a3cd0 0x7f9ae80a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9b00101180 tx=0x7f9af400b410 comp rx=0 tx=0).stop 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 msgr2=0x7f9b0019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 0x7f9b0019b2b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9af0000c00 tx=0x7f9af00047b0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 shutdown_connections 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9ae80a3cd0 0x7f9ae80a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b00108a50 0x7f9b0019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b00102a90 0x7f9b0019b2b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 --2- 172.21.15.57:0/85567966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b00103450 0x7f9b0019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:35.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 >> 172.21.15.57:0/85567966 conn(0x7f9b00079ba0 msgr2=0x7f9b0010de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 shutdown_connections 2024-04-21T08:44:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:35.494+0000 7f9b04dbb700 1 -- 172.21.15.57:0/85567966 wait complete. 2024-04-21T08:44:36.290 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808870 got 287762808875 for osd.9 2024-04-21T08:44:36.290 DEBUG:teuthology.parallel:result is None 2024-04-21T08:44:36.291 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-21T08:44:36.291 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:44:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:36 smithi057 ceph-mon[27968]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:36 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/85567966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:44:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:36 smithi136 ceph-mon[31791]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:36 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/85567966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:44:36.544 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:36 smithi129 ceph-mon[31272]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:36 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/85567966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-21T08:44:38.290 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:38 smithi057 ceph-mon[27968]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:38.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:38 smithi136 ceph-mon[31791]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:38.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 -- 172.21.15.57:0/1925693831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 msgr2=0x7fe98810e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:38.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 --2- 172.21.15.57:0/1925693831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98810e7f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe98000b0a0 tx=0x7fe98000b3b0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 -- 172.21.15.57:0/1925693831 shutdown_connections 2024-04-21T08:44:38.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 --2- 172.21.15.57:0/1925693831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98810e7f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 --2- 172.21.15.57:0/1925693831 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe988107920 0x7fe988107d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 --2- 172.21.15.57:0/1925693831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe988101970 0x7fe988101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 -- 172.21.15.57:0/1925693831 >> 172.21.15.57:0/1925693831 conn(0x7fe9880788f0 msgr2=0x7fe988078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 -- 172.21.15.57:0/1925693831 shutdown_connections 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.395+0000 7fe98e717700 1 -- 172.21.15.57:0/1925693831 wait complete. 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 Processor -- start 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 -- start start 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe988101970 0x7fe988199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe988107920 0x7fe9881a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe988111b60 con 0x7fe988102330 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9881119e0 con 0x7fe988107920 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98e717700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe988111ce0 con 0x7fe988101970 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe9877fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:38.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe9877fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47708/0 (socket says 172.21.15.57:47708) 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe9877fe700 1 -- 172.21.15.57:0/141067438 learned_addr learned my addr 172.21.15.57:0/141067438 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe987fff700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe988101970 0x7fe988199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.396+0000 7fe98ccb4700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe988107920 0x7fe9881a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9877fe700 1 -- 172.21.15.57:0/141067438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe988101970 msgr2=0x7fe988199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9877fe700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe988101970 0x7fe988199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9877fe700 1 -- 172.21.15.57:0/141067438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe988107920 msgr2=0x7fe9881a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9877fe700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe988107920 0x7fe9881a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9877fe700 1 -- 172.21.15.57:0/141067438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9881a53c0 con 0x7fe988102330 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9877fe700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98819a340 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe97000c9b0 tx=0x7fe97000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970007b00 con 0x7fe988102330 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe97000ce50 con 0x7fe988102330 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9881a5590 con 0x7fe988102330 2024-04-21T08:44:38.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970018660 con 0x7fe988102330 2024-04-21T08:44:38.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.397+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9881a5aa0 con 0x7fe988102330 2024-04-21T08:44:38.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.398+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe988103060 con 0x7fe988102330 2024-04-21T08:44:38.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.399+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe97000f450 con 0x7fe988102330 2024-04-21T08:44:38.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.400+0000 7fe9857fa700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9740a3cd0 0x7fe9740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:38.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.400+0000 7fe987fff700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9740a3cd0 0x7fe9740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:38.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.400+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fe9700d0800 con 0x7fe988102330 2024-04-21T08:44:38.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.401+0000 7fe987fff700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9740a3cd0 0x7fe9740a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe97800abd0 tx=0x7fe97800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.405+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe970014030 con 0x7fe988102330 2024-04-21T08:44:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:38 smithi129 ceph-mon[31272]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:38.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.607+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9880fff90 con 0x7fe9740a3cd0 2024-04-21T08:44:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.617+0000 7fe9857fa700 1 -- 172.21.15.57:0/141067438 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340578 (secure 0 0 0) 0x7fe9880fff90 con 0x7fe9740a3cd0 2024-04-21T08:44:38.622 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9740a3cd0 msgr2=0x7fe9740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9740a3cd0 0x7fe9740a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe97800abd0 tx=0x7fe97800a380 comp rx=0 tx=0).stop 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 msgr2=0x7fe98819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98819a340 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe97000c9b0 tx=0x7fe97000ccc0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 shutdown_connections 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe9740a3cd0 0x7fe9740a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe988107920 0x7fe9881a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe988101970 0x7fe988199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 --2- 172.21.15.57:0/141067438 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe988102330 0x7fe98819a340 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 >> 172.21.15.57:0/141067438 conn(0x7fe9880788f0 msgr2=0x7fe9880ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.620+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 shutdown_connections 2024-04-21T08:44:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:38.621+0000 7fe98e717700 1 -- 172.21.15.57:0/141067438 wait complete. 2024-04-21T08:44:38.626 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:44:39.472 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-21T08:44:36.630885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654756,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":295,"num_read_kb":262,"num_write":262,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361240,"kb_used_data":12008,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580584,"statfs":{"total":1151940427776,"available":1151570518016,"internally_reserved":0,"allocated":12296192,"data_stored":7949222,"data_compressed":362688,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003564"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467638+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:43:46.467638+0000","last_peered":"2024-04-21T08:43:46.467638+0000","last_clean":"2024-04-21T08:43:46.467638+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:43:46.467638+0000","last_undegraded":"2024-04-21T08:43:46.467638+0000","last_fullsized":"2024-04-21T08:43:46.467638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543979+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:43:46.543979+0000","last_peered":"2024-04-21T08:43:46.543979+0000","last_clean":"2024-04-21T08:43:46.543979+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:43:46.543979+0000","last_undegraded":"2024-04-21T08:43:46.543979+0000","last_fullsized":"2024-04-21T08:43:46.543979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543611+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:43:46.543611+0000","last_peered":"2024-04-21T08:43:46.543611+0000","last_clean":"2024-04-21T08:43:46.543611+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:43:46.543611+0000","last_undegraded":"2024-04-21T08:43:46.543611+0000","last_fullsized":"2024-04-21T08:43:46.543611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468092+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:43:46.468092+0000","last_peered":"2024-04-21T08:43:46.468092+0000","last_clean":"2024-04-21T08:43:46.468092+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:43:46.468092+0000","last_undegraded":"2024-04-21T08:43:46.468092+0000","last_fullsized":"2024-04-21T08:43:46.468092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471964+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:43:46.471964+0000","last_peered":"2024-04-21T08:43:46.471964+0000","last_clean":"2024-04-21T08:43:46.471964+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:43:46.471964+0000","last_undegraded":"2024-04-21T08:43:46.471964+0000","last_fullsized":"2024-04-21T08:43:46.471964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471110+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:43:46.471110+0000","last_peered":"2024-04-21T08:43:46.471110+0000","last_clean":"2024-04-21T08:43:46.471110+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:43:46.471110+0000","last_undegraded":"2024-04-21T08:43:46.471110+0000","last_fullsized":"2024-04-21T08:43:46.471110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469621+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:43:46.469621+0000","last_peered":"2024-04-21T08:43:46.469621+0000","last_clean":"2024-04-21T08:43:46.469621+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:43:46.469621+0000","last_undegraded":"2024-04-21T08:43:46.469621+0000","last_fullsized":"2024-04-21T08:43:46.469621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468896+0000","last_change":"2024-04-21T08:42:57.403514+0000","last_active":"2024-04-21T08:43:46.468896+0000","last_peered":"2024-04-21T08:43:46.468896+0000","last_clean":"2024-04-21T08:43:46.468896+0000","last_became_active":"2024-04-21T08:42:57.403233+0000","last_became_peered":"2024-04-21T08:42:57.403233+0000","last_unstale":"2024-04-21T08:43:46.468896+0000","last_undegraded":"2024-04-21T08:43:46.468896+0000","last_fullsized":"2024-04-21T08:43:46.468896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:14:36.542619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462365+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:43:46.462365+0000","last_peered":"2024-04-21T08:43:46.462365+0000","last_clean":"2024-04-21T08:43:46.462365+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:43:46.462365+0000","last_undegraded":"2024-04-21T08:43:46.462365+0000","last_fullsized":"2024-04-21T08:43:46.462365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471526+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:43:46.471526+0000","last_peered":"2024-04-21T08:43:46.471526+0000","last_clean":"2024-04-21T08:43:46.471526+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:43:46.471526+0000","last_undegraded":"2024-04-21T08:43:46.471526+0000","last_fullsized":"2024-04-21T08:43:46.471526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470448+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:43:46.470448+0000","last_peered":"2024-04-21T08:43:46.470448+0000","last_clean":"2024-04-21T08:43:46.470448+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:43:46.470448+0000","last_undegraded":"2024-04-21T08:43:46.470448+0000","last_fullsized":"2024-04-21T08:43:46.470448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467965+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:43:46.467965+0000","last_peered":"2024-04-21T08:43:46.467965+0000","last_clean":"2024-04-21T08:43:46.467965+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:43:46.467965+0000","last_undegraded":"2024-04-21T08:43:46.467965+0000","last_fullsized":"2024-04-21T08:43:46.467965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469497+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:43:46.469497+0000","last_peered":"2024-04-21T08:43:46.469497+0000","last_clean":"2024-04-21T08:43:46.469497+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:43:46.469497+0000","last_undegraded":"2024-04-21T08:43:46.469497+0000","last_fullsized":"2024-04-21T08:43:46.469497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543545+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:43:46.543545+0000","last_peered":"2024-04-21T08:43:46.543545+0000","last_clean":"2024-04-21T08:43:46.543545+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:43:46.543545+0000","last_undegraded":"2024-04-21T08:43:46.543545+0000","last_fullsized":"2024-04-21T08:43:46.543545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543486+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:43:46.543486+0000","last_peered":"2024-04-21T08:43:46.543486+0000","last_clean":"2024-04-21T08:43:46.543486+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:43:46.543486+0000","last_undegraded":"2024-04-21T08:43:46.543486+0000","last_fullsized":"2024-04-21T08:43:46.543486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461684+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:43:46.461684+0000","last_peered":"2024-04-21T08:43:46.461684+0000","last_clean":"2024-04-21T08:43:46.461684+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:43:46.461684+0000","last_undegraded":"2024-04-21T08:43:46.461684+0000","last_fullsized":"2024-04-21T08:43:46.461684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467268+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:43:46.467268+0000","last_peered":"2024-04-21T08:43:46.467268+0000","last_clean":"2024-04-21T08:43:46.467268+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:43:46.467268+0000","last_undegraded":"2024-04-21T08:43:46.467268+0000","last_fullsized":"2024-04-21T08:43:46.467268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465448+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:43:46.465448+0000","last_peered":"2024-04-21T08:43:46.465448+0000","last_clean":"2024-04-21T08:43:46.465448+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:43:46.465448+0000","last_undegraded":"2024-04-21T08:43:46.465448+0000","last_fullsized":"2024-04-21T08:43:46.465448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464834+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:43:46.464834+0000","last_peered":"2024-04-21T08:43:46.464834+0000","last_clean":"2024-04-21T08:43:46.464834+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:43:46.464834+0000","last_undegraded":"2024-04-21T08:43:46.464834+0000","last_fullsized":"2024-04-21T08:43:46.464834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464246+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:43:46.464246+0000","last_peered":"2024-04-21T08:43:46.464246+0000","last_clean":"2024-04-21T08:43:46.464246+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:43:46.464246+0000","last_undegraded":"2024-04-21T08:43:46.464246+0000","last_fullsized":"2024-04-21T08:43:46.464246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465773+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:43:46.465773+0000","last_peered":"2024-04-21T08:43:46.465773+0000","last_clean":"2024-04-21T08:43:46.465773+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:43:46.465773+0000","last_undegraded":"2024-04-21T08:43:46.465773+0000","last_fullsized":"2024-04-21T08:43:46.465773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470003+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:43:46.470003+0000","last_peered":"2024-04-21T08:43:46.470003+0000","last_clean":"2024-04-21T08:43:46.470003+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:43:46.470003+0000","last_undegraded":"2024-04-21T08:43:46.470003+0000","last_fullsized":"2024-04-21T08:43:46.470003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467123+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:43:46.467123+0000","last_peered":"2024-04-21T08:43:46.467123+0000","last_clean":"2024-04-21T08:43:46.467123+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:43:46.467123+0000","last_undegraded":"2024-04-21T08:43:46.467123+0000","last_fullsized":"2024-04-21T08:43:46.467123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470310+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:43:46.470310+0000","last_peered":"2024-04-21T08:43:46.470310+0000","last_clean":"2024-04-21T08:43:46.470310+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:43:46.470310+0000","last_undegraded":"2024-04-21T08:43:46.470310+0000","last_fullsized":"2024-04-21T08:43:46.470310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544031+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:43:46.544031+0000","last_peered":"2024-04-21T08:43:46.544031+0000","last_clean":"2024-04-21T08:43:46.544031+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:43:46.544031+0000","last_undegraded":"2024-04-21T08:43:46.544031+0000","last_fullsized":"2024-04-21T08:43:46.544031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.474224+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:43:46.474224+0000","last_peered":"2024-04-21T08:43:46.474224+0000","last_clean":"2024-04-21T08:43:46.474224+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:43:46.474224+0000","last_undegraded":"2024-04-21T08:43:46.474224+0000","last_fullsized":"2024-04-21T08:43:46.474224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.473063+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:43:46.473063+0000","last_peered":"2024-04-21T08:43:46.473063+0000","last_clean":"2024-04-21T08:43:46.473063+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:43:46.473063+0000","last_undegraded":"2024-04-21T08:43:46.473063+0000","last_fullsized":"2024-04-21T08:43:46.473063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462022+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:43:46.462022+0000","last_peered":"2024-04-21T08:43:46.462022+0000","last_clean":"2024-04-21T08:43:46.462022+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:43:46.462022+0000","last_undegraded":"2024-04-21T08:43:46.462022+0000","last_fullsized":"2024-04-21T08:43:46.462022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467138+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:43:46.467138+0000","last_peered":"2024-04-21T08:43:46.467138+0000","last_clean":"2024-04-21T08:43:46.467138+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:43:46.467138+0000","last_undegraded":"2024-04-21T08:43:46.467138+0000","last_fullsized":"2024-04-21T08:43:46.467138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471429+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:43:46.471429+0000","last_peered":"2024-04-21T08:43:46.471429+0000","last_clean":"2024-04-21T08:43:46.471429+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:43:46.471429+0000","last_undegraded":"2024-04-21T08:43:46.471429+0000","last_fullsized":"2024-04-21T08:43:46.471429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467687+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:43:46.467687+0000","last_peered":"2024-04-21T08:43:46.467687+0000","last_clean":"2024-04-21T08:43:46.467687+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:43:46.467687+0000","last_undegraded":"2024-04-21T08:43:46.467687+0000","last_fullsized":"2024-04-21T08:43:46.467687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464514+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:43:46.464514+0000","last_peered":"2024-04-21T08:43:46.464514+0000","last_clean":"2024-04-21T08:43:46.464514+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:43:46.464514+0000","last_undegraded":"2024-04-21T08:43:46.464514+0000","last_fullsized":"2024-04-21T08:43:46.464514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.474267+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:43:46.474267+0000","last_peered":"2024-04-21T08:43:46.474267+0000","last_clean":"2024-04-21T08:43:46.474267+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:43:46.474267+0000","last_undegraded":"2024-04-21T08:43:46.474267+0000","last_fullsized":"2024-04-21T08:43:46.474267+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471085+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:43:46.471085+0000","last_peered":"2024-04-21T08:43:46.471085+0000","last_clean":"2024-04-21T08:43:46.471085+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:43:46.471085+0000","last_undegraded":"2024-04-21T08:43:46.471085+0000","last_fullsized":"2024-04-21T08:43:46.471085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464483+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:43:46.464483+0000","last_peered":"2024-04-21T08:43:46.464483+0000","last_clean":"2024-04-21T08:43:46.464483+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:43:46.464483+0000","last_undegraded":"2024-04-21T08:43:46.464483+0000","last_fullsized":"2024-04-21T08:43:46.464483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471190+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:43:46.471190+0000","last_peered":"2024-04-21T08:43:46.471190+0000","last_clean":"2024-04-21T08:43:46.471190+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:43:46.471190+0000","last_undegraded":"2024-04-21T08:43:46.471190+0000","last_fullsized":"2024-04-21T08:43:46.471190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467470+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:43:46.467470+0000","last_peered":"2024-04-21T08:43:46.467470+0000","last_clean":"2024-04-21T08:43:46.467470+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:43:46.467470+0000","last_undegraded":"2024-04-21T08:43:46.467470+0000","last_fullsized":"2024-04-21T08:43:46.467470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467094+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:43:46.467094+0000","last_peered":"2024-04-21T08:43:46.467094+0000","last_clean":"2024-04-21T08:43:46.467094+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:43:46.467094+0000","last_undegraded":"2024-04-21T08:43:46.467094+0000","last_fullsized":"2024-04-21T08:43:46.467094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468992+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:43:46.468992+0000","last_peered":"2024-04-21T08:43:46.468992+0000","last_clean":"2024-04-21T08:43:46.468992+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:43:46.468992+0000","last_undegraded":"2024-04-21T08:43:46.468992+0000","last_fullsized":"2024-04-21T08:43:46.468992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469305+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:43:46.469305+0000","last_peered":"2024-04-21T08:43:46.469305+0000","last_clean":"2024-04-21T08:43:46.469305+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:43:46.469305+0000","last_undegraded":"2024-04-21T08:43:46.469305+0000","last_fullsized":"2024-04-21T08:43:46.469305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543945+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:43:46.543945+0000","last_peered":"2024-04-21T08:43:46.543945+0000","last_clean":"2024-04-21T08:43:46.543945+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:43:46.543945+0000","last_undegraded":"2024-04-21T08:43:46.543945+0000","last_fullsized":"2024-04-21T08:43:46.543945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467471+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:43:46.467471+0000","last_peered":"2024-04-21T08:43:46.467471+0000","last_clean":"2024-04-21T08:43:46.467471+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:43:46.467471+0000","last_undegraded":"2024-04-21T08:43:46.467471+0000","last_fullsized":"2024-04-21T08:43:46.467471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465117+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:43:46.465117+0000","last_peered":"2024-04-21T08:43:46.465117+0000","last_clean":"2024-04-21T08:43:46.465117+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:43:46.465117+0000","last_undegraded":"2024-04-21T08:43:46.465117+0000","last_fullsized":"2024-04-21T08:43:46.465117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469213+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:43:46.469213+0000","last_peered":"2024-04-21T08:43:46.469213+0000","last_clean":"2024-04-21T08:43:46.469213+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:43:46.469213+0000","last_undegraded":"2024-04-21T08:43:46.469213+0000","last_fullsized":"2024-04-21T08:43:46.469213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471349+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:43:46.471349+0000","last_peered":"2024-04-21T08:43:46.471349+0000","last_clean":"2024-04-21T08:43:46.471349+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:43:46.471349+0000","last_undegraded":"2024-04-21T08:43:46.471349+0000","last_fullsized":"2024-04-21T08:43:46.471349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462293+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:43:46.462293+0000","last_peered":"2024-04-21T08:43:46.462293+0000","last_clean":"2024-04-21T08:43:46.462293+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:43:46.462293+0000","last_undegraded":"2024-04-21T08:43:46.462293+0000","last_fullsized":"2024-04-21T08:43:46.462293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465010+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:43:46.465010+0000","last_peered":"2024-04-21T08:43:46.465010+0000","last_clean":"2024-04-21T08:43:46.465010+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:43:46.465010+0000","last_undegraded":"2024-04-21T08:43:46.465010+0000","last_fullsized":"2024-04-21T08:43:46.465010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470011+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:43:46.470011+0000","last_peered":"2024-04-21T08:43:46.470011+0000","last_clean":"2024-04-21T08:43:46.470011+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:43:46.470011+0000","last_undegraded":"2024-04-21T08:43:46.470011+0000","last_fullsized":"2024-04-21T08:43:46.470011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544018+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:43:46.544018+0000","last_peered":"2024-04-21T08:43:46.544018+0000","last_clean":"2024-04-21T08:43:46.544018+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:43:46.544018+0000","last_undegraded":"2024-04-21T08:43:46.544018+0000","last_fullsized":"2024-04-21T08:43:46.544018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468411+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:43:46.468411+0000","last_peered":"2024-04-21T08:43:46.468411+0000","last_clean":"2024-04-21T08:43:46.468411+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:43:46.468411+0000","last_undegraded":"2024-04-21T08:43:46.468411+0000","last_fullsized":"2024-04-21T08:43:46.468411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467612+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:43:46.467612+0000","last_peered":"2024-04-21T08:43:46.467612+0000","last_clean":"2024-04-21T08:43:46.467612+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:43:46.467612+0000","last_undegraded":"2024-04-21T08:43:46.467612+0000","last_fullsized":"2024-04-21T08:43:46.467612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462127+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:43:46.462127+0000","last_peered":"2024-04-21T08:43:46.462127+0000","last_clean":"2024-04-21T08:43:46.462127+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:43:46.462127+0000","last_undegraded":"2024-04-21T08:43:46.462127+0000","last_fullsized":"2024-04-21T08:43:46.462127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465162+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:43:46.465162+0000","last_peered":"2024-04-21T08:43:46.465162+0000","last_clean":"2024-04-21T08:43:46.465162+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:43:46.465162+0000","last_undegraded":"2024-04-21T08:43:46.465162+0000","last_fullsized":"2024-04-21T08:43:46.465162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'30","reported_seq":211,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:44:35.553050+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:44:35.553050+0000","last_peered":"2024-04-21T08:44:35.553050+0000","last_clean":"2024-04-21T08:44:35.553050+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:44:35.553050+0000","last_undegraded":"2024-04-21T08:44:35.553050+0000","last_fullsized":"2024-04-21T08:44:35.553050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":155,"num_read_kb":139,"num_write":30,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472730+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:43:46.472730+0000","last_peered":"2024-04-21T08:43:46.472730+0000","last_clean":"2024-04-21T08:43:46.472730+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:43:46.472730+0000","last_undegraded":"2024-04-21T08:43:46.472730+0000","last_fullsized":"2024-04-21T08:43:46.472730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467923+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:43:46.467923+0000","last_peered":"2024-04-21T08:43:46.467923+0000","last_clean":"2024-04-21T08:43:46.467923+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:43:46.467923+0000","last_undegraded":"2024-04-21T08:43:46.467923+0000","last_fullsized":"2024-04-21T08:43:46.467923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.481120+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:43:46.481120+0000","last_peered":"2024-04-21T08:43:46.481120+0000","last_clean":"2024-04-21T08:43:46.481120+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:43:46.481120+0000","last_undegraded":"2024-04-21T08:43:46.481120+0000","last_fullsized":"2024-04-21T08:43:46.481120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469155+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:43:46.469155+0000","last_peered":"2024-04-21T08:43:46.469155+0000","last_clean":"2024-04-21T08:43:46.469155+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:43:46.469155+0000","last_undegraded":"2024-04-21T08:43:46.469155+0000","last_fullsized":"2024-04-21T08:43:46.469155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466548+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:43:46.466548+0000","last_peered":"2024-04-21T08:43:46.466548+0000","last_clean":"2024-04-21T08:43:46.466548+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:43:46.466548+0000","last_undegraded":"2024-04-21T08:43:46.466548+0000","last_fullsized":"2024-04-21T08:43:46.466548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467992+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:43:46.467992+0000","last_peered":"2024-04-21T08:43:46.467992+0000","last_clean":"2024-04-21T08:43:46.467992+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:43:46.467992+0000","last_undegraded":"2024-04-21T08:43:46.467992+0000","last_fullsized":"2024-04-21T08:43:46.467992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469216+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:43:46.469216+0000","last_peered":"2024-04-21T08:43:46.469216+0000","last_clean":"2024-04-21T08:43:46.469216+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:43:46.469216+0000","last_undegraded":"2024-04-21T08:43:46.469216+0000","last_fullsized":"2024-04-21T08:43:46.469216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465356+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:43:46.465356+0000","last_peered":"2024-04-21T08:43:46.465356+0000","last_clean":"2024-04-21T08:43:46.465356+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:43:46.465356+0000","last_undegraded":"2024-04-21T08:43:46.465356+0000","last_fullsized":"2024-04-21T08:43:46.465356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467448+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:43:46.467448+0000","last_peered":"2024-04-21T08:43:46.467448+0000","last_clean":"2024-04-21T08:43:46.467448+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:43:46.467448+0000","last_undegraded":"2024-04-21T08:43:46.467448+0000","last_fullsized":"2024-04-21T08:43:46.467448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472759+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:43:46.472759+0000","last_peered":"2024-04-21T08:43:46.472759+0000","last_clean":"2024-04-21T08:43:46.472759+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:43:46.472759+0000","last_undegraded":"2024-04-21T08:43:46.472759+0000","last_fullsized":"2024-04-21T08:43:46.472759+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469753+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:43:46.469753+0000","last_peered":"2024-04-21T08:43:46.469753+0000","last_clean":"2024-04-21T08:43:46.469753+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:43:46.469753+0000","last_undegraded":"2024-04-21T08:43:46.469753+0000","last_fullsized":"2024-04-21T08:43:46.469753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470522+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:43:46.470522+0000","last_peered":"2024-04-21T08:43:46.470522+0000","last_clean":"2024-04-21T08:43:46.470522+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:43:46.470522+0000","last_undegraded":"2024-04-21T08:43:46.470522+0000","last_fullsized":"2024-04-21T08:43:46.470522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466954+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:43:46.466954+0000","last_peered":"2024-04-21T08:43:46.466954+0000","last_clean":"2024-04-21T08:43:46.466954+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:43:46.466954+0000","last_undegraded":"2024-04-21T08:43:46.466954+0000","last_fullsized":"2024-04-21T08:43:46.466954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466836+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:43:46.466836+0000","last_peered":"2024-04-21T08:43:46.466836+0000","last_clean":"2024-04-21T08:43:46.466836+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:43:46.466836+0000","last_undegraded":"2024-04-21T08:43:46.466836+0000","last_fullsized":"2024-04-21T08:43:46.466836+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472725+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:43:46.472725+0000","last_peered":"2024-04-21T08:43:46.472725+0000","last_clean":"2024-04-21T08:43:46.472725+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:43:46.472725+0000","last_undegraded":"2024-04-21T08:43:46.472725+0000","last_fullsized":"2024-04-21T08:43:46.472725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472625+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:43:46.472625+0000","last_peered":"2024-04-21T08:43:46.472625+0000","last_clean":"2024-04-21T08:43:46.472625+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:43:46.472625+0000","last_undegraded":"2024-04-21T08:43:46.472625+0000","last_fullsized":"2024-04-21T08:43:46.472625+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470957+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:43:46.470957+0000","last_peered":"2024-04-21T08:43:46.470957+0000","last_clean":"2024-04-21T08:43:46.470957+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:43:46.470957+0000","last_undegraded":"2024-04-21T08:43:46.470957+0000","last_fullsized":"2024-04-21T08:43:46.470957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465494+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:43:46.465494+0000","last_peered":"2024-04-21T08:43:46.465494+0000","last_clean":"2024-04-21T08:43:46.465494+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:43:46.465494+0000","last_undegraded":"2024-04-21T08:43:46.465494+0000","last_fullsized":"2024-04-21T08:43:46.465494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471435+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:43:46.471435+0000","last_peered":"2024-04-21T08:43:46.471435+0000","last_clean":"2024-04-21T08:43:46.471435+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:43:46.471435+0000","last_undegraded":"2024-04-21T08:43:46.471435+0000","last_fullsized":"2024-04-21T08:43:46.471435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469273+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:43:46.469273+0000","last_peered":"2024-04-21T08:43:46.469273+0000","last_clean":"2024-04-21T08:43:46.469273+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:43:46.469273+0000","last_undegraded":"2024-04-21T08:43:46.469273+0000","last_fullsized":"2024-04-21T08:43:46.469273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469956+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:43:46.469956+0000","last_peered":"2024-04-21T08:43:46.469956+0000","last_clean":"2024-04-21T08:43:46.469956+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:43:46.469956+0000","last_undegraded":"2024-04-21T08:43:46.469956+0000","last_fullsized":"2024-04-21T08:43:46.469956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470468+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:43:46.470468+0000","last_peered":"2024-04-21T08:43:46.470468+0000","last_clean":"2024-04-21T08:43:46.470468+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:43:46.470468+0000","last_undegraded":"2024-04-21T08:43:46.470468+0000","last_fullsized":"2024-04-21T08:43:46.470468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469370+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:43:46.469370+0000","last_peered":"2024-04-21T08:43:46.469370+0000","last_clean":"2024-04-21T08:43:46.469370+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:43:46.469370+0000","last_undegraded":"2024-04-21T08:43:46.469370+0000","last_fullsized":"2024-04-21T08:43:46.469370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467639+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:43:46.467639+0000","last_peered":"2024-04-21T08:43:46.467639+0000","last_clean":"2024-04-21T08:43:46.467639+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:43:46.467639+0000","last_undegraded":"2024-04-21T08:43:46.467639+0000","last_fullsized":"2024-04-21T08:43:46.467639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468886+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:43:46.468886+0000","last_peered":"2024-04-21T08:43:46.468886+0000","last_clean":"2024-04-21T08:43:46.468886+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:43:46.468886+0000","last_undegraded":"2024-04-21T08:43:46.468886+0000","last_fullsized":"2024-04-21T08:43:46.468886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470338+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:43:46.470338+0000","last_peered":"2024-04-21T08:43:46.470338+0000","last_clean":"2024-04-21T08:43:46.470338+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:43:46.470338+0000","last_undegraded":"2024-04-21T08:43:46.470338+0000","last_fullsized":"2024-04-21T08:43:46.470338+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469310+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:43:46.469310+0000","last_peered":"2024-04-21T08:43:46.469310+0000","last_clean":"2024-04-21T08:43:46.469310+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:43:46.469310+0000","last_undegraded":"2024-04-21T08:43:46.469310+0000","last_fullsized":"2024-04-21T08:43:46.469310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467944+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:43:46.467944+0000","last_peered":"2024-04-21T08:43:46.467944+0000","last_clean":"2024-04-21T08:43:46.467944+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:43:46.467944+0000","last_undegraded":"2024-04-21T08:43:46.467944+0000","last_fullsized":"2024-04-21T08:43:46.467944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468894+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:43:46.468894+0000","last_peered":"2024-04-21T08:43:46.468894+0000","last_clean":"2024-04-21T08:43:46.468894+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:43:46.468894+0000","last_undegraded":"2024-04-21T08:43:46.468894+0000","last_fullsized":"2024-04-21T08:43:46.468894+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467404+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:43:46.467404+0000","last_peered":"2024-04-21T08:43:46.467404+0000","last_clean":"2024-04-21T08:43:46.467404+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:43:46.467404+0000","last_undegraded":"2024-04-21T08:43:46.467404+0000","last_fullsized":"2024-04-21T08:43:46.467404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468750+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:43:46.468750+0000","last_peered":"2024-04-21T08:43:46.468750+0000","last_clean":"2024-04-21T08:43:46.468750+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:43:46.468750+0000","last_undegraded":"2024-04-21T08:43:46.468750+0000","last_fullsized":"2024-04-21T08:43:46.468750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468961+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:43:46.468961+0000","last_peered":"2024-04-21T08:43:46.468961+0000","last_clean":"2024-04-21T08:43:46.468961+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:43:46.468961+0000","last_undegraded":"2024-04-21T08:43:46.468961+0000","last_fullsized":"2024-04-21T08:43:46.468961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467330+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:43:46.467330+0000","last_peered":"2024-04-21T08:43:46.467330+0000","last_clean":"2024-04-21T08:43:46.467330+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:43:46.467330+0000","last_undegraded":"2024-04-21T08:43:46.467330+0000","last_fullsized":"2024-04-21T08:43:46.467330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472844+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:43:46.472844+0000","last_peered":"2024-04-21T08:43:46.472844+0000","last_clean":"2024-04-21T08:43:46.472844+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:43:46.472844+0000","last_undegraded":"2024-04-21T08:43:46.472844+0000","last_fullsized":"2024-04-21T08:43:46.472844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469562+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:43:46.469562+0000","last_peered":"2024-04-21T08:43:46.469562+0000","last_clean":"2024-04-21T08:43:46.469562+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:43:46.469562+0000","last_undegraded":"2024-04-21T08:43:46.469562+0000","last_fullsized":"2024-04-21T08:43:46.469562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469163+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:43:46.469163+0000","last_peered":"2024-04-21T08:43:46.469163+0000","last_clean":"2024-04-21T08:43:46.469163+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:43:46.469163+0000","last_undegraded":"2024-04-21T08:43:46.469163+0000","last_fullsized":"2024-04-21T08:43:46.469163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465569+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:43:46.465569+0000","last_peered":"2024-04-21T08:43:46.465569+0000","last_clean":"2024-04-21T08:43:46.465569+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:43:46.465569+0000","last_undegraded":"2024-04-21T08:43:46.465569+0000","last_fullsized":"2024-04-21T08:43:46.465569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469957+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:43:46.469957+0000","last_peered":"2024-04-21T08:43:46.469957+0000","last_clean":"2024-04-21T08:43:46.469957+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:43:46.469957+0000","last_undegraded":"2024-04-21T08:43:46.469957+0000","last_fullsized":"2024-04-21T08:43:46.469957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.463976+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:43:46.463976+0000","last_peered":"2024-04-21T08:43:46.463976+0000","last_clean":"2024-04-21T08:43:46.463976+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:43:46.463976+0000","last_undegraded":"2024-04-21T08:43:46.463976+0000","last_fullsized":"2024-04-21T08:43:46.463976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468150+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:43:46.468150+0000","last_peered":"2024-04-21T08:43:46.468150+0000","last_clean":"2024-04-21T08:43:46.468150+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:43:46.468150+0000","last_undegraded":"2024-04-21T08:43:46.468150+0000","last_fullsized":"2024-04-21T08:43:46.468150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544057+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:43:46.544057+0000","last_peered":"2024-04-21T08:43:46.544057+0000","last_clean":"2024-04-21T08:43:46.544057+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:43:46.544057+0000","last_undegraded":"2024-04-21T08:43:46.544057+0000","last_fullsized":"2024-04-21T08:43:46.544057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.474488+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:43:46.474488+0000","last_peered":"2024-04-21T08:43:46.474488+0000","last_clean":"2024-04-21T08:43:46.474488+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:43:46.474488+0000","last_undegraded":"2024-04-21T08:43:46.474488+0000","last_fullsized":"2024-04-21T08:43:46.474488+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461691+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:43:46.461691+0000","last_peered":"2024-04-21T08:43:46.461691+0000","last_clean":"2024-04-21T08:43:46.461691+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:43:46.461691+0000","last_undegraded":"2024-04-21T08:43:46.461691+0000","last_fullsized":"2024-04-21T08:43:46.461691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470661+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:43:46.470661+0000","last_peered":"2024-04-21T08:43:46.470661+0000","last_clean":"2024-04-21T08:43:46.470661+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:43:46.470661+0000","last_undegraded":"2024-04-21T08:43:46.470661+0000","last_fullsized":"2024-04-21T08:43:46.470661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.475804+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:43:46.475804+0000","last_peered":"2024-04-21T08:43:46.475804+0000","last_clean":"2024-04-21T08:43:46.475804+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:43:46.475804+0000","last_undegraded":"2024-04-21T08:43:46.475804+0000","last_fullsized":"2024-04-21T08:43:46.475804+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469313+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:43:46.469313+0000","last_peered":"2024-04-21T08:43:46.469313+0000","last_clean":"2024-04-21T08:43:46.469313+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:43:46.469313+0000","last_undegraded":"2024-04-21T08:43:46.469313+0000","last_fullsized":"2024-04-21T08:43:46.469313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466131+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:43:46.466131+0000","last_peered":"2024-04-21T08:43:46.466131+0000","last_clean":"2024-04-21T08:43:46.466131+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:43:46.466131+0000","last_undegraded":"2024-04-21T08:43:46.466131+0000","last_fullsized":"2024-04-21T08:43:46.466131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467936+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:43:46.467936+0000","last_peered":"2024-04-21T08:43:46.467936+0000","last_clean":"2024-04-21T08:43:46.467936+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:43:46.467936+0000","last_undegraded":"2024-04-21T08:43:46.467936+0000","last_fullsized":"2024-04-21T08:43:46.467936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464107+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:43:46.464107+0000","last_peered":"2024-04-21T08:43:46.464107+0000","last_clean":"2024-04-21T08:43:46.464107+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:43:46.464107+0000","last_undegraded":"2024-04-21T08:43:46.464107+0000","last_fullsized":"2024-04-21T08:43:46.464107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469645+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:43:46.469645+0000","last_peered":"2024-04-21T08:43:46.469645+0000","last_clean":"2024-04-21T08:43:46.469645+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:43:46.469645+0000","last_undegraded":"2024-04-21T08:43:46.469645+0000","last_fullsized":"2024-04-21T08:43:46.469645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467401+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:43:46.467401+0000","last_peered":"2024-04-21T08:43:46.467401+0000","last_clean":"2024-04-21T08:43:46.467401+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:43:46.467401+0000","last_undegraded":"2024-04-21T08:43:46.467401+0000","last_fullsized":"2024-04-21T08:43:46.467401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.477438+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:43:46.477438+0000","last_peered":"2024-04-21T08:43:46.477438+0000","last_clean":"2024-04-21T08:43:46.477438+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:43:46.477438+0000","last_undegraded":"2024-04-21T08:43:46.477438+0000","last_fullsized":"2024-04-21T08:43:46.477438+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470124+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:43:46.470124+0000","last_peered":"2024-04-21T08:43:46.470124+0000","last_clean":"2024-04-21T08:43:46.470124+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:43:46.470124+0000","last_undegraded":"2024-04-21T08:43:46.470124+0000","last_fullsized":"2024-04-21T08:43:46.470124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461703+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:43:46.461703+0000","last_peered":"2024-04-21T08:43:46.461703+0000","last_clean":"2024-04-21T08:43:46.461703+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:43:46.461703+0000","last_undegraded":"2024-04-21T08:43:46.461703+0000","last_fullsized":"2024-04-21T08:43:46.461703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464686+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:43:46.464686+0000","last_peered":"2024-04-21T08:43:46.464686+0000","last_clean":"2024-04-21T08:43:46.464686+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:43:46.464686+0000","last_undegraded":"2024-04-21T08:43:46.464686+0000","last_fullsized":"2024-04-21T08:43:46.464686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471152+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:43:46.471152+0000","last_peered":"2024-04-21T08:43:46.471152+0000","last_clean":"2024-04-21T08:43:46.471152+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:43:46.471152+0000","last_undegraded":"2024-04-21T08:43:46.471152+0000","last_fullsized":"2024-04-21T08:43:46.471152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543623+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:43:46.543623+0000","last_peered":"2024-04-21T08:43:46.543623+0000","last_clean":"2024-04-21T08:43:46.543623+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:43:46.543623+0000","last_undegraded":"2024-04-21T08:43:46.543623+0000","last_fullsized":"2024-04-21T08:43:46.543623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467727+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:43:46.467727+0000","last_peered":"2024-04-21T08:43:46.467727+0000","last_clean":"2024-04-21T08:43:46.467727+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:43:46.467727+0000","last_undegraded":"2024-04-21T08:43:46.467727+0000","last_fullsized":"2024-04-21T08:43:46.467727+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469816+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:43:46.469816+0000","last_peered":"2024-04-21T08:43:46.469816+0000","last_clean":"2024-04-21T08:43:46.469816+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:43:46.469816+0000","last_undegraded":"2024-04-21T08:43:46.469816+0000","last_fullsized":"2024-04-21T08:43:46.469816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466128+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:43:46.466128+0000","last_peered":"2024-04-21T08:43:46.466128+0000","last_clean":"2024-04-21T08:43:46.466128+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:43:46.466128+0000","last_undegraded":"2024-04-21T08:43:46.466128+0000","last_fullsized":"2024-04-21T08:43:46.466128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469639+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:43:46.469639+0000","last_peered":"2024-04-21T08:43:46.469639+0000","last_clean":"2024-04-21T08:43:46.469639+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:43:46.469639+0000","last_undegraded":"2024-04-21T08:43:46.469639+0000","last_fullsized":"2024-04-21T08:43:46.469639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470251+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:43:46.470251+0000","last_peered":"2024-04-21T08:43:46.470251+0000","last_clean":"2024-04-21T08:43:46.470251+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:43:46.470251+0000","last_undegraded":"2024-04-21T08:43:46.470251+0000","last_fullsized":"2024-04-21T08:43:46.470251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466250+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:43:46.466250+0000","last_peered":"2024-04-21T08:43:46.466250+0000","last_clean":"2024-04-21T08:43:46.466250+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:43:46.466250+0000","last_undegraded":"2024-04-21T08:43:46.466250+0000","last_fullsized":"2024-04-21T08:43:46.466250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464249+0000","last_change":"2024-04-21T08:42:57.408595+0000","last_active":"2024-04-21T08:43:46.464249+0000","last_peered":"2024-04-21T08:43:46.464249+0000","last_clean":"2024-04-21T08:43:46.464249+0000","last_became_active":"2024-04-21T08:42:57.408193+0000","last_became_peered":"2024-04-21T08:42:57.408193+0000","last_unstale":"2024-04-21T08:43:46.464249+0000","last_undegraded":"2024-04-21T08:43:46.464249+0000","last_fullsized":"2024-04-21T08:43:46.464249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:04:57.559111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464082+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:43:46.464082+0000","last_peered":"2024-04-21T08:43:46.464082+0000","last_clean":"2024-04-21T08:43:46.464082+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:43:46.464082+0000","last_undegraded":"2024-04-21T08:43:46.464082+0000","last_fullsized":"2024-04-21T08:43:46.464082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543777+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:43:46.543777+0000","last_peered":"2024-04-21T08:43:46.543777+0000","last_clean":"2024-04-21T08:43:46.543777+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:43:46.543777+0000","last_undegraded":"2024-04-21T08:43:46.543777+0000","last_fullsized":"2024-04-21T08:43:46.543777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544928+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:43:46.544928+0000","last_peered":"2024-04-21T08:43:46.544928+0000","last_clean":"2024-04-21T08:43:46.544928+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:43:46.544928+0000","last_undegraded":"2024-04-21T08:43:46.544928+0000","last_fullsized":"2024-04-21T08:43:46.544928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467072+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:43:46.467072+0000","last_peered":"2024-04-21T08:43:46.467072+0000","last_clean":"2024-04-21T08:43:46.467072+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:43:46.467072+0000","last_undegraded":"2024-04-21T08:43:46.467072+0000","last_fullsized":"2024-04-21T08:43:46.467072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464821+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:43:46.464821+0000","last_peered":"2024-04-21T08:43:46.464821+0000","last_clean":"2024-04-21T08:43:46.464821+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:43:46.464821+0000","last_undegraded":"2024-04-21T08:43:46.464821+0000","last_fullsized":"2024-04-21T08:43:46.464821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470477+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:43:46.470477+0000","last_peered":"2024-04-21T08:43:46.470477+0000","last_clean":"2024-04-21T08:43:46.470477+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:43:46.470477+0000","last_undegraded":"2024-04-21T08:43:46.470477+0000","last_fullsized":"2024-04-21T08:43:46.470477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467763+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:43:46.467763+0000","last_peered":"2024-04-21T08:43:46.467763+0000","last_clean":"2024-04-21T08:43:46.467763+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:43:46.467763+0000","last_undegraded":"2024-04-21T08:43:46.467763+0000","last_fullsized":"2024-04-21T08:43:46.467763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469506+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:43:46.469506+0000","last_peered":"2024-04-21T08:43:46.469506+0000","last_clean":"2024-04-21T08:43:46.469506+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:43:46.469506+0000","last_undegraded":"2024-04-21T08:43:46.469506+0000","last_fullsized":"2024-04-21T08:43:46.469506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461889+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:43:46.461889+0000","last_peered":"2024-04-21T08:43:46.461889+0000","last_clean":"2024-04-21T08:43:46.461889+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:43:46.461889+0000","last_undegraded":"2024-04-21T08:43:46.461889+0000","last_fullsized":"2024-04-21T08:43:46.461889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469608+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:43:46.469608+0000","last_peered":"2024-04-21T08:43:46.469608+0000","last_clean":"2024-04-21T08:43:46.469608+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:43:46.469608+0000","last_undegraded":"2024-04-21T08:43:46.469608+0000","last_fullsized":"2024-04-21T08:43:46.469608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466783+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:43:46.466783+0000","last_peered":"2024-04-21T08:43:46.466783+0000","last_clean":"2024-04-21T08:43:46.466783+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:43:46.466783+0000","last_undegraded":"2024-04-21T08:43:46.466783+0000","last_fullsized":"2024-04-21T08:43:46.466783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":251796,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64154,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":159,"num_read_kb":143,"num_write":79,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":192462,"data_compressed":4884,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":500204,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":1149992,"data_compressed":58949,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":504848,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":500774,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":1095216,"data_compressed":57321,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":496074,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":559430,"data_compressed":22277,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":496164,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":559430,"data_compressed":22277,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":496178,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":500192,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":1090720,"data_compressed":57321,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63550,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:39.478 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-21T08:44:39.737 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:40.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:40 smithi136 ceph-mon[31791]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:40.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:40 smithi136 ceph-mon[31791]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:40.141 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:40 smithi057 ceph-mon[27968]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:40.141 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:40 smithi057 ceph-mon[27968]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:40 smithi129 ceph-mon[31272]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:40 smithi129 ceph-mon[31272]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:41.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.624+0000 7f5fe1326700 1 -- 172.21.15.57:0/3316839857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc102ac0 msgr2=0x7f5fdc102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.624+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3316839857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc102f40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009990 tx=0x7f5fd0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 -- 172.21.15.57:0/3316839857 shutdown_connections 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3316839857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc108a70 0x7f5fdc108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3316839857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fdc103480 0x7f5fdc10f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3316839857 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc102f40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 -- 172.21.15.57:0/3316839857 >> 172.21.15.57:0/3316839857 conn(0x7f5fdc079bb0 msgr2=0x7f5fdc079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 -- 172.21.15.57:0/3316839857 shutdown_connections 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 -- 172.21.15.57:0/3316839857 wait complete. 2024-04-21T08:44:41.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 Processor -- start 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.625+0000 7f5fe1326700 1 -- start start 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fe1326700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fe1326700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 0x7f5fdc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fe1326700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fdc108a70 0x7f5fdc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fe1326700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc112ca0 con 0x7f5fdc103480 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fe1326700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc112b20 con 0x7f5fdc102ac0 2024-04-21T08:44:41.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fe1326700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc112e20 con 0x7f5fdc108a70 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 0x7f5fdc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 0x7f5fdc075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47714/0 (socket says 172.21.15.57:47714) 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 -- 172.21.15.57:0/3234441935 learned_addr learned my addr 172.21.15.57:0/3234441935 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fdb7fe700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fdc108a70 0x7f5fdc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fdaffd700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 -- 172.21.15.57:0/3234441935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fdc108a70 msgr2=0x7f5fdc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fdc108a70 0x7f5fdc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 -- 172.21.15.57:0/3234441935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc102ac0 msgr2=0x7f5fdc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 -- 172.21.15.57:0/3234441935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc1a7670 con 0x7f5fdc103480 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fdaffd700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fda7fc700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 0x7f5fdc075280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00057c0 tx=0x7f5fd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.626+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0027070 con 0x7f5fdc103480 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.627+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fdc1a7900 con 0x7f5fdc103480 2024-04-21T08:44:41.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.627+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd0003f20 con 0x7f5fdc103480 2024-04-21T08:44:41.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.627+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0022c30 con 0x7f5fdc103480 2024-04-21T08:44:41.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.627+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fdc1a7e30 con 0x7f5fdc103480 2024-04-21T08:44:41.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.628+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fdc069730 con 0x7f5fdc103480 2024-04-21T08:44:41.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.629+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5fd002a070 con 0x7f5fdc103480 2024-04-21T08:44:41.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.629+0000 7f5fc3fff700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5fc40a3cd0 0x7f5fc40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:41.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.630+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f5fd00d1ff0 con 0x7f5fdc103480 2024-04-21T08:44:41.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.630+0000 7f5fdaffd700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5fc40a3cd0 0x7f5fc40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:41.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.630+0000 7f5fdaffd700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5fc40a3cd0 0x7f5fc40a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc007900 tx=0x7f5fcc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:41.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.633+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fd0031020 con 0x7f5fdc103480 2024-04-21T08:44:41.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.835+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5fdc062e80 con 0x7f5fc40a3cd0 2024-04-21T08:44:41.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.844+0000 7f5fc3fff700 1 -- 172.21.15.57:0/3234441935 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340578 (secure 0 0 0) 0x7f5fdc062e80 con 0x7f5fc40a3cd0 2024-04-21T08:44:41.849 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5fc40a3cd0 msgr2=0x7f5fc40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5fc40a3cd0 0x7f5fc40a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc007900 tx=0x7f5fcc008040 comp rx=0 tx=0).stop 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 msgr2=0x7f5fdc075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 0x7f5fdc075280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00057c0 tx=0x7f5fd00057f0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 shutdown_connections 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f5fc40a3cd0 0x7f5fc40a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fdc102ac0 0x7f5fdc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fdc108a70 0x7f5fdc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 --2- 172.21.15.57:0/3234441935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fdc103480 0x7f5fdc075280 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 >> 172.21.15.57:0/3234441935 conn(0x7f5fdc079bb0 msgr2=0x7f5fdc1017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:41.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 shutdown_connections 2024-04-21T08:44:41.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:41.848+0000 7f5fe1326700 1 -- 172.21.15.57:0/3234441935 wait complete. 2024-04-21T08:44:41.853 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:44:42.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:42 smithi136 ceph-mon[31791]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:42 smithi129 ceph-mon[31272]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:42.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:42 smithi057 ceph-mon[27968]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:42.654 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-21T08:44:40.631890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654756,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":295,"num_read_kb":262,"num_write":262,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":361240,"kb_used_data":12008,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580584,"statfs":{"total":1151940427776,"available":1151570518016,"internally_reserved":0,"allocated":12296192,"data_stored":7949222,"data_compressed":362688,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003265"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467638+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:43:46.467638+0000","last_peered":"2024-04-21T08:43:46.467638+0000","last_clean":"2024-04-21T08:43:46.467638+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:43:46.467638+0000","last_undegraded":"2024-04-21T08:43:46.467638+0000","last_fullsized":"2024-04-21T08:43:46.467638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543979+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:43:46.543979+0000","last_peered":"2024-04-21T08:43:46.543979+0000","last_clean":"2024-04-21T08:43:46.543979+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:43:46.543979+0000","last_undegraded":"2024-04-21T08:43:46.543979+0000","last_fullsized":"2024-04-21T08:43:46.543979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543611+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:43:46.543611+0000","last_peered":"2024-04-21T08:43:46.543611+0000","last_clean":"2024-04-21T08:43:46.543611+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:43:46.543611+0000","last_undegraded":"2024-04-21T08:43:46.543611+0000","last_fullsized":"2024-04-21T08:43:46.543611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468092+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:43:46.468092+0000","last_peered":"2024-04-21T08:43:46.468092+0000","last_clean":"2024-04-21T08:43:46.468092+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:43:46.468092+0000","last_undegraded":"2024-04-21T08:43:46.468092+0000","last_fullsized":"2024-04-21T08:43:46.468092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471964+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:43:46.471964+0000","last_peered":"2024-04-21T08:43:46.471964+0000","last_clean":"2024-04-21T08:43:46.471964+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:43:46.471964+0000","last_undegraded":"2024-04-21T08:43:46.471964+0000","last_fullsized":"2024-04-21T08:43:46.471964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471110+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:43:46.471110+0000","last_peered":"2024-04-21T08:43:46.471110+0000","last_clean":"2024-04-21T08:43:46.471110+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:43:46.471110+0000","last_undegraded":"2024-04-21T08:43:46.471110+0000","last_fullsized":"2024-04-21T08:43:46.471110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469621+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:43:46.469621+0000","last_peered":"2024-04-21T08:43:46.469621+0000","last_clean":"2024-04-21T08:43:46.469621+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:43:46.469621+0000","last_undegraded":"2024-04-21T08:43:46.469621+0000","last_fullsized":"2024-04-21T08:43:46.469621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468896+0000","last_change":"2024-04-21T08:42:57.403514+0000","last_active":"2024-04-21T08:43:46.468896+0000","last_peered":"2024-04-21T08:43:46.468896+0000","last_clean":"2024-04-21T08:43:46.468896+0000","last_became_active":"2024-04-21T08:42:57.403233+0000","last_became_peered":"2024-04-21T08:42:57.403233+0000","last_unstale":"2024-04-21T08:43:46.468896+0000","last_undegraded":"2024-04-21T08:43:46.468896+0000","last_fullsized":"2024-04-21T08:43:46.468896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:14:36.542619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462365+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:43:46.462365+0000","last_peered":"2024-04-21T08:43:46.462365+0000","last_clean":"2024-04-21T08:43:46.462365+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:43:46.462365+0000","last_undegraded":"2024-04-21T08:43:46.462365+0000","last_fullsized":"2024-04-21T08:43:46.462365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471526+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:43:46.471526+0000","last_peered":"2024-04-21T08:43:46.471526+0000","last_clean":"2024-04-21T08:43:46.471526+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:43:46.471526+0000","last_undegraded":"2024-04-21T08:43:46.471526+0000","last_fullsized":"2024-04-21T08:43:46.471526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470448+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:43:46.470448+0000","last_peered":"2024-04-21T08:43:46.470448+0000","last_clean":"2024-04-21T08:43:46.470448+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:43:46.470448+0000","last_undegraded":"2024-04-21T08:43:46.470448+0000","last_fullsized":"2024-04-21T08:43:46.470448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467965+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:43:46.467965+0000","last_peered":"2024-04-21T08:43:46.467965+0000","last_clean":"2024-04-21T08:43:46.467965+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:43:46.467965+0000","last_undegraded":"2024-04-21T08:43:46.467965+0000","last_fullsized":"2024-04-21T08:43:46.467965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469497+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:43:46.469497+0000","last_peered":"2024-04-21T08:43:46.469497+0000","last_clean":"2024-04-21T08:43:46.469497+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:43:46.469497+0000","last_undegraded":"2024-04-21T08:43:46.469497+0000","last_fullsized":"2024-04-21T08:43:46.469497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543545+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:43:46.543545+0000","last_peered":"2024-04-21T08:43:46.543545+0000","last_clean":"2024-04-21T08:43:46.543545+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:43:46.543545+0000","last_undegraded":"2024-04-21T08:43:46.543545+0000","last_fullsized":"2024-04-21T08:43:46.543545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543486+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:43:46.543486+0000","last_peered":"2024-04-21T08:43:46.543486+0000","last_clean":"2024-04-21T08:43:46.543486+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:43:46.543486+0000","last_undegraded":"2024-04-21T08:43:46.543486+0000","last_fullsized":"2024-04-21T08:43:46.543486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461684+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:43:46.461684+0000","last_peered":"2024-04-21T08:43:46.461684+0000","last_clean":"2024-04-21T08:43:46.461684+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:43:46.461684+0000","last_undegraded":"2024-04-21T08:43:46.461684+0000","last_fullsized":"2024-04-21T08:43:46.461684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467268+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:43:46.467268+0000","last_peered":"2024-04-21T08:43:46.467268+0000","last_clean":"2024-04-21T08:43:46.467268+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:43:46.467268+0000","last_undegraded":"2024-04-21T08:43:46.467268+0000","last_fullsized":"2024-04-21T08:43:46.467268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465448+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:43:46.465448+0000","last_peered":"2024-04-21T08:43:46.465448+0000","last_clean":"2024-04-21T08:43:46.465448+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:43:46.465448+0000","last_undegraded":"2024-04-21T08:43:46.465448+0000","last_fullsized":"2024-04-21T08:43:46.465448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464834+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:43:46.464834+0000","last_peered":"2024-04-21T08:43:46.464834+0000","last_clean":"2024-04-21T08:43:46.464834+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:43:46.464834+0000","last_undegraded":"2024-04-21T08:43:46.464834+0000","last_fullsized":"2024-04-21T08:43:46.464834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464246+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:43:46.464246+0000","last_peered":"2024-04-21T08:43:46.464246+0000","last_clean":"2024-04-21T08:43:46.464246+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:43:46.464246+0000","last_undegraded":"2024-04-21T08:43:46.464246+0000","last_fullsized":"2024-04-21T08:43:46.464246+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465773+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:43:46.465773+0000","last_peered":"2024-04-21T08:43:46.465773+0000","last_clean":"2024-04-21T08:43:46.465773+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:43:46.465773+0000","last_undegraded":"2024-04-21T08:43:46.465773+0000","last_fullsized":"2024-04-21T08:43:46.465773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470003+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:43:46.470003+0000","last_peered":"2024-04-21T08:43:46.470003+0000","last_clean":"2024-04-21T08:43:46.470003+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:43:46.470003+0000","last_undegraded":"2024-04-21T08:43:46.470003+0000","last_fullsized":"2024-04-21T08:43:46.470003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467123+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:43:46.467123+0000","last_peered":"2024-04-21T08:43:46.467123+0000","last_clean":"2024-04-21T08:43:46.467123+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:43:46.467123+0000","last_undegraded":"2024-04-21T08:43:46.467123+0000","last_fullsized":"2024-04-21T08:43:46.467123+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470310+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:43:46.470310+0000","last_peered":"2024-04-21T08:43:46.470310+0000","last_clean":"2024-04-21T08:43:46.470310+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:43:46.470310+0000","last_undegraded":"2024-04-21T08:43:46.470310+0000","last_fullsized":"2024-04-21T08:43:46.470310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544031+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:43:46.544031+0000","last_peered":"2024-04-21T08:43:46.544031+0000","last_clean":"2024-04-21T08:43:46.544031+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:43:46.544031+0000","last_undegraded":"2024-04-21T08:43:46.544031+0000","last_fullsized":"2024-04-21T08:43:46.544031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.474224+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:43:46.474224+0000","last_peered":"2024-04-21T08:43:46.474224+0000","last_clean":"2024-04-21T08:43:46.474224+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:43:46.474224+0000","last_undegraded":"2024-04-21T08:43:46.474224+0000","last_fullsized":"2024-04-21T08:43:46.474224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.473063+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:43:46.473063+0000","last_peered":"2024-04-21T08:43:46.473063+0000","last_clean":"2024-04-21T08:43:46.473063+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:43:46.473063+0000","last_undegraded":"2024-04-21T08:43:46.473063+0000","last_fullsized":"2024-04-21T08:43:46.473063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462022+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:43:46.462022+0000","last_peered":"2024-04-21T08:43:46.462022+0000","last_clean":"2024-04-21T08:43:46.462022+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:43:46.462022+0000","last_undegraded":"2024-04-21T08:43:46.462022+0000","last_fullsized":"2024-04-21T08:43:46.462022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467138+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:43:46.467138+0000","last_peered":"2024-04-21T08:43:46.467138+0000","last_clean":"2024-04-21T08:43:46.467138+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:43:46.467138+0000","last_undegraded":"2024-04-21T08:43:46.467138+0000","last_fullsized":"2024-04-21T08:43:46.467138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471429+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:43:46.471429+0000","last_peered":"2024-04-21T08:43:46.471429+0000","last_clean":"2024-04-21T08:43:46.471429+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:43:46.471429+0000","last_undegraded":"2024-04-21T08:43:46.471429+0000","last_fullsized":"2024-04-21T08:43:46.471429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467687+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:43:46.467687+0000","last_peered":"2024-04-21T08:43:46.467687+0000","last_clean":"2024-04-21T08:43:46.467687+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:43:46.467687+0000","last_undegraded":"2024-04-21T08:43:46.467687+0000","last_fullsized":"2024-04-21T08:43:46.467687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464514+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:43:46.464514+0000","last_peered":"2024-04-21T08:43:46.464514+0000","last_clean":"2024-04-21T08:43:46.464514+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:43:46.464514+0000","last_undegraded":"2024-04-21T08:43:46.464514+0000","last_fullsized":"2024-04-21T08:43:46.464514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.474267+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:43:46.474267+0000","last_peered":"2024-04-21T08:43:46.474267+0000","last_clean":"2024-04-21T08:43:46.474267+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:43:46.474267+0000","last_undegraded":"2024-04-21T08:43:46.474267+0000","last_fullsized":"2024-04-21T08:43:46.474267+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471085+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:43:46.471085+0000","last_peered":"2024-04-21T08:43:46.471085+0000","last_clean":"2024-04-21T08:43:46.471085+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:43:46.471085+0000","last_undegraded":"2024-04-21T08:43:46.471085+0000","last_fullsized":"2024-04-21T08:43:46.471085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464483+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:43:46.464483+0000","last_peered":"2024-04-21T08:43:46.464483+0000","last_clean":"2024-04-21T08:43:46.464483+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:43:46.464483+0000","last_undegraded":"2024-04-21T08:43:46.464483+0000","last_fullsized":"2024-04-21T08:43:46.464483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471190+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:43:46.471190+0000","last_peered":"2024-04-21T08:43:46.471190+0000","last_clean":"2024-04-21T08:43:46.471190+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:43:46.471190+0000","last_undegraded":"2024-04-21T08:43:46.471190+0000","last_fullsized":"2024-04-21T08:43:46.471190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467470+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:43:46.467470+0000","last_peered":"2024-04-21T08:43:46.467470+0000","last_clean":"2024-04-21T08:43:46.467470+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:43:46.467470+0000","last_undegraded":"2024-04-21T08:43:46.467470+0000","last_fullsized":"2024-04-21T08:43:46.467470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467094+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:43:46.467094+0000","last_peered":"2024-04-21T08:43:46.467094+0000","last_clean":"2024-04-21T08:43:46.467094+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:43:46.467094+0000","last_undegraded":"2024-04-21T08:43:46.467094+0000","last_fullsized":"2024-04-21T08:43:46.467094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468992+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:43:46.468992+0000","last_peered":"2024-04-21T08:43:46.468992+0000","last_clean":"2024-04-21T08:43:46.468992+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:43:46.468992+0000","last_undegraded":"2024-04-21T08:43:46.468992+0000","last_fullsized":"2024-04-21T08:43:46.468992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469305+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:43:46.469305+0000","last_peered":"2024-04-21T08:43:46.469305+0000","last_clean":"2024-04-21T08:43:46.469305+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:43:46.469305+0000","last_undegraded":"2024-04-21T08:43:46.469305+0000","last_fullsized":"2024-04-21T08:43:46.469305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543945+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:43:46.543945+0000","last_peered":"2024-04-21T08:43:46.543945+0000","last_clean":"2024-04-21T08:43:46.543945+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:43:46.543945+0000","last_undegraded":"2024-04-21T08:43:46.543945+0000","last_fullsized":"2024-04-21T08:43:46.543945+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467471+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:43:46.467471+0000","last_peered":"2024-04-21T08:43:46.467471+0000","last_clean":"2024-04-21T08:43:46.467471+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:43:46.467471+0000","last_undegraded":"2024-04-21T08:43:46.467471+0000","last_fullsized":"2024-04-21T08:43:46.467471+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465117+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:43:46.465117+0000","last_peered":"2024-04-21T08:43:46.465117+0000","last_clean":"2024-04-21T08:43:46.465117+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:43:46.465117+0000","last_undegraded":"2024-04-21T08:43:46.465117+0000","last_fullsized":"2024-04-21T08:43:46.465117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469213+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:43:46.469213+0000","last_peered":"2024-04-21T08:43:46.469213+0000","last_clean":"2024-04-21T08:43:46.469213+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:43:46.469213+0000","last_undegraded":"2024-04-21T08:43:46.469213+0000","last_fullsized":"2024-04-21T08:43:46.469213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471349+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:43:46.471349+0000","last_peered":"2024-04-21T08:43:46.471349+0000","last_clean":"2024-04-21T08:43:46.471349+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:43:46.471349+0000","last_undegraded":"2024-04-21T08:43:46.471349+0000","last_fullsized":"2024-04-21T08:43:46.471349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462293+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:43:46.462293+0000","last_peered":"2024-04-21T08:43:46.462293+0000","last_clean":"2024-04-21T08:43:46.462293+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:43:46.462293+0000","last_undegraded":"2024-04-21T08:43:46.462293+0000","last_fullsized":"2024-04-21T08:43:46.462293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465010+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:43:46.465010+0000","last_peered":"2024-04-21T08:43:46.465010+0000","last_clean":"2024-04-21T08:43:46.465010+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:43:46.465010+0000","last_undegraded":"2024-04-21T08:43:46.465010+0000","last_fullsized":"2024-04-21T08:43:46.465010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470011+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:43:46.470011+0000","last_peered":"2024-04-21T08:43:46.470011+0000","last_clean":"2024-04-21T08:43:46.470011+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:43:46.470011+0000","last_undegraded":"2024-04-21T08:43:46.470011+0000","last_fullsized":"2024-04-21T08:43:46.470011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544018+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:43:46.544018+0000","last_peered":"2024-04-21T08:43:46.544018+0000","last_clean":"2024-04-21T08:43:46.544018+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:43:46.544018+0000","last_undegraded":"2024-04-21T08:43:46.544018+0000","last_fullsized":"2024-04-21T08:43:46.544018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468411+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:43:46.468411+0000","last_peered":"2024-04-21T08:43:46.468411+0000","last_clean":"2024-04-21T08:43:46.468411+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:43:46.468411+0000","last_undegraded":"2024-04-21T08:43:46.468411+0000","last_fullsized":"2024-04-21T08:43:46.468411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467612+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:43:46.467612+0000","last_peered":"2024-04-21T08:43:46.467612+0000","last_clean":"2024-04-21T08:43:46.467612+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:43:46.467612+0000","last_undegraded":"2024-04-21T08:43:46.467612+0000","last_fullsized":"2024-04-21T08:43:46.467612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.462127+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:43:46.462127+0000","last_peered":"2024-04-21T08:43:46.462127+0000","last_clean":"2024-04-21T08:43:46.462127+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:43:46.462127+0000","last_undegraded":"2024-04-21T08:43:46.462127+0000","last_fullsized":"2024-04-21T08:43:46.462127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465162+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:43:46.465162+0000","last_peered":"2024-04-21T08:43:46.465162+0000","last_clean":"2024-04-21T08:43:46.465162+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:43:46.465162+0000","last_undegraded":"2024-04-21T08:43:46.465162+0000","last_fullsized":"2024-04-21T08:43:46.465162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'30","reported_seq":211,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:44:35.553050+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:44:35.553050+0000","last_peered":"2024-04-21T08:44:35.553050+0000","last_clean":"2024-04-21T08:44:35.553050+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:44:35.553050+0000","last_undegraded":"2024-04-21T08:44:35.553050+0000","last_fullsized":"2024-04-21T08:44:35.553050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":155,"num_read_kb":139,"num_write":30,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472730+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:43:46.472730+0000","last_peered":"2024-04-21T08:43:46.472730+0000","last_clean":"2024-04-21T08:43:46.472730+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:43:46.472730+0000","last_undegraded":"2024-04-21T08:43:46.472730+0000","last_fullsized":"2024-04-21T08:43:46.472730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467923+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:43:46.467923+0000","last_peered":"2024-04-21T08:43:46.467923+0000","last_clean":"2024-04-21T08:43:46.467923+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:43:46.467923+0000","last_undegraded":"2024-04-21T08:43:46.467923+0000","last_fullsized":"2024-04-21T08:43:46.467923+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.481120+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:43:46.481120+0000","last_peered":"2024-04-21T08:43:46.481120+0000","last_clean":"2024-04-21T08:43:46.481120+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:43:46.481120+0000","last_undegraded":"2024-04-21T08:43:46.481120+0000","last_fullsized":"2024-04-21T08:43:46.481120+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469155+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:43:46.469155+0000","last_peered":"2024-04-21T08:43:46.469155+0000","last_clean":"2024-04-21T08:43:46.469155+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:43:46.469155+0000","last_undegraded":"2024-04-21T08:43:46.469155+0000","last_fullsized":"2024-04-21T08:43:46.469155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466548+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:43:46.466548+0000","last_peered":"2024-04-21T08:43:46.466548+0000","last_clean":"2024-04-21T08:43:46.466548+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:43:46.466548+0000","last_undegraded":"2024-04-21T08:43:46.466548+0000","last_fullsized":"2024-04-21T08:43:46.466548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467992+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:43:46.467992+0000","last_peered":"2024-04-21T08:43:46.467992+0000","last_clean":"2024-04-21T08:43:46.467992+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:43:46.467992+0000","last_undegraded":"2024-04-21T08:43:46.467992+0000","last_fullsized":"2024-04-21T08:43:46.467992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469216+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:43:46.469216+0000","last_peered":"2024-04-21T08:43:46.469216+0000","last_clean":"2024-04-21T08:43:46.469216+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:43:46.469216+0000","last_undegraded":"2024-04-21T08:43:46.469216+0000","last_fullsized":"2024-04-21T08:43:46.469216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465356+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:43:46.465356+0000","last_peered":"2024-04-21T08:43:46.465356+0000","last_clean":"2024-04-21T08:43:46.465356+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:43:46.465356+0000","last_undegraded":"2024-04-21T08:43:46.465356+0000","last_fullsized":"2024-04-21T08:43:46.465356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467448+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:43:46.467448+0000","last_peered":"2024-04-21T08:43:46.467448+0000","last_clean":"2024-04-21T08:43:46.467448+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:43:46.467448+0000","last_undegraded":"2024-04-21T08:43:46.467448+0000","last_fullsized":"2024-04-21T08:43:46.467448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472759+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:43:46.472759+0000","last_peered":"2024-04-21T08:43:46.472759+0000","last_clean":"2024-04-21T08:43:46.472759+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:43:46.472759+0000","last_undegraded":"2024-04-21T08:43:46.472759+0000","last_fullsized":"2024-04-21T08:43:46.472759+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469753+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:43:46.469753+0000","last_peered":"2024-04-21T08:43:46.469753+0000","last_clean":"2024-04-21T08:43:46.469753+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:43:46.469753+0000","last_undegraded":"2024-04-21T08:43:46.469753+0000","last_fullsized":"2024-04-21T08:43:46.469753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470522+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:43:46.470522+0000","last_peered":"2024-04-21T08:43:46.470522+0000","last_clean":"2024-04-21T08:43:46.470522+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:43:46.470522+0000","last_undegraded":"2024-04-21T08:43:46.470522+0000","last_fullsized":"2024-04-21T08:43:46.470522+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466954+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:43:46.466954+0000","last_peered":"2024-04-21T08:43:46.466954+0000","last_clean":"2024-04-21T08:43:46.466954+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:43:46.466954+0000","last_undegraded":"2024-04-21T08:43:46.466954+0000","last_fullsized":"2024-04-21T08:43:46.466954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466836+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:43:46.466836+0000","last_peered":"2024-04-21T08:43:46.466836+0000","last_clean":"2024-04-21T08:43:46.466836+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:43:46.466836+0000","last_undegraded":"2024-04-21T08:43:46.466836+0000","last_fullsized":"2024-04-21T08:43:46.466836+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472725+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:43:46.472725+0000","last_peered":"2024-04-21T08:43:46.472725+0000","last_clean":"2024-04-21T08:43:46.472725+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:43:46.472725+0000","last_undegraded":"2024-04-21T08:43:46.472725+0000","last_fullsized":"2024-04-21T08:43:46.472725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472625+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:43:46.472625+0000","last_peered":"2024-04-21T08:43:46.472625+0000","last_clean":"2024-04-21T08:43:46.472625+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:43:46.472625+0000","last_undegraded":"2024-04-21T08:43:46.472625+0000","last_fullsized":"2024-04-21T08:43:46.472625+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470957+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:43:46.470957+0000","last_peered":"2024-04-21T08:43:46.470957+0000","last_clean":"2024-04-21T08:43:46.470957+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:43:46.470957+0000","last_undegraded":"2024-04-21T08:43:46.470957+0000","last_fullsized":"2024-04-21T08:43:46.470957+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465494+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:43:46.465494+0000","last_peered":"2024-04-21T08:43:46.465494+0000","last_clean":"2024-04-21T08:43:46.465494+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:43:46.465494+0000","last_undegraded":"2024-04-21T08:43:46.465494+0000","last_fullsized":"2024-04-21T08:43:46.465494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471435+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:43:46.471435+0000","last_peered":"2024-04-21T08:43:46.471435+0000","last_clean":"2024-04-21T08:43:46.471435+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:43:46.471435+0000","last_undegraded":"2024-04-21T08:43:46.471435+0000","last_fullsized":"2024-04-21T08:43:46.471435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469273+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:43:46.469273+0000","last_peered":"2024-04-21T08:43:46.469273+0000","last_clean":"2024-04-21T08:43:46.469273+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:43:46.469273+0000","last_undegraded":"2024-04-21T08:43:46.469273+0000","last_fullsized":"2024-04-21T08:43:46.469273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469956+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:43:46.469956+0000","last_peered":"2024-04-21T08:43:46.469956+0000","last_clean":"2024-04-21T08:43:46.469956+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:43:46.469956+0000","last_undegraded":"2024-04-21T08:43:46.469956+0000","last_fullsized":"2024-04-21T08:43:46.469956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470468+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:43:46.470468+0000","last_peered":"2024-04-21T08:43:46.470468+0000","last_clean":"2024-04-21T08:43:46.470468+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:43:46.470468+0000","last_undegraded":"2024-04-21T08:43:46.470468+0000","last_fullsized":"2024-04-21T08:43:46.470468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469370+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:43:46.469370+0000","last_peered":"2024-04-21T08:43:46.469370+0000","last_clean":"2024-04-21T08:43:46.469370+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:43:46.469370+0000","last_undegraded":"2024-04-21T08:43:46.469370+0000","last_fullsized":"2024-04-21T08:43:46.469370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467639+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:43:46.467639+0000","last_peered":"2024-04-21T08:43:46.467639+0000","last_clean":"2024-04-21T08:43:46.467639+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:43:46.467639+0000","last_undegraded":"2024-04-21T08:43:46.467639+0000","last_fullsized":"2024-04-21T08:43:46.467639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468886+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:43:46.468886+0000","last_peered":"2024-04-21T08:43:46.468886+0000","last_clean":"2024-04-21T08:43:46.468886+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:43:46.468886+0000","last_undegraded":"2024-04-21T08:43:46.468886+0000","last_fullsized":"2024-04-21T08:43:46.468886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470338+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:43:46.470338+0000","last_peered":"2024-04-21T08:43:46.470338+0000","last_clean":"2024-04-21T08:43:46.470338+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:43:46.470338+0000","last_undegraded":"2024-04-21T08:43:46.470338+0000","last_fullsized":"2024-04-21T08:43:46.470338+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469310+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:43:46.469310+0000","last_peered":"2024-04-21T08:43:46.469310+0000","last_clean":"2024-04-21T08:43:46.469310+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:43:46.469310+0000","last_undegraded":"2024-04-21T08:43:46.469310+0000","last_fullsized":"2024-04-21T08:43:46.469310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467944+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:43:46.467944+0000","last_peered":"2024-04-21T08:43:46.467944+0000","last_clean":"2024-04-21T08:43:46.467944+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:43:46.467944+0000","last_undegraded":"2024-04-21T08:43:46.467944+0000","last_fullsized":"2024-04-21T08:43:46.467944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468894+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:43:46.468894+0000","last_peered":"2024-04-21T08:43:46.468894+0000","last_clean":"2024-04-21T08:43:46.468894+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:43:46.468894+0000","last_undegraded":"2024-04-21T08:43:46.468894+0000","last_fullsized":"2024-04-21T08:43:46.468894+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467404+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:43:46.467404+0000","last_peered":"2024-04-21T08:43:46.467404+0000","last_clean":"2024-04-21T08:43:46.467404+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:43:46.467404+0000","last_undegraded":"2024-04-21T08:43:46.467404+0000","last_fullsized":"2024-04-21T08:43:46.467404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468750+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:43:46.468750+0000","last_peered":"2024-04-21T08:43:46.468750+0000","last_clean":"2024-04-21T08:43:46.468750+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:43:46.468750+0000","last_undegraded":"2024-04-21T08:43:46.468750+0000","last_fullsized":"2024-04-21T08:43:46.468750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468961+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:43:46.468961+0000","last_peered":"2024-04-21T08:43:46.468961+0000","last_clean":"2024-04-21T08:43:46.468961+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:43:46.468961+0000","last_undegraded":"2024-04-21T08:43:46.468961+0000","last_fullsized":"2024-04-21T08:43:46.468961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467330+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:43:46.467330+0000","last_peered":"2024-04-21T08:43:46.467330+0000","last_clean":"2024-04-21T08:43:46.467330+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:43:46.467330+0000","last_undegraded":"2024-04-21T08:43:46.467330+0000","last_fullsized":"2024-04-21T08:43:46.467330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.472844+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:43:46.472844+0000","last_peered":"2024-04-21T08:43:46.472844+0000","last_clean":"2024-04-21T08:43:46.472844+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:43:46.472844+0000","last_undegraded":"2024-04-21T08:43:46.472844+0000","last_fullsized":"2024-04-21T08:43:46.472844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469562+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:43:46.469562+0000","last_peered":"2024-04-21T08:43:46.469562+0000","last_clean":"2024-04-21T08:43:46.469562+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:43:46.469562+0000","last_undegraded":"2024-04-21T08:43:46.469562+0000","last_fullsized":"2024-04-21T08:43:46.469562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469163+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:43:46.469163+0000","last_peered":"2024-04-21T08:43:46.469163+0000","last_clean":"2024-04-21T08:43:46.469163+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:43:46.469163+0000","last_undegraded":"2024-04-21T08:43:46.469163+0000","last_fullsized":"2024-04-21T08:43:46.469163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.465569+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:43:46.465569+0000","last_peered":"2024-04-21T08:43:46.465569+0000","last_clean":"2024-04-21T08:43:46.465569+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:43:46.465569+0000","last_undegraded":"2024-04-21T08:43:46.465569+0000","last_fullsized":"2024-04-21T08:43:46.465569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469957+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:43:46.469957+0000","last_peered":"2024-04-21T08:43:46.469957+0000","last_clean":"2024-04-21T08:43:46.469957+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:43:46.469957+0000","last_undegraded":"2024-04-21T08:43:46.469957+0000","last_fullsized":"2024-04-21T08:43:46.469957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.463976+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:43:46.463976+0000","last_peered":"2024-04-21T08:43:46.463976+0000","last_clean":"2024-04-21T08:43:46.463976+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:43:46.463976+0000","last_undegraded":"2024-04-21T08:43:46.463976+0000","last_fullsized":"2024-04-21T08:43:46.463976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.468150+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:43:46.468150+0000","last_peered":"2024-04-21T08:43:46.468150+0000","last_clean":"2024-04-21T08:43:46.468150+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:43:46.468150+0000","last_undegraded":"2024-04-21T08:43:46.468150+0000","last_fullsized":"2024-04-21T08:43:46.468150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544057+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:43:46.544057+0000","last_peered":"2024-04-21T08:43:46.544057+0000","last_clean":"2024-04-21T08:43:46.544057+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:43:46.544057+0000","last_undegraded":"2024-04-21T08:43:46.544057+0000","last_fullsized":"2024-04-21T08:43:46.544057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.474488+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:43:46.474488+0000","last_peered":"2024-04-21T08:43:46.474488+0000","last_clean":"2024-04-21T08:43:46.474488+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:43:46.474488+0000","last_undegraded":"2024-04-21T08:43:46.474488+0000","last_fullsized":"2024-04-21T08:43:46.474488+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461691+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:43:46.461691+0000","last_peered":"2024-04-21T08:43:46.461691+0000","last_clean":"2024-04-21T08:43:46.461691+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:43:46.461691+0000","last_undegraded":"2024-04-21T08:43:46.461691+0000","last_fullsized":"2024-04-21T08:43:46.461691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470661+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:43:46.470661+0000","last_peered":"2024-04-21T08:43:46.470661+0000","last_clean":"2024-04-21T08:43:46.470661+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:43:46.470661+0000","last_undegraded":"2024-04-21T08:43:46.470661+0000","last_fullsized":"2024-04-21T08:43:46.470661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.475804+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:43:46.475804+0000","last_peered":"2024-04-21T08:43:46.475804+0000","last_clean":"2024-04-21T08:43:46.475804+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:43:46.475804+0000","last_undegraded":"2024-04-21T08:43:46.475804+0000","last_fullsized":"2024-04-21T08:43:46.475804+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469313+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:43:46.469313+0000","last_peered":"2024-04-21T08:43:46.469313+0000","last_clean":"2024-04-21T08:43:46.469313+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:43:46.469313+0000","last_undegraded":"2024-04-21T08:43:46.469313+0000","last_fullsized":"2024-04-21T08:43:46.469313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466131+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:43:46.466131+0000","last_peered":"2024-04-21T08:43:46.466131+0000","last_clean":"2024-04-21T08:43:46.466131+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:43:46.466131+0000","last_undegraded":"2024-04-21T08:43:46.466131+0000","last_fullsized":"2024-04-21T08:43:46.466131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467936+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:43:46.467936+0000","last_peered":"2024-04-21T08:43:46.467936+0000","last_clean":"2024-04-21T08:43:46.467936+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:43:46.467936+0000","last_undegraded":"2024-04-21T08:43:46.467936+0000","last_fullsized":"2024-04-21T08:43:46.467936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464107+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:43:46.464107+0000","last_peered":"2024-04-21T08:43:46.464107+0000","last_clean":"2024-04-21T08:43:46.464107+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:43:46.464107+0000","last_undegraded":"2024-04-21T08:43:46.464107+0000","last_fullsized":"2024-04-21T08:43:46.464107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469645+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:43:46.469645+0000","last_peered":"2024-04-21T08:43:46.469645+0000","last_clean":"2024-04-21T08:43:46.469645+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:43:46.469645+0000","last_undegraded":"2024-04-21T08:43:46.469645+0000","last_fullsized":"2024-04-21T08:43:46.469645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467401+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:43:46.467401+0000","last_peered":"2024-04-21T08:43:46.467401+0000","last_clean":"2024-04-21T08:43:46.467401+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:43:46.467401+0000","last_undegraded":"2024-04-21T08:43:46.467401+0000","last_fullsized":"2024-04-21T08:43:46.467401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.477438+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:43:46.477438+0000","last_peered":"2024-04-21T08:43:46.477438+0000","last_clean":"2024-04-21T08:43:46.477438+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:43:46.477438+0000","last_undegraded":"2024-04-21T08:43:46.477438+0000","last_fullsized":"2024-04-21T08:43:46.477438+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470124+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:43:46.470124+0000","last_peered":"2024-04-21T08:43:46.470124+0000","last_clean":"2024-04-21T08:43:46.470124+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:43:46.470124+0000","last_undegraded":"2024-04-21T08:43:46.470124+0000","last_fullsized":"2024-04-21T08:43:46.470124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461703+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:43:46.461703+0000","last_peered":"2024-04-21T08:43:46.461703+0000","last_clean":"2024-04-21T08:43:46.461703+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:43:46.461703+0000","last_undegraded":"2024-04-21T08:43:46.461703+0000","last_fullsized":"2024-04-21T08:43:46.461703+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464686+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:43:46.464686+0000","last_peered":"2024-04-21T08:43:46.464686+0000","last_clean":"2024-04-21T08:43:46.464686+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:43:46.464686+0000","last_undegraded":"2024-04-21T08:43:46.464686+0000","last_fullsized":"2024-04-21T08:43:46.464686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.471152+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:43:46.471152+0000","last_peered":"2024-04-21T08:43:46.471152+0000","last_clean":"2024-04-21T08:43:46.471152+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:43:46.471152+0000","last_undegraded":"2024-04-21T08:43:46.471152+0000","last_fullsized":"2024-04-21T08:43:46.471152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543623+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:43:46.543623+0000","last_peered":"2024-04-21T08:43:46.543623+0000","last_clean":"2024-04-21T08:43:46.543623+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:43:46.543623+0000","last_undegraded":"2024-04-21T08:43:46.543623+0000","last_fullsized":"2024-04-21T08:43:46.543623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467727+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:43:46.467727+0000","last_peered":"2024-04-21T08:43:46.467727+0000","last_clean":"2024-04-21T08:43:46.467727+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:43:46.467727+0000","last_undegraded":"2024-04-21T08:43:46.467727+0000","last_fullsized":"2024-04-21T08:43:46.467727+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469816+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:43:46.469816+0000","last_peered":"2024-04-21T08:43:46.469816+0000","last_clean":"2024-04-21T08:43:46.469816+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:43:46.469816+0000","last_undegraded":"2024-04-21T08:43:46.469816+0000","last_fullsized":"2024-04-21T08:43:46.469816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466128+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:43:46.466128+0000","last_peered":"2024-04-21T08:43:46.466128+0000","last_clean":"2024-04-21T08:43:46.466128+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:43:46.466128+0000","last_undegraded":"2024-04-21T08:43:46.466128+0000","last_fullsized":"2024-04-21T08:43:46.466128+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469639+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:43:46.469639+0000","last_peered":"2024-04-21T08:43:46.469639+0000","last_clean":"2024-04-21T08:43:46.469639+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:43:46.469639+0000","last_undegraded":"2024-04-21T08:43:46.469639+0000","last_fullsized":"2024-04-21T08:43:46.469639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470251+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:43:46.470251+0000","last_peered":"2024-04-21T08:43:46.470251+0000","last_clean":"2024-04-21T08:43:46.470251+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:43:46.470251+0000","last_undegraded":"2024-04-21T08:43:46.470251+0000","last_fullsized":"2024-04-21T08:43:46.470251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466250+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:43:46.466250+0000","last_peered":"2024-04-21T08:43:46.466250+0000","last_clean":"2024-04-21T08:43:46.466250+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:43:46.466250+0000","last_undegraded":"2024-04-21T08:43:46.466250+0000","last_fullsized":"2024-04-21T08:43:46.466250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464249+0000","last_change":"2024-04-21T08:42:57.408595+0000","last_active":"2024-04-21T08:43:46.464249+0000","last_peered":"2024-04-21T08:43:46.464249+0000","last_clean":"2024-04-21T08:43:46.464249+0000","last_became_active":"2024-04-21T08:42:57.408193+0000","last_became_peered":"2024-04-21T08:42:57.408193+0000","last_unstale":"2024-04-21T08:43:46.464249+0000","last_undegraded":"2024-04-21T08:43:46.464249+0000","last_fullsized":"2024-04-21T08:43:46.464249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:04:57.559111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464082+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:43:46.464082+0000","last_peered":"2024-04-21T08:43:46.464082+0000","last_clean":"2024-04-21T08:43:46.464082+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:43:46.464082+0000","last_undegraded":"2024-04-21T08:43:46.464082+0000","last_fullsized":"2024-04-21T08:43:46.464082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.543777+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:43:46.543777+0000","last_peered":"2024-04-21T08:43:46.543777+0000","last_clean":"2024-04-21T08:43:46.543777+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:43:46.543777+0000","last_undegraded":"2024-04-21T08:43:46.543777+0000","last_fullsized":"2024-04-21T08:43:46.543777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.544928+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:43:46.544928+0000","last_peered":"2024-04-21T08:43:46.544928+0000","last_clean":"2024-04-21T08:43:46.544928+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:43:46.544928+0000","last_undegraded":"2024-04-21T08:43:46.544928+0000","last_fullsized":"2024-04-21T08:43:46.544928+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467072+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:43:46.467072+0000","last_peered":"2024-04-21T08:43:46.467072+0000","last_clean":"2024-04-21T08:43:46.467072+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:43:46.467072+0000","last_undegraded":"2024-04-21T08:43:46.467072+0000","last_fullsized":"2024-04-21T08:43:46.467072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.464821+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:43:46.464821+0000","last_peered":"2024-04-21T08:43:46.464821+0000","last_clean":"2024-04-21T08:43:46.464821+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:43:46.464821+0000","last_undegraded":"2024-04-21T08:43:46.464821+0000","last_fullsized":"2024-04-21T08:43:46.464821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.470477+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:43:46.470477+0000","last_peered":"2024-04-21T08:43:46.470477+0000","last_clean":"2024-04-21T08:43:46.470477+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:43:46.470477+0000","last_undegraded":"2024-04-21T08:43:46.470477+0000","last_fullsized":"2024-04-21T08:43:46.470477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.467763+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:43:46.467763+0000","last_peered":"2024-04-21T08:43:46.467763+0000","last_clean":"2024-04-21T08:43:46.467763+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:43:46.467763+0000","last_undegraded":"2024-04-21T08:43:46.467763+0000","last_fullsized":"2024-04-21T08:43:46.467763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469506+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:43:46.469506+0000","last_peered":"2024-04-21T08:43:46.469506+0000","last_clean":"2024-04-21T08:43:46.469506+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:43:46.469506+0000","last_undegraded":"2024-04-21T08:43:46.469506+0000","last_fullsized":"2024-04-21T08:43:46.469506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.461889+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:43:46.461889+0000","last_peered":"2024-04-21T08:43:46.461889+0000","last_clean":"2024-04-21T08:43:46.461889+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:43:46.461889+0000","last_undegraded":"2024-04-21T08:43:46.461889+0000","last_fullsized":"2024-04-21T08:43:46.461889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.469608+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:43:46.469608+0000","last_peered":"2024-04-21T08:43:46.469608+0000","last_clean":"2024-04-21T08:43:46.469608+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:43:46.469608+0000","last_undegraded":"2024-04-21T08:43:46.469608+0000","last_fullsized":"2024-04-21T08:43:46.469608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-21T08:43:46.466783+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:43:46.466783+0000","last_peered":"2024-04-21T08:43:46.466783+0000","last_clean":"2024-04-21T08:43:46.466783+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:43:46.466783+0000","last_undegraded":"2024-04-21T08:43:46.466783+0000","last_fullsized":"2024-04-21T08:43:46.466783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":251796,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64154,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":159,"num_read_kb":143,"num_write":79,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":192462,"data_compressed":4884,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":995328,"data_stored":500204,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":1149992,"data_compressed":58949,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":504848,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":999424,"data_stored":500774,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":1095216,"data_compressed":57321,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":496074,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":559430,"data_compressed":22277,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":496164,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1056768,"data_stored":559430,"data_compressed":22277,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":496178,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":500192,"data_compressed":20649,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":1090720,"data_compressed":57321,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":35036,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63550,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:44:42.659 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-21T08:44:42.659 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-21T08:44:42.659 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-21T08:44:42.660 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph health --format=json 2024-04-21T08:44:42.910 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:43.306 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:43 smithi057 ceph-mon[27968]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:43.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:43 smithi136 ceph-mon[31791]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:43.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:43 smithi129 ceph-mon[31272]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:44:44.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:44 smithi136 ceph-mon[31791]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:44.389 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:44 smithi057 ceph-mon[27968]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:44.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:44 smithi129 ceph-mon[31272]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:44:44.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 -- 172.21.15.57:0/1124908577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f806b900 msgr2=0x7f67f810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 --2- 172.21.15.57:0/1124908577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f806b900 0x7f67f810efa0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009990 tx=0x7f67e0009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 -- 172.21.15.57:0/1124908577 shutdown_connections 2024-04-21T08:44:44.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 --2- 172.21.15.57:0/1124908577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f810f4e0 0x7f67f81118d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 --2- 172.21.15.57:0/1124908577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f806b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 --2- 172.21.15.57:0/1124908577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f806b900 0x7f67f810efa0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 -- 172.21.15.57:0/1124908577 >> 172.21.15.57:0/1124908577 conn(0x7f67f80788b0 msgr2=0x7f67f8078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 -- 172.21.15.57:0/1124908577 shutdown_connections 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.672+0000 7f67fce76700 1 -- 172.21.15.57:0/1124908577 wait complete. 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 Processor -- start 2024-04-21T08:44:44.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 -- start start 2024-04-21T08:44:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f810ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f806b900 0x7f67f810be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 0x7f67f810c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f67f8114df0 con 0x7f67f810f4e0 2024-04-21T08:44:44.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67f8114c70 con 0x7f67f806b900 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67fce76700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f67f8114f70 con 0x7f67f806afe0 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.673+0000 7f67f6d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 0x7f67f810c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f810ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 0x7f67f810c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33384/0 (socket says 172.21.15.57:33384) 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f659c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f810ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:44494/0 (socket says 172.21.15.57:44494) 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 -- 172.21.15.57:0/668388675 learned_addr learned my addr 172.21.15.57:0/668388675 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67effff700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f806b900 0x7f67f810be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 -- 172.21.15.57:0/668388675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 msgr2=0x7f67f810ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f810ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 -- 172.21.15.57:0/668388675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f806b900 msgr2=0x7f67f810be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f806b900 0x7f67f810be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 -- 172.21.15.57:0/668388675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f810cc90 con 0x7f67f810f4e0 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f659c700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f810ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:44.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67effff700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f806b900 0x7f67f810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67f6d9d700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 0x7f67f810c390 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f67f0007d40 tx=0x7f67f000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f0016020 con 0x7f67f810f4e0 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f81abb90 con 0x7f67f810f4e0 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.674+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67f000ac90 con 0x7f67f810f4e0 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.675+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f00115e0 con 0x7f67f810f4e0 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.675+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f81ac050 con 0x7f67f810f4e0 2024-04-21T08:44:44.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.676+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8101f50 con 0x7f67f810f4e0 2024-04-21T08:44:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.677+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f67f0019020 con 0x7f67f810f4e0 2024-04-21T08:44:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.677+0000 7f67ef7fe700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67e40a3cd0 0x7f67e40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.678+0000 7f67f659c700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67e40a3cd0 0x7f67e40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.678+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f67f00d0c50 con 0x7f67f810f4e0 2024-04-21T08:44:44.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.678+0000 7f67f659c700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67e40a3cd0 0x7f67e40a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67e800afd0 tx=0x7f67e800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:44.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.681+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67f0020020 con 0x7f67f810f4e0 2024-04-21T08:44:44.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.936+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f67f8069730 con 0x7f67f810f4e0 2024-04-21T08:44:44.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.937+0000 7f67ef7fe700 1 -- 172.21.15.57:0/668388675 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f67f009d2a0 con 0x7f67f810f4e0 2024-04-21T08:44:44.936 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:44.936 INFO:teuthology.orchestra.run.smithi057.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-21T08:44:44.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67e40a3cd0 msgr2=0x7f67e40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67e40a3cd0 0x7f67e40a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67e800afd0 tx=0x7f67e800a380 comp rx=0 tx=0).stop 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 msgr2=0x7f67f810c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 0x7f67f810c390 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f67f0007d40 tx=0x7f67f000a340 comp rx=0 tx=0).stop 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 shutdown_connections 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f67e40a3cd0 0x7f67e40a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67f810f4e0 0x7f67f810c390 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f806afe0 0x7f67f810ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 --2- 172.21.15.57:0/668388675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67f806b900 0x7f67f810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 >> 172.21.15.57:0/668388675 conn(0x7f67f80788b0 msgr2=0x7f67f81107f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 shutdown_connections 2024-04-21T08:44:44.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:44.939+0000 7f67fce76700 1 -- 172.21.15.57:0/668388675 wait complete. 2024-04-21T08:44:45.317 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:45 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/668388675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:44:45.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:45 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/668388675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:44:45.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:45 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/668388675' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-21T08:44:45.773 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-21T08:44:45.773 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 -- 172.21.15.57:0/2594971261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 msgr2=0x7fc58810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 --2- 172.21.15.57:0/2594971261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc58810a0c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc580009ca0 tx=0x7fc58002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 -- 172.21.15.57:0/2594971261 shutdown_connections 2024-04-21T08:44:45.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 --2- 172.21.15.57:0/2594971261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc58810a0c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 --2- 172.21.15.57:0/2594971261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc5881053a0 0x7fc588107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 --2- 172.21.15.57:0/2594971261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588102a70 0x7fc588104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 -- 172.21.15.57:0/2594971261 >> 172.21.15.57:0/2594971261 conn(0x7fc5880fc3e0 msgr2=0x7fc5880fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 -- 172.21.15.57:0/2594971261 shutdown_connections 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 -- 172.21.15.57:0/2594971261 wait complete. 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.880+0000 7fc59004e640 1 Processor -- start 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 -- start start 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc588102a70 0x7fc58819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5881053a0 0x7fc58819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc5881a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc58810daa0 con 0x7fc588107cd0 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc58810d920 con 0x7fc5881053a0 2024-04-21T08:44:45.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc59004e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc58810dc20 con 0x7fc588102a70 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc5881a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc5881a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33406/0 (socket says 172.21.15.57:33406) 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 -- 172.21.15.57:0/2622345400 learned_addr learned my addr 172.21.15.57:0/2622345400 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58d5e9640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5881053a0 0x7fc58819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58ddea640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc588102a70 0x7fc58819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 -- 172.21.15.57:0/2622345400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc588102a70 msgr2=0x7fc58819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc588102a70 0x7fc58819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 -- 172.21.15.57:0/2622345400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5881053a0 msgr2=0x7fc58819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5881053a0 0x7fc58819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 -- 172.21.15.57:0/2622345400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5881a1ef0 con 0x7fc588107cd0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.881+0000 7fc58e5eb640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc5881a1880 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc58002c700 tx=0x7fc580034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.882+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580039040 con 0x7fc588107cd0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.882+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc580044070 con 0x7fc588107cd0 2024-04-21T08:44:45.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.882+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5881a2180 con 0x7fc588107cd0 2024-04-21T08:44:45.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.882+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800042a0 con 0x7fc588107cd0 2024-04-21T08:44:45.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.882+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5881a2670 con 0x7fc588107cd0 2024-04-21T08:44:45.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.883+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5881097c0 con 0x7fc588107cd0 2024-04-21T08:44:45.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.884+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc5800325c0 con 0x7fc588107cd0 2024-04-21T08:44:45.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.884+0000 7fc57effd640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc5740a3bb0 0x7fc5740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:45.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.885+0000 7fc58ddea640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc5740a3bb0 0x7fc5740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:45.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.885+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fc5800f2b80 con 0x7fc588107cd0 2024-04-21T08:44:45.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.885+0000 7fc58ddea640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc5740a3bb0 0x7fc5740a6070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc584005fd0 tx=0x7fc584005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:45.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:45.889+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5800bf1c0 con 0x7fc588107cd0 2024-04-21T08:44:46.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.075+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc588062b50 con 0x7fc588107cd0 2024-04-21T08:44:46.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.077+0000 7fc57effd640 1 -- 172.21.15.57:0/2622345400 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24603 (secure 0 0 0) 0x7fc5800beac0 con 0x7fc588107cd0 2024-04-21T08:44:46.076 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:46.077 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":91,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:43:46.457056+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:46.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc5740a3bb0 msgr2=0x7fc5740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc5740a3bb0 0x7fc5740a6070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc584005fd0 tx=0x7fc584005850 comp rx=0 tx=0).stop 2024-04-21T08:44:46.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 msgr2=0x7fc5881a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc5881a1880 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc58002c700 tx=0x7fc580034760 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 shutdown_connections 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc5740a3bb0 0x7fc5740a6070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588107cd0 0x7fc5881a1880 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5881053a0 0x7fc58819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 --2- 172.21.15.57:0/2622345400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc588102a70 0x7fc58819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 >> 172.21.15.57:0/2622345400 conn(0x7fc5880fc3e0 msgr2=0x7fc588102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 shutdown_connections 2024-04-21T08:44:46.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.079+0000 7fc59004e640 1 -- 172.21.15.57:0/2622345400 wait complete. 2024-04-21T08:44:46.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:46.091 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 -- 172.21.15.57:0/1983243552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe138101ac0 msgr2=0x7fe138109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 --2- 172.21.15.57:0/1983243552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe138101ac0 0x7fe138109360 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe130009ca0 tx=0x7fe13002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 -- 172.21.15.57:0/1983243552 shutdown_connections 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 --2- 172.21.15.57:0/1983243552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe138101ac0 0x7fe138109360 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 --2- 172.21.15.57:0/1983243552 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe138101100 0x7fe138101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 --2- 172.21.15.57:0/1983243552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1381024b0 0x7fe1381028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 -- 172.21.15.57:0/1983243552 >> 172.21.15.57:0/1983243552 conn(0x7fe1380fbb90 msgr2=0x7fe1380fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 -- 172.21.15.57:0/1983243552 shutdown_connections 2024-04-21T08:44:46.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.197+0000 7fe13efad640 1 -- 172.21.15.57:0/1983243552 wait complete. 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 Processor -- start 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 -- start start 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe138101100 0x7fe138198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe138101ac0 0x7fe1381986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 0x7fe13819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe13810c9e0 con 0x7fe1381024b0 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe13810c860 con 0x7fe138101100 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13efad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe13810cb60 con 0x7fe138101ac0 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 0x7fe13819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 0x7fe13819f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33428/0 (socket says 172.21.15.57:33428) 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 -- 172.21.15.57:0/3412290292 learned_addr learned my addr 172.21.15.57:0/3412290292 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe137fff640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe138101ac0 0x7fe1381986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13cd49640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe138101100 0x7fe138198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 -- 172.21.15.57:0/3412290292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe138101ac0 msgr2=0x7fe1381986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe138101ac0 0x7fe1381986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 -- 172.21.15.57:0/3412290292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe138101100 msgr2=0x7fe138198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe138101100 0x7fe138198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 -- 172.21.15.57:0/3412290292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13819fda0 con 0x7fe1381024b0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13cd49640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe138101100 0x7fe138198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.198+0000 7fe13d54a640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 0x7fe13819f730 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe130032520 tx=0x7fe1300347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.199+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130039040 con 0x7fe1381024b0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.199+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe130044070 con 0x7fe1381024b0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.199+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1300042a0 con 0x7fe1381024b0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.199+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1381a0030 con 0x7fe1381024b0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.199+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1381a0570 con 0x7fe1381024b0 2024-04-21T08:44:46.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.200+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1381a0890 con 0x7fe1381024b0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.201+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe1300325c0 con 0x7fe1381024b0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.202+0000 7fe135ffb640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe1280a3c30 0x7fe1280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.202+0000 7fe13cd49640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe1280a3c30 0x7fe1280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.202+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fe1300f2a80 con 0x7fe1381024b0 2024-04-21T08:44:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.202+0000 7fe13cd49640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe1280a3c30 0x7fe1280a60f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005d00 tx=0x7fe12c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.206+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1300f4e20 con 0x7fe1381024b0 2024-04-21T08:44:46.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:46 smithi057 ceph-mon[27968]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:46 smithi057 ceph-mon[27968]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:44:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-04-21T08:44:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:46 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-21T08:44:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:46 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2622345400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:46 smithi136 ceph-mon[31791]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:46 smithi136 ceph-mon[31791]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:44:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-04-21T08:44:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:46 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-21T08:44:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:46 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2622345400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.397+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1381a0a60 con 0x7fe1381024b0 2024-04-21T08:44:46.396 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.397+0000 7fe135ffb640 1 -- 172.21.15.57:0/3412290292 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7fe1300bf0c0 con 0x7fe1381024b0 2024-04-21T08:44:46.397 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:44:46.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe1280a3c30 msgr2=0x7fe1280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe1280a3c30 0x7fe1280a60f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005d00 tx=0x7fe12c005ed0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 msgr2=0x7fe13819f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 0x7fe13819f730 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe130032520 tx=0x7fe1300347b0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 shutdown_connections 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe1280a3c30 0x7fe1280a60f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1381024b0 0x7fe13819f730 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe138101ac0 0x7fe1381986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 --2- 172.21.15.57:0/3412290292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe138101100 0x7fe138198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 >> 172.21.15.57:0/3412290292 conn(0x7fe1380fbb90 msgr2=0x7fe1380fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 shutdown_connections 2024-04-21T08:44:46.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.400+0000 7fe13efad640 1 -- 172.21.15.57:0/3412290292 wait complete. 2024-04-21T08:44:46.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1514569434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8100620 msgr2=0x7f32b8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1514569434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8100620 0x7f32b8100a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009900 tx=0x7f32b002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1514569434 shutdown_connections 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1514569434 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b81021d0 0x7f32b8108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1514569434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b8101810 0x7f32b8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1514569434 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8100620 0x7f32b8100a40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1514569434 >> 172.21.15.57:0/1514569434 conn(0x7f32b80fbb90 msgr2=0x7f32b80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1514569434 shutdown_connections 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.516+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1514569434 wait complete. 2024-04-21T08:44:46.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 Processor -- start 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 -- start start 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b8100620 0x7f32b8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 0x7f32b8198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b81021d0 0x7f32b819f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f32b810caf0 con 0x7f32b8101810 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32b810c970 con 0x7f32b8100620 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bfbb6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f32b810cc70 con 0x7f32b81021d0 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bd151640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 0x7f32b8198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bd151640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 0x7f32b8198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33440/0 (socket says 172.21.15.57:33440) 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bd151640 1 -- 172.21.15.57:0/1306241059 learned_addr learned my addr 172.21.15.57:0/1306241059 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32bd952640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b8100620 0x7f32b8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.517+0000 7f32be153640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b81021d0 0x7f32b819f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd151640 1 -- 172.21.15.57:0/1306241059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b81021d0 msgr2=0x7f32b819f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd151640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b81021d0 0x7f32b819f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd151640 1 -- 172.21.15.57:0/1306241059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b8100620 msgr2=0x7f32b8198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd151640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b8100620 0x7f32b8198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd151640 1 -- 172.21.15.57:0/1306241059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b819fff0 con 0x7f32b8101810 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd952640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b8100620 0x7f32b8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bd151640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 0x7f32b8198960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00b4a0 tx=0x7f32ac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac014070 con 0x7f32b8101810 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32ac010ac0 con 0x7f32b8101810 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b81a02e0 con 0x7f32b8101810 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac017c30 con 0x7f32b8101810 2024-04-21T08:44:46.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.518+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b81a08b0 con 0x7f32b8101810 2024-04-21T08:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.519+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b81a0b70 con 0x7f32b8101810 2024-04-21T08:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.520+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f32ac017070 con 0x7f32b8101810 2024-04-21T08:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.521+0000 7f32b6ffd640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f32a00a3c30 0x7f32a00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.521+0000 7f32bd952640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f32a00a3c30 0x7f32a00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.521+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f32ac0cfbe0 con 0x7f32b8101810 2024-04-21T08:44:46.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.521+0000 7f32bd952640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f32a00a3c30 0x7f32a00a60f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f32b002c240 tx=0x7f32b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.525+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f32ac09c1e0 con 0x7f32b8101810 2024-04-21T08:44:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:46 smithi129 ceph-mon[31272]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:44:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:46 smithi129 ceph-mon[31272]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-21T08:44:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]: dispatch 2024-04-21T08:44:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:46 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-04-21T08:44:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:46 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2622345400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:46.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.711+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f32b81a0d40 con 0x7f32b8101810 2024-04-21T08:44:46.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.712+0000 7f32b6ffd640 1 -- 172.21.15.57:0/1306241059 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f32ac09bae0 con 0x7f32b8101810 2024-04-21T08:44:46.711 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:44:46.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f32a00a3c30 msgr2=0x7f32a00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f32a00a3c30 0x7f32a00a60f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f32b002c240 tx=0x7f32b0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 msgr2=0x7f32b8198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 0x7f32b8198960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00b4a0 tx=0x7f32ac00b960 comp rx=0 tx=0).stop 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 shutdown_connections 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f32a00a3c30 0x7f32a00a60f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32b81021d0 0x7f32b819f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32b8101810 0x7f32b8198960 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 --2- 172.21.15.57:0/1306241059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32b8100620 0x7f32b8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 >> 172.21.15.57:0/1306241059 conn(0x7f32b80fbb90 msgr2=0x7f32b8109510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 shutdown_connections 2024-04-21T08:44:46.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.714+0000 7f32bfbb6640 1 -- 172.21.15.57:0/1306241059 wait complete. 2024-04-21T08:44:46.725 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 -- 172.21.15.57:0/3598643218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4102040 msgr2=0x7f8da41024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 --2- 172.21.15.57:0/3598643218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4102040 0x7f8da41024c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009900 tx=0x7f8d9002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 -- 172.21.15.57:0/3598643218 shutdown_connections 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 --2- 172.21.15.57:0/3598643218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8da4102a00 0x7f8da4109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 --2- 172.21.15.57:0/3598643218 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4102040 0x7f8da41024c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 --2- 172.21.15.57:0/3598643218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da4100e60 0x7f8da4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 -- 172.21.15.57:0/3598643218 >> 172.21.15.57:0/3598643218 conn(0x7f8da40fc440 msgr2=0x7f8da40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 -- 172.21.15.57:0/3598643218 shutdown_connections 2024-04-21T08:44:46.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.830+0000 7f8daab75640 1 -- 172.21.15.57:0/3598643218 wait complete. 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 Processor -- start 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 -- start start 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 0x7f8da4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da4102040 0x7f8da41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8da4102a00 0x7f8da419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410c9f0 con 0x7f8da4100e60 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410c870 con 0x7f8da4102040 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8daab75640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410cb70 con 0x7f8da4102a00 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 0x7f8da4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 0x7f8da4198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33454/0 (socket says 172.21.15.57:33454) 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 -- 172.21.15.57:0/3486196376 learned_addr learned my addr 172.21.15.57:0/3486196376 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da3fff640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da4102040 0x7f8da41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da9112640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8da4102a00 0x7f8da419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 -- 172.21.15.57:0/3486196376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8da4102a00 msgr2=0x7f8da419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8da4102a00 0x7f8da419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 -- 172.21.15.57:0/3486196376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da4102040 msgr2=0x7f8da41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da4102040 0x7f8da41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.831+0000 7f8da8911640 1 -- 172.21.15.57:0/3486196376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da419fda0 con 0x7f8da4100e60 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.832+0000 7f8da8911640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 0x7f8da4198180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c970 tx=0x7f8d9800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.832+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98007c10 con 0x7f8da4100e60 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.832+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d98007db0 con 0x7f8da4100e60 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.832+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da41a0090 con 0x7f8da4100e60 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.832+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800f870 con 0x7f8da4100e60 2024-04-21T08:44:46.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.832+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da41a05e0 con 0x7f8da4100e60 2024-04-21T08:44:46.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.833+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da41a08a0 con 0x7f8da4100e60 2024-04-21T08:44:46.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.834+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8d9800fa10 con 0x7f8da4100e60 2024-04-21T08:44:46.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.834+0000 7f8da1ffb640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d940a3bb0 0x7f8d940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:46.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.835+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8d980d0bc0 con 0x7f8da4100e60 2024-04-21T08:44:46.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.835+0000 7f8da3fff640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d940a3bb0 0x7f8d940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:46.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.835+0000 7f8da3fff640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d940a3bb0 0x7f8d940a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8da4199690 tx=0x7f8d90005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:46.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:46.839+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d9809d2b0 con 0x7f8da4100e60 2024-04-21T08:44:47.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.026+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8da41a0a70 con 0x7f8da4100e60 2024-04-21T08:44:47.026 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.026+0000 7f8da1ffb640 1 -- 172.21.15.57:0/3486196376 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f8d9809cbb0 con 0x7f8da4100e60 2024-04-21T08:44:47.026 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:44:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d940a3bb0 msgr2=0x7f8d940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d940a3bb0 0x7f8d940a6070 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8da4199690 tx=0x7f8d90005be0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 msgr2=0x7f8da4198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 0x7f8da4198180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c970 tx=0x7f8d9800ce30 comp rx=0 tx=0).stop 2024-04-21T08:44:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 shutdown_connections 2024-04-21T08:44:47.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8d940a3bb0 0x7f8d940a6070 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8da4102a00 0x7f8da419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da4102040 0x7f8da41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 --2- 172.21.15.57:0/3486196376 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8da4100e60 0x7f8da4198180 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 >> 172.21.15.57:0/3486196376 conn(0x7f8da40fc440 msgr2=0x7f8da40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 shutdown_connections 2024-04-21T08:44:47.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.028+0000 7f8daab75640 1 -- 172.21.15.57:0/3486196376 wait complete. 2024-04-21T08:44:47.039 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.144+0000 7f5641ea7640 1 -- 172.21.15.57:0/3029054876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c102a30 msgr2=0x7f563c104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.144+0000 7f5641ea7640 1 --2- 172.21.15.57:0/3029054876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c102a30 0x7f563c104e20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f562c009900 tx=0x7f562c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- 172.21.15.57:0/3029054876 shutdown_connections 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 --2- 172.21.15.57:0/3029054876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f563c107c90 0x7f563c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 --2- 172.21.15.57:0/3029054876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f563c105360 0x7f563c107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 --2- 172.21.15.57:0/3029054876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c102a30 0x7f563c104e20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- 172.21.15.57:0/3029054876 >> 172.21.15.57:0/3029054876 conn(0x7f563c0fc440 msgr2=0x7f563c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- 172.21.15.57:0/3029054876 shutdown_connections 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- 172.21.15.57:0/3029054876 wait complete. 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 Processor -- start 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- start start 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f563c102a30 0x7f563c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f563c105360 0x7f563c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 0x7f563c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f563c10daf0 con 0x7f563c107c90 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f563c10d970 con 0x7f563c102a30 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.145+0000 7f5641ea7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f563c10dc70 con 0x7f563c105360 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 0x7f563c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 0x7f563c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33476/0 (socket says 172.21.15.57:33476) 2024-04-21T08:44:47.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 -- 172.21.15.57:0/4183173402 learned_addr learned my addr 172.21.15.57:0/4183173402 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563b7fe640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f563c102a30 0x7f563c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563affd640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f563c105360 0x7f563c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 -- 172.21.15.57:0/4183173402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f563c105360 msgr2=0x7f563c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f563c105360 0x7f563c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 -- 172.21.15.57:0/4183173402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f563c102a30 msgr2=0x7f563c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f563c102a30 0x7f563c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 -- 172.21.15.57:0/4183173402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c19fe20 con 0x7f563c107c90 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f563bfff640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 0x7f563c19f7b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f563400eaa0 tx=0x7f563400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563400cdb0 con 0x7f563c107c90 2024-04-21T08:44:47.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5634005020 con 0x7f563c107c90 2024-04-21T08:44:47.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56340116d0 con 0x7f563c107c90 2024-04-21T08:44:47.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c1a0110 con 0x7f563c107c90 2024-04-21T08:44:47.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.146+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c1a0660 con 0x7f563c107c90 2024-04-21T08:44:47.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.147+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563c0fdf30 con 0x7f563c107c90 2024-04-21T08:44:47.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.149+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f56340051d0 con 0x7f563c107c90 2024-04-21T08:44:47.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.149+0000 7f5638ff9640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f56240a3bb0 0x7f56240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.149+0000 7f563b7fe640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f56240a3bb0 0x7f56240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.149+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f56340d0bd0 con 0x7f563c107c90 2024-04-21T08:44:47.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.150+0000 7f563b7fe640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f56240a3bb0 0x7f56240a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f562c009230 tx=0x7f562c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.153+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5634016030 con 0x7f563c107c90 2024-04-21T08:44:47.310 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-04-21T08:44:47.310 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:47 smithi057 ceph-mon[27968]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-21T08:44:47.310 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:47 smithi057 ceph-mon[27968]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:44:47.310 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:47 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3412290292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.310 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:47 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1306241059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.310 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:47 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3486196376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.343 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.344+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f563c02d5c0 con 0x7f563c107c90 2024-04-21T08:44:47.344 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.344+0000 7f5638ff9640 1 -- 172.21.15.57:0/4183173402 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f563409cbe0 con 0x7f563c107c90 2024-04-21T08:44:47.344 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:44:47.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f56240a3bb0 msgr2=0x7f56240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.346 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f56240a3bb0 0x7f56240a6070 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f562c009230 tx=0x7f562c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 msgr2=0x7f563c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 0x7f563c19f7b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f563400eaa0 tx=0x7f563400ef60 comp rx=0 tx=0).stop 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 shutdown_connections 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f56240a3bb0 0x7f56240a6070 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f563c107c90 0x7f563c19f7b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f563c105360 0x7f563c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 --2- 172.21.15.57:0/4183173402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f563c102a30 0x7f563c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 >> 172.21.15.57:0/4183173402 conn(0x7f563c0fc440 msgr2=0x7f563c103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 shutdown_connections 2024-04-21T08:44:47.347 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.347+0000 7f5641ea7640 1 -- 172.21.15.57:0/4183173402 wait complete. 2024-04-21T08:44:47.357 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 -- 172.21.15.57:0/2053939476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78107c50 msgr2=0x7fae7810a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2053939476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78107c50 0x7fae7810a0a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fae70009ca0 tx=0x7fae7002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 -- 172.21.15.57:0/2053939476 shutdown_connections 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2053939476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78107c50 0x7fae7810a0a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2053939476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae78105320 0x7fae78107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2053939476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae781029f0 0x7fae78104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 -- 172.21.15.57:0/2053939476 >> 172.21.15.57:0/2053939476 conn(0x7fae780fc3e0 msgr2=0x7fae780fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 -- 172.21.15.57:0/2053939476 shutdown_connections 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.460+0000 7fae7cca8640 1 -- 172.21.15.57:0/2053939476 wait complete. 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 Processor -- start 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 -- start start 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae781029f0 0x7fae78102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 0x7fae781009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae78107c50 0x7fae78100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fae7810dac0 con 0x7fae78105320 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae7810d940 con 0x7fae78107c50 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7cca8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fae7810dc40 con 0x7fae781029f0 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae75d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 0x7fae781009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae75d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 0x7fae781009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33490/0 (socket says 172.21.15.57:33490) 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae75d9b640 1 -- 172.21.15.57:0/2668979568 learned_addr learned my addr 172.21.15.57:0/2668979568 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:47.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.461+0000 7fae7659c640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae781029f0 0x7fae78102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae76d9d640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae78107c50 0x7fae78100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae75d9b640 1 -- 172.21.15.57:0/2668979568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae781029f0 msgr2=0x7fae78102360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae75d9b640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae781029f0 0x7fae78102360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae75d9b640 1 -- 172.21.15.57:0/2668979568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae78107c50 msgr2=0x7fae78100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae75d9b640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae78107c50 0x7fae78100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae75d9b640 1 -- 172.21.15.57:0/2668979568 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae78101770 con 0x7fae78105320 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae75d9b640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 0x7fae781009a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005fd0 tx=0x7fae6c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c017070 con 0x7fae78105320 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae781a2290 con 0x7fae78105320 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae6c009c30 con 0x7fae78105320 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c003d30 con 0x7fae78105320 2024-04-21T08:44:47.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.462+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae781a27e0 con 0x7fae78105320 2024-04-21T08:44:47.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.463+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae780fded0 con 0x7fae78105320 2024-04-21T08:44:47.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.464+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fae6c003ed0 con 0x7fae78105320 2024-04-21T08:44:47.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.465+0000 7fae677fe640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fae600a3bb0 0x7fae600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.465+0000 7fae7659c640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fae600a3bb0 0x7fae600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.465+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fae6c0d0bf0 con 0x7fae78105320 2024-04-21T08:44:47.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.465+0000 7fae7659c640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fae600a3bb0 0x7fae600a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae680058c0 tx=0x7fae6800a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.469+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae6c020030 con 0x7fae78105320 2024-04-21T08:44:47.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-04-21T08:44:47.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:47 smithi129 ceph-mon[31272]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-21T08:44:47.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:47 smithi129 ceph-mon[31272]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:44:47.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:47 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3412290292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:47 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1306241059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:47 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3486196376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 6]}]': finished 2024-04-21T08:44:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:47 smithi136 ceph-mon[31791]: from='mgr.14152 172.21.15.57:0/2936021575' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-04-21T08:44:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:47 smithi136 ceph-mon[31791]: osdmap e92: 12 total, 12 up, 12 in 2024-04-21T08:44:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:47 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3412290292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:47 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1306241059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:47 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3486196376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:47.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.695+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae7802d810 con 0x7fae78105320 2024-04-21T08:44:47.695 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.696+0000 7fae677fe640 1 -- 172.21.15.57:0/2668979568 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fae6c009de0 con 0x7fae78105320 2024-04-21T08:44:47.695 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:47.696 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:44:47.696 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fae600a3bb0 msgr2=0x7fae600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fae600a3bb0 0x7fae600a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae680058c0 tx=0x7fae6800a250 comp rx=0 tx=0).stop 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 msgr2=0x7fae781009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 0x7fae781009a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fae6c005fd0 tx=0x7fae6c00dde0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 shutdown_connections 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fae600a3bb0 0x7fae600a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae78107c50 0x7fae78100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae78105320 0x7fae781009a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 --2- 172.21.15.57:0/2668979568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae781029f0 0x7fae78102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 >> 172.21.15.57:0/2668979568 conn(0x7fae780fc3e0 msgr2=0x7fae78103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 shutdown_connections 2024-04-21T08:44:47.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.698+0000 7fae7cca8640 1 -- 172.21.15.57:0/2668979568 wait complete. 2024-04-21T08:44:47.709 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 -- 172.21.15.57:0/535121998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c102a20 msgr2=0x7fa01c104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 --2- 172.21.15.57:0/535121998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c102a20 0x7fa01c104e10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa00c009900 tx=0x7fa00c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 -- 172.21.15.57:0/535121998 shutdown_connections 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 --2- 172.21.15.57:0/535121998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c107c80 0x7fa01c10a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 --2- 172.21.15.57:0/535121998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa01c105350 0x7fa01c107740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 --2- 172.21.15.57:0/535121998 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c102a20 0x7fa01c104e10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 -- 172.21.15.57:0/535121998 >> 172.21.15.57:0/535121998 conn(0x7fa01c0fc410 msgr2=0x7fa01c0fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 -- 172.21.15.57:0/535121998 shutdown_connections 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 -- 172.21.15.57:0/535121998 wait complete. 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 Processor -- start 2024-04-21T08:44:47.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.811+0000 7fa022546640 1 -- start start 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa022546640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa01c102a20 0x7fa01c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa022546640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c105350 0x7fa01c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa022546640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 0x7fa01c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa022546640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c10dae0 con 0x7fa01c107c80 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa022546640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c10d960 con 0x7fa01c102a20 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa022546640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c10dc60 con 0x7fa01c105350 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 0x7fa01c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 0x7fa01c19f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33514/0 (socket says 172.21.15.57:33514) 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 -- 172.21.15.57:0/2531470898 learned_addr learned my addr 172.21.15.57:0/2531470898 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa01b7fe640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c105350 0x7fa01c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa01bfff640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa01c102a20 0x7fa01c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 -- 172.21.15.57:0/2531470898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c105350 msgr2=0x7fa01c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c105350 0x7fa01c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 -- 172.21.15.57:0/2531470898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa01c102a20 msgr2=0x7fa01c1981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa01c102a20 0x7fa01c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 -- 172.21.15.57:0/2531470898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c19fdd0 con 0x7fa01c107c80 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa01b7fe640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c105350 0x7fa01c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.812+0000 7fa020ae3640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 0x7fa01c19f760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa01400eaa0 tx=0x7fa01400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.813+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01400cdb0 con 0x7fa01c107c80 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.813+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa014005020 con 0x7fa01c107c80 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.813+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01c1a00c0 con 0x7fa01c107c80 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.813+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0140116d0 con 0x7fa01c107c80 2024-04-21T08:44:47.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.813+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01c1a0610 con 0x7fa01c107c80 2024-04-21T08:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.814+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa01c0fdf00 con 0x7fa01c107c80 2024-04-21T08:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.815+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa0140051d0 con 0x7fa01c107c80 2024-04-21T08:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.816+0000 7fa0197fa640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0040a3bb0 0x7fa0040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.816+0000 7fa01bfff640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0040a3bb0 0x7fa0040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:47.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.816+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa0140d09f0 con 0x7fa01c107c80 2024-04-21T08:44:47.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.816+0000 7fa01bfff640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0040a3bb0 0x7fa0040a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0096c0 tx=0x7fa00c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:47.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:47.820+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa014016030 con 0x7fa01c107c80 2024-04-21T08:44:48.009 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.010+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa01c062b50 con 0x7fa01c107c80 2024-04-21T08:44:48.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.012+0000 7fa0197fa640 1 -- 172.21.15.57:0/2531470898 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7fa01409d050 con 0x7fa01c107c80 2024-04-21T08:44:48.012 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:48.012 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0040a3bb0 msgr2=0x7fa0040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0040a3bb0 0x7fa0040a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0096c0 tx=0x7fa00c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 msgr2=0x7fa01c19f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 0x7fa01c19f760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa01400eaa0 tx=0x7fa01400ef60 comp rx=0 tx=0).stop 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 shutdown_connections 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0040a3bb0 0x7fa0040a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c107c80 0x7fa01c19f760 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa01c105350 0x7fa01c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 --2- 172.21.15.57:0/2531470898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa01c102a20 0x7fa01c1981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 >> 172.21.15.57:0/2531470898 conn(0x7fa01c0fc410 msgr2=0x7fa01c103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 shutdown_connections 2024-04-21T08:44:48.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.014+0000 7fa022546640 1 -- 172.21.15.57:0/2531470898 wait complete. 2024-04-21T08:44:48.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:44:48.026 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.131+0000 7fd060584640 1 -- 172.21.15.57:0/988136040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058103b50 msgr2=0x7fd05810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.131+0000 7fd060584640 1 --2- 172.21.15.57:0/988136040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058103b50 0x7fd05810a3e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd050009ca0 tx=0x7fd05002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.131+0000 7fd060584640 1 -- 172.21.15.57:0/988136040 shutdown_connections 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.131+0000 7fd060584640 1 --2- 172.21.15.57:0/988136040 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058103b50 0x7fd05810a3e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.131+0000 7fd060584640 1 --2- 172.21.15.57:0/988136040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058103190 0x7fd058103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.131+0000 7fd060584640 1 --2- 172.21.15.57:0/988136040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058101fa0 0x7fd0581023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- 172.21.15.57:0/988136040 >> 172.21.15.57:0/988136040 conn(0x7fd0580fd530 msgr2=0x7fd0580ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- 172.21.15.57:0/988136040 shutdown_connections 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- 172.21.15.57:0/988136040 wait complete. 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 Processor -- start 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- start start 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 0x7fd058198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058103190 0x7fd0581986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058103b50 0x7fd05819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd05810dae0 con 0x7fd058101fa0 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd05810d960 con 0x7fd058103b50 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd060584640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd05810dc60 con 0x7fd058103190 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd05e320640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 0x7fd058198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd05e320640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 0x7fd058198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33532/0 (socket says 172.21.15.57:33532) 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.132+0000 7fd05e320640 1 -- 172.21.15.57:0/2263136506 learned_addr learned my addr 172.21.15.57:0/2263136506 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:48.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05eb21640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058103b50 0x7fd05819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05db1f640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058103190 0x7fd0581986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05e320640 1 -- 172.21.15.57:0/2263136506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058103190 msgr2=0x7fd0581986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05e320640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058103190 0x7fd0581986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05e320640 1 -- 172.21.15.57:0/2263136506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058103b50 msgr2=0x7fd05819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05e320640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058103b50 0x7fd05819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05e320640 1 -- 172.21.15.57:0/2263136506 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05819fd80 con 0x7fd058101fa0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05eb21640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058103b50 0x7fd05819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd05e320640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 0x7fd058198160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd05400d770 tx=0x7fd05400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054017070 con 0x7fd058101fa0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd054009c30 con 0x7fd058101fa0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0581a0070 con 0x7fd058101fa0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054003cc0 con 0x7fd058101fa0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.133+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581a05c0 con 0x7fd058101fa0 2024-04-21T08:44:48.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.134+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd058109340 con 0x7fd058101fa0 2024-04-21T08:44:48.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.135+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd054003e60 con 0x7fd058101fa0 2024-04-21T08:44:48.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.136+0000 7fd04f7fe640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd0440a3bb0 0x7fd0440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.136+0000 7fd05db1f640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd0440a3bb0 0x7fd0440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.136+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fd0540d0220 con 0x7fd058101fa0 2024-04-21T08:44:48.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.137+0000 7fd05db1f640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd0440a3bb0 0x7fd0440a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd048005fd0 tx=0x7fd048005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.140 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.141+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd05409c820 con 0x7fd058101fa0 2024-04-21T08:44:48.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.327+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd058062b80 con 0x7fd058101fa0 2024-04-21T08:44:48.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.328+0000 7fd04f7fe640 1 -- 172.21.15.57:0/2263136506 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd054009de0 con 0x7fd058101fa0 2024-04-21T08:44:48.327 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:44:48.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd0440a3bb0 msgr2=0x7fd0440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd0440a3bb0 0x7fd0440a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd048005fd0 tx=0x7fd048005e20 comp rx=0 tx=0).stop 2024-04-21T08:44:48.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 msgr2=0x7fd058198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 0x7fd058198160 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd05400d770 tx=0x7fd05400dc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 shutdown_connections 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd0440a3bb0 0x7fd0440a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd058103b50 0x7fd05819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058103190 0x7fd0581986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 --2- 172.21.15.57:0/2263136506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd058101fa0 0x7fd058198160 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 >> 172.21.15.57:0/2263136506 conn(0x7fd0580fd530 msgr2=0x7fd058106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 shutdown_connections 2024-04-21T08:44:48.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.330+0000 7fd060584640 1 -- 172.21.15.57:0/2263136506 wait complete. 2024-04-21T08:44:48.340 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:44:48.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2837615863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 msgr2=0x7f0f60100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2837615863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f60100c40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009900 tx=0x7f0f5002bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2837615863 shutdown_connections 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2837615863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 0x7f0f60101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2837615863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f60101180 0x7f0f601015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2837615863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f60100c40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2837615863 >> 172.21.15.57:0/2837615863 conn(0x7f0f600fc440 msgr2=0x7f0f600fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2837615863 shutdown_connections 2024-04-21T08:44:48.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.446+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2837615863 wait complete. 2024-04-21T08:44:48.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 Processor -- start 2024-04-21T08:44:48.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 -- start start 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f6019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f60101180 0x7f0f6019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 0x7f0f601a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6010fd40 con 0x7f0f60100820 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6010fbc0 con 0x7f0f60101b20 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f64fb6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6010fec0 con 0x7f0f60101180 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f6019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f6019a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33550/0 (socket says 172.21.15.57:33550) 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 -- 172.21.15.57:0/2397007774 learned_addr learned my addr 172.21.15.57:0/2397007774 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5ed9d640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 0x7f0f601a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5dd9b640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f60101180 0x7f0f6019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 -- 172.21.15.57:0/2397007774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f60101180 msgr2=0x7f0f6019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f60101180 0x7f0f6019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 -- 172.21.15.57:0/2397007774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 msgr2=0x7f0f601a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 0x7f0f601a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 -- 172.21.15.57:0/2397007774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f601a2070 con 0x7f0f60100820 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5ed9d640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 0x7f0f601a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.447+0000 7f0f5e59c640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f6019a450 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0f50036040 tx=0x7f0f500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.448+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50043020 con 0x7f0f60100820 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.448+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f500054c0 con 0x7f0f60100820 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.448+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f601a2300 con 0x7f0f60100820 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.448+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50003db0 con 0x7f0f60100820 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.448+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f601a2920 con 0x7f0f60100820 2024-04-21T08:44:48.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.449+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f6010b570 con 0x7f0f60100820 2024-04-21T08:44:48.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.450+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0f50046020 con 0x7f0f60100820 2024-04-21T08:44:48.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.450+0000 7f0f4f7fe640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0f480a3c30 0x7f0f480a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.451+0000 7f0f5dd9b640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0f480a3c30 0x7f0f480a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.451+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f0f500f2c50 con 0x7f0f60100820 2024-04-21T08:44:48.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.451+0000 7f0f5dd9b640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0f480a3c30 0x7f0f480a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0f54005fd0 tx=0x7f0f54005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.455+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f500f8020 con 0x7f0f60100820 2024-04-21T08:44:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:48 smithi129 ceph-mon[31272]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T08:44:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:48 smithi129 ceph-mon[31272]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:44:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:48 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4183173402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:48 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2668979568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:48 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2531470898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:48.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:48 smithi057 ceph-mon[27968]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T08:44:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:48 smithi057 ceph-mon[27968]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:44:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:48 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4183173402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:48 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2668979568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:48 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2531470898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:48.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:48 smithi136 ceph-mon[31791]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-04-21T08:44:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:48 smithi136 ceph-mon[31791]: osdmap e93: 12 total, 12 up, 12 in 2024-04-21T08:44:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:48 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4183173402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:48 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2668979568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:48 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2531470898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:48.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.644+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f60062b50 con 0x7f0f60100820 2024-04-21T08:44:48.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.645+0000 7f0f4f7fe640 1 -- 172.21.15.57:0/2397007774 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f0f5004d020 con 0x7f0f60100820 2024-04-21T08:44:48.644 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0f480a3c30 msgr2=0x7f0f480a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0f480a3c30 0x7f0f480a60f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0f54005fd0 tx=0x7f0f54005e20 comp rx=0 tx=0).stop 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 msgr2=0x7f0f6019a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f6019a450 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f0f50036040 tx=0x7f0f500042f0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 shutdown_connections 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0f480a3c30 0x7f0f480a60f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f60101b20 0x7f0f601a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f60101180 0x7f0f6019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 --2- 172.21.15.57:0/2397007774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f60100820 0x7f0f6019a450 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 >> 172.21.15.57:0/2397007774 conn(0x7f0f600fc440 msgr2=0x7f0f600fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 shutdown_connections 2024-04-21T08:44:48.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.647+0000 7f0f64fb6640 1 -- 172.21.15.57:0/2397007774 wait complete. 2024-04-21T08:44:48.658 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:44:48.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 -- 172.21.15.57:0/1517690822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284103190 msgr2=0x7f9284103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 --2- 172.21.15.57:0/1517690822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284103190 0x7f9284103610 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9274009900 tx=0x7f927402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:48.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 -- 172.21.15.57:0/1517690822 shutdown_connections 2024-04-21T08:44:48.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 --2- 172.21.15.57:0/1517690822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 0x7f928410a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 --2- 172.21.15.57:0/1517690822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284103190 0x7f9284103610 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 --2- 172.21.15.57:0/1517690822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9284101fa0 0x7f92841023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.763+0000 7f928b6e4640 1 -- 172.21.15.57:0/1517690822 >> 172.21.15.57:0/1517690822 conn(0x7f92840fd530 msgr2=0x7f92840ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 -- 172.21.15.57:0/1517690822 shutdown_connections 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 -- 172.21.15.57:0/1517690822 wait complete. 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 Processor -- start 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 -- start start 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 0x7f9284075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9284103190 0x7f9284074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 0x7f9284074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f928410dae0 con 0x7f9284101fa0 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f928410d960 con 0x7f9284103190 2024-04-21T08:44:48.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f928b6e4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f928410dc60 con 0x7f9284103b50 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f9289480640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 0x7f9284075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f9289480640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 0x7f9284075d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33564/0 (socket says 172.21.15.57:33564) 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.764+0000 7f9289480640 1 -- 172.21.15.57:0/2199029564 learned_addr learned my addr 172.21.15.57:0/2199029564 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9288c7f640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9284103190 0x7f9284074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289c81640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 0x7f9284074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289480640 1 -- 172.21.15.57:0/2199029564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 msgr2=0x7f9284074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289480640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 0x7f9284074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289480640 1 -- 172.21.15.57:0/2199029564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9284103190 msgr2=0x7f9284074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289480640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9284103190 0x7f9284074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289480640 1 -- 172.21.15.57:0/2199029564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9284075110 con 0x7f9284101fa0 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289c81640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 0x7f9284074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:48.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f9289480640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 0x7f9284075d00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f928000d950 tx=0x7f928000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9280017020 con 0x7f9284101fa0 2024-04-21T08:44:48.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9280009c30 con 0x7f9284101fa0 2024-04-21T08:44:48.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92841a22e0 con 0x7f9284101fa0 2024-04-21T08:44:48.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928000f6b0 con 0x7f9284101fa0 2024-04-21T08:44:48.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.765+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92841a2830 con 0x7f9284101fa0 2024-04-21T08:44:48.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.766+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9284109390 con 0x7f9284101fa0 2024-04-21T08:44:48.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.767+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f928001a020 con 0x7f9284101fa0 2024-04-21T08:44:48.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.768+0000 7f927a7fc640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f92700a3bb0 0x7f92700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:48.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.768+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f92800d0550 con 0x7f9284101fa0 2024-04-21T08:44:48.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.768+0000 7f9288c7f640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f92700a3bb0 0x7f92700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:48.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.768+0000 7f9288c7f640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f92700a3bb0 0x7f92700a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9284075a70 tx=0x7f9274005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:48.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.772+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9280009de0 con 0x7f9284101fa0 2024-04-21T08:44:48.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.958+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9284062b50 con 0x7f9284101fa0 2024-04-21T08:44:48.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.959+0000 7f927a7fc640 1 -- 172.21.15.57:0/2199029564 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f928009cbb0 con 0x7f9284101fa0 2024-04-21T08:44:48.958 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:44:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f92700a3bb0 msgr2=0x7f92700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f92700a3bb0 0x7f92700a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9284075a70 tx=0x7f9274005be0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 msgr2=0x7f9284075d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 0x7f9284075d00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f928000d950 tx=0x7f928000de10 comp rx=0 tx=0).stop 2024-04-21T08:44:48.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 shutdown_connections 2024-04-21T08:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f92700a3bb0 0x7f92700a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9284103b50 0x7f9284074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9284103190 0x7f9284074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 --2- 172.21.15.57:0/2199029564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9284101fa0 0x7f9284075d00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 >> 172.21.15.57:0/2199029564 conn(0x7f92840fd530 msgr2=0x7f9284106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:48.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 shutdown_connections 2024-04-21T08:44:48.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:48.961+0000 7f928b6e4640 1 -- 172.21.15.57:0/2199029564 wait complete. 2024-04-21T08:44:48.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.077+0000 7f1c865ca640 1 -- 172.21.15.57:0/964423930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 msgr2=0x7f1c8010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 --2- 172.21.15.57:0/964423930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c8010a0c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009ca0 tx=0x7f1c7802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 -- 172.21.15.57:0/964423930 shutdown_connections 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 --2- 172.21.15.57:0/964423930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c8010a0c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 --2- 172.21.15.57:0/964423930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80105340 0x7f1c80107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 --2- 172.21.15.57:0/964423930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80102a10 0x7f1c80104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 -- 172.21.15.57:0/964423930 >> 172.21.15.57:0/964423930 conn(0x7f1c800fc400 msgr2=0x7f1c800fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 -- 172.21.15.57:0/964423930 shutdown_connections 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 -- 172.21.15.57:0/964423930 wait complete. 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 Processor -- start 2024-04-21T08:44:49.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.078+0000 7f1c865ca640 1 -- start start 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80102a10 0x7f1c80102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80105340 0x7f1c801009c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c80100f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010dad0 con 0x7f1c80107c70 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010d950 con 0x7f1c80105340 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8010dc50 con 0x7f1c80102a10 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c80100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c80100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33582/0 (socket says 172.21.15.57:33582) 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 -- 172.21.15.57:0/4196855606 learned_addr learned my addr 172.21.15.57:0/4196855606 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7ffff640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80102a10 0x7f1c80102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7f7fe640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80105340 0x7f1c801009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 -- 172.21.15.57:0/4196855606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80102a10 msgr2=0x7f1c80102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80102a10 0x7f1c80102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 -- 172.21.15.57:0/4196855606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80105340 msgr2=0x7f1c801009c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80105340 0x7f1c801009c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 -- 172.21.15.57:0/4196855606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c80101790 con 0x7f1c80107c70 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7ffff640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80102a10 0x7f1c80102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7f7fe640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80105340 0x7f1c801009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c84b67640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c80100f00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1c78032520 tx=0x7f1c780347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78039040 con 0x7f1c80107c70 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c78044070 con 0x7f1c80107c70 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c801a2290 con 0x7f1c80107c70 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.079+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c780042a0 con 0x7f1c80107c70 2024-04-21T08:44:49.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.080+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c801a27e0 con 0x7f1c80107c70 2024-04-21T08:44:49.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.081+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c800fdef0 con 0x7f1c80107c70 2024-04-21T08:44:49.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.082+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1c780325c0 con 0x7f1c80107c70 2024-04-21T08:44:49.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.082+0000 7f1c7d7fa640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c680a3bb0 0x7f1c680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.082+0000 7f1c7ffff640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c680a3bb0 0x7f1c680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.082+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f1c780f3230 con 0x7f1c80107c70 2024-04-21T08:44:49.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.083+0000 7f1c7ffff640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c680a3bb0 0x7f1c680a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009b40 tx=0x7f1c74005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.087+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c780bf890 con 0x7f1c80107c70 2024-04-21T08:44:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.273+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1c8002d5c0 con 0x7f1c80107c70 2024-04-21T08:44:49.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.274+0000 7f1c7d7fa640 1 -- 172.21.15.57:0/4196855606 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f1c780bf190 con 0x7f1c80107c70 2024-04-21T08:44:49.273 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:44:49.275 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c680a3bb0 msgr2=0x7f1c680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c680a3bb0 0x7f1c680a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009b40 tx=0x7f1c74005e50 comp rx=0 tx=0).stop 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 msgr2=0x7f1c80100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c80100f00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1c78032520 tx=0x7f1c780347b0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 shutdown_connections 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c680a3bb0 0x7f1c680a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c80107c70 0x7f1c80100f00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c80105340 0x7f1c801009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 --2- 172.21.15.57:0/4196855606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c80102a10 0x7f1c80102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 >> 172.21.15.57:0/4196855606 conn(0x7f1c800fc400 msgr2=0x7f1c801033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 shutdown_connections 2024-04-21T08:44:49.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.276+0000 7f1c865ca640 1 -- 172.21.15.57:0/4196855606 wait complete. 2024-04-21T08:44:49.286 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 -- 172.21.15.57:0/3516023273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102050 msgr2=0x7fe3581024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 --2- 172.21.15.57:0/3516023273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102050 0x7fe3581024d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe344009900 tx=0x7fe34402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 -- 172.21.15.57:0/3516023273 shutdown_connections 2024-04-21T08:44:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 --2- 172.21.15.57:0/3516023273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe358102a10 0x7fe358109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 --2- 172.21.15.57:0/3516023273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102050 0x7fe3581024d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 --2- 172.21.15.57:0/3516023273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358100e70 0x7fe358101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 -- 172.21.15.57:0/3516023273 >> 172.21.15.57:0/3516023273 conn(0x7fe3580fc470 msgr2=0x7fe3580fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 -- 172.21.15.57:0/3516023273 shutdown_connections 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.389+0000 7fe35ef32640 1 -- 172.21.15.57:0/3516023273 wait complete. 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 Processor -- start 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 -- start start 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe358100e70 0x7fe35806b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358102050 0x7fe35819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 0x7fe35819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810ca70 con 0x7fe358102a10 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810c8f0 con 0x7fe358100e70 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ef32640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810cbf0 con 0x7fe358102050 2024-04-21T08:44:49.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 0x7fe35819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 0x7fe35819f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33602/0 (socket says 172.21.15.57:33602) 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 -- 172.21.15.57:0/1136426072 learned_addr learned my addr 172.21.15.57:0/1136426072 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35ccce640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe358100e70 0x7fe35806b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe357fff640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358102050 0x7fe35819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 -- 172.21.15.57:0/1136426072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358102050 msgr2=0x7fe35819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358102050 0x7fe35819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 -- 172.21.15.57:0/1136426072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe358100e70 msgr2=0x7fe35806b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe358100e70 0x7fe35806b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe35d4cf640 1 -- 172.21.15.57:0/1136426072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35819fda0 con 0x7fe358102a10 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.390+0000 7fe357fff640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358102050 0x7fe35819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.391+0000 7fe35d4cf640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 0x7fe35819f490 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe35000cce0 tx=0x7fe35000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.391+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35000f450 con 0x7fe358102a10 2024-04-21T08:44:49.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.391+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35000faa0 con 0x7fe358102a10 2024-04-21T08:44:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.391+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350018b70 con 0x7fe358102a10 2024-04-21T08:44:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.391+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35819ffd0 con 0x7fe358102a10 2024-04-21T08:44:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.391+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3581a04a0 con 0x7fe358102a10 2024-04-21T08:44:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.392+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3581081b0 con 0x7fe358102a10 2024-04-21T08:44:49.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.393+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe350018d10 con 0x7fe358102a10 2024-04-21T08:44:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.393+0000 7fe355ffb640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe3480a3bb0 0x7fe3480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.394+0000 7fe35ccce640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe3480a3bb0 0x7fe3480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.394+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fe3500d0c30 con 0x7fe358102a10 2024-04-21T08:44:49.395 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.394+0000 7fe35ccce640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe3480a3bb0 0x7fe3480a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe34c005fd0 tx=0x7fe34c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.398+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe350014030 con 0x7fe358102a10 2024-04-21T08:44:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2263136506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2397007774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2199029564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:49 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4196855606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2263136506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2397007774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2199029564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:49 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4196855606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.627+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe358062bf0 con 0x7fe358102a10 2024-04-21T08:44:49.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.628+0000 7fe355ffb640 1 -- 172.21.15.57:0/1136426072 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fe35009d230 con 0x7fe358102a10 2024-04-21T08:44:49.627 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:49.627 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:44:49.628 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe3480a3bb0 msgr2=0x7fe3480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe3480a3bb0 0x7fe3480a6070 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe34c005fd0 tx=0x7fe34c00b390 comp rx=0 tx=0).stop 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 msgr2=0x7fe35819f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 0x7fe35819f490 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe35000cce0 tx=0x7fe35000a620 comp rx=0 tx=0).stop 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 shutdown_connections 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe3480a3bb0 0x7fe3480a6070 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe358102a10 0x7fe35819f490 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe358102050 0x7fe35819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 --2- 172.21.15.57:0/1136426072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe358100e70 0x7fe35806b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 >> 172.21.15.57:0/1136426072 conn(0x7fe3580fc470 msgr2=0x7fe358105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 shutdown_connections 2024-04-21T08:44:49.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.630+0000 7fe35ef32640 1 -- 172.21.15.57:0/1136426072 wait complete. 2024-04-21T08:44:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2263136506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2397007774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2199029564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:49 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4196855606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:44:49.641 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:44:49.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 -- 172.21.15.57:0/1886903494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984103bf0 msgr2=0x7fa98410a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 --2- 172.21.15.57:0/1886903494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984103bf0 0x7fa98410a480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009cd0 tx=0x7fa97c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 -- 172.21.15.57:0/1886903494 shutdown_connections 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 --2- 172.21.15.57:0/1886903494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984103bf0 0x7fa98410a480 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 --2- 172.21.15.57:0/1886903494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 0x7fa9841036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 --2- 172.21.15.57:0/1886903494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984100260 0x7fa984100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 -- 172.21.15.57:0/1886903494 >> 172.21.15.57:0/1886903494 conn(0x7fa9840fbb90 msgr2=0x7fa9840fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 -- 172.21.15.57:0/1886903494 shutdown_connections 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 -- 172.21.15.57:0/1886903494 wait complete. 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.744+0000 7fa9894d2640 1 Processor -- start 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 -- start start 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 0x7fa984198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 0x7fa984198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984103bf0 0x7fa98419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa98410dbe0 con 0x7fa984100260 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa98410da60 con 0x7fa984103230 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9894d2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa98410dd60 con 0x7fa984103bf0 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 0x7fa984198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 0x7fa984198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33624/0 (socket says 172.21.15.57:33624) 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 -- 172.21.15.57:0/2198149882 learned_addr learned my addr 172.21.15.57:0/2198149882 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9837fe640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984103bf0 0x7fa98419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa97bfff640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 0x7fa984198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 -- 172.21.15.57:0/2198149882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984103bf0 msgr2=0x7fa98419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984103bf0 0x7fa98419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 -- 172.21.15.57:0/2198149882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 msgr2=0x7fa984198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 0x7fa984198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 -- 172.21.15.57:0/2198149882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98419fe70 con 0x7fa984100260 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa9837fe640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984103bf0 0x7fa98419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:49.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa97bfff640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 0x7fa984198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa982ffd640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 0x7fa984198250 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa97400b450 tx=0x7fa97400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa974015020 con 0x7fa984100260 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.745+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa974011d40 con 0x7fa984100260 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.746+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9841a0160 con 0x7fa984100260 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.746+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97400f340 con 0x7fa984100260 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.746+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9841a06e0 con 0x7fa984100260 2024-04-21T08:44:49.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.747+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9841093d0 con 0x7fa984100260 2024-04-21T08:44:49.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.748+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa974018020 con 0x7fa984100260 2024-04-21T08:44:49.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.748+0000 7fa980ff9640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9700a3bb0 0x7fa9700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:49.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.748+0000 7fa97bfff640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9700a3bb0 0x7fa9700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:49.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.748+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa9740d02e0 con 0x7fa984100260 2024-04-21T08:44:49.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.749+0000 7fa97bfff640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9700a3bb0 0x7fa9700a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa96c006fd0 tx=0x7fa96c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:49.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.753+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa97401f020 con 0x7fa984100260 2024-04-21T08:44:49.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.941+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa984062b50 con 0x7fa984100260 2024-04-21T08:44:49.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.942+0000 7fa980ff9640 1 -- 172.21.15.57:0/2198149882 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7fa97400d040 con 0x7fa984100260 2024-04-21T08:44:49.942 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:49.942 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9700a3bb0 msgr2=0x7fa9700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9700a3bb0 0x7fa9700a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa96c006fd0 tx=0x7fa96c008040 comp rx=0 tx=0).stop 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 msgr2=0x7fa984198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 0x7fa984198250 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa97400b450 tx=0x7fa97400b910 comp rx=0 tx=0).stop 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 shutdown_connections 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa9700a3bb0 0x7fa9700a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa984103bf0 0x7fa98419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa984103230 0x7fa984198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 --2- 172.21.15.57:0/2198149882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa984100260 0x7fa984198250 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:49.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 >> 172.21.15.57:0/2198149882 conn(0x7fa9840fbb90 msgr2=0x7fa9841065a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:49.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.944+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 shutdown_connections 2024-04-21T08:44:49.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:49.945+0000 7fa9894d2640 1 -- 172.21.15.57:0/2198149882 wait complete. 2024-04-21T08:44:49.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:44:50.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.058+0000 7fe2ff996640 1 -- 172.21.15.57:0/1426104727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 msgr2=0x7fe2f8104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.058+0000 7fe2ff996640 1 --2- 172.21.15.57:0/1426104727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8104de0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009ca0 tx=0x7fe2e802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- 172.21.15.57:0/1426104727 shutdown_connections 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 --2- 172.21.15.57:0/1426104727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 0x7fe2f810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 --2- 172.21.15.57:0/1426104727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f8105320 0x7fe2f8107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 --2- 172.21.15.57:0/1426104727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8104de0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- 172.21.15.57:0/1426104727 >> 172.21.15.57:0/1426104727 conn(0x7fe2f80fc3e0 msgr2=0x7fe2f80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- 172.21.15.57:0/1426104727 shutdown_connections 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- 172.21.15.57:0/1426104727 wait complete. 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 Processor -- start 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- start start 2024-04-21T08:44:50.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f8105320 0x7fe2f8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 0x7fe2f819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f810db10 con 0x7fe2f81029f0 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f810d990 con 0x7fe2f8107c50 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2ff996640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f810dc90 con 0x7fe2f8105320 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.059+0000 7fe2fd732640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33650/0 (socket says 172.21.15.57:33650) 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 -- 172.21.15.57:0/2994913063 learned_addr learned my addr 172.21.15.57:0/2994913063 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fcf31640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f8105320 0x7fe2f8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 -- 172.21.15.57:0/2994913063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f8105320 msgr2=0x7fe2f8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f8105320 0x7fe2f8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fdf33640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 0x7fe2f819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 -- 172.21.15.57:0/2994913063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 msgr2=0x7fe2f819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 0x7fe2f819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 -- 172.21.15.57:0/2994913063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f819fe20 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fdf33640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 0x7fe2f819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2fd732640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8198200 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8000f80 tx=0x7fe2e8035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8039040 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f81a00b0 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e8003970 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8004460 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.060+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f81a05a0 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.061+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f80fded0 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.062+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe2e8046030 con 0x7fe2f81029f0 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.063+0000 7fe2f67fc640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe2e00a3bb0 0x7fe2e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.063+0000 7fe2fcf31640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe2e00a3bb0 0x7fe2e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.063+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fe2e80f2980 con 0x7fe2f81029f0 2024-04-21T08:44:50.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.063+0000 7fe2fcf31640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe2e00a3bb0 0x7fe2e00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8199710 tx=0x7fe2ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.066+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2e804d060 con 0x7fe2f81029f0 2024-04-21T08:44:50.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.294+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2f802d5c0 con 0x7fe2f81029f0 2024-04-21T08:44:50.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.295+0000 7fe2f67fc640 1 -- 172.21.15.57:0/2994913063 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fe2e80bf070 con 0x7fe2f81029f0 2024-04-21T08:44:50.294 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:44:50.294 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:44:50.295 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:44:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe2e00a3bb0 msgr2=0x7fe2e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe2e00a3bb0 0x7fe2e00a6070 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8199710 tx=0x7fe2ec008040 comp rx=0 tx=0).stop 2024-04-21T08:44:50.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 msgr2=0x7fe2f8198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8198200 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8000f80 tx=0x7fe2e8035d60 comp rx=0 tx=0).stop 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 shutdown_connections 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fe2e00a3bb0 0x7fe2e00a6070 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f8107c50 0x7fe2f819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2f8105320 0x7fe2f8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 --2- 172.21.15.57:0/2994913063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2f81029f0 0x7fe2f8198200 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 >> 172.21.15.57:0/2994913063 conn(0x7fe2f80fc3e0 msgr2=0x7fe2f8103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 shutdown_connections 2024-04-21T08:44:50.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.297+0000 7fe2ff996640 1 -- 172.21.15.57:0/2994913063 wait complete. 2024-04-21T08:44:50.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:42:56.988323+0000', 'modified': '2024-04-21T08:43:22.134494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6839/4293224778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 4293224778}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 4293224778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6839/123336774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 123336774}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 123336774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24503]} 2024-04-21T08:44:50.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:44:50.309 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:44:50.414 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.412+0000 7f2cb485c640 1 -- 172.21.15.57:0/3418009172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0101fe0 msgr2=0x7f2cb0102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.412+0000 7f2cb485c640 1 --2- 172.21.15.57:0/3418009172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0101fe0 0x7f2cb0102460 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009900 tx=0x7f2ca402bc30 comp rx=0 tx=0).stop 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 -- 172.21.15.57:0/3418009172 shutdown_connections 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 --2- 172.21.15.57:0/3418009172 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb01029a0 0x7f2cb01092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 --2- 172.21.15.57:0/3418009172 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0101fe0 0x7f2cb0102460 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 --2- 172.21.15.57:0/3418009172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb0100e90 0x7f2cb01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 -- 172.21.15.57:0/3418009172 >> 172.21.15.57:0/3418009172 conn(0x7f2cb00fc400 msgr2=0x7f2cb00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 -- 172.21.15.57:0/3418009172 shutdown_connections 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 -- 172.21.15.57:0/3418009172 wait complete. 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 Processor -- start 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.413+0000 7f2cb485c640 1 -- start start 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 0x7f2cb0197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0101fe0 0x7f2cb01975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb01029a0 0x7f2cb019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb010c9e0 con 0x7f2cb0100e90 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb010c860 con 0x7f2cb01029a0 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb010cb60 con 0x7f2cb0101fe0 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 0x7f2cb0197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 0x7f2cb0197080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33672/0 (socket says 172.21.15.57:33672) 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 -- 172.21.15.57:0/875792755 learned_addr learned my addr 172.21.15.57:0/875792755 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0101fe0 0x7f2cb01975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 -- 172.21.15.57:0/875792755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0101fe0 msgr2=0x7f2cb01975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0101fe0 0x7f2cb01975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 -- 172.21.15.57:0/875792755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb01029a0 msgr2=0x7f2cb019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2caed9d640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb01029a0 0x7f2cb019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb01029a0 0x7f2cb019e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 -- 172.21.15.57:0/875792755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb019eca0 con 0x7f2cb0100e90 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0101fe0 0x7f2cb01975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cae59c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 0x7f2cb0197080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000c920 tx=0x7f2ca000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2c9f7fe640 1 -- 172.21.15.57:0/875792755 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0007c70 con 0x7f2cb0100e90 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb019ef90 con 0x7f2cb0100e90 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2c9f7fe640 1 -- 172.21.15.57:0/875792755 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ca0007e10 con 0x7f2cb0100e90 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2c9f7fe640 1 -- 172.21.15.57:0/875792755 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0018820 con 0x7f2cb0100e90 2024-04-21T08:44:50.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.414+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb019f3f0 con 0x7f2cb0100e90 2024-04-21T08:44:50.423 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2c9f7fe640 1 -- 172.21.15.57:0/875792755 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2ca000f450 con 0x7f2cb0100e90 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 Processor -- start 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 -- start start 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40ce40 0x55818f40d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f40dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f40e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2c9f7fe640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2c980a3bb0 0x7f2c980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55818f371b10 con 0x55818f40d830 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55818f30f240 con 0x55818f40ce40 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cb7455740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55818f36dad0 con 0x55818f40e2d0 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.417+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f40dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2caed9d640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2c980a3bb0 0x7f2c980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f40dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33674/0 (socket says 172.21.15.57:33674) 2024-04-21T08:44:50.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 -- 172.21.15.57:0/2713575571 learned_addr learned my addr 172.21.15.57:0/2713575571 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2c9f7fe640 1 -- 172.21.15.57:0/875792755 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f2ca00d09a0 con 0x7f2cb0100e90 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f40e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2caed9d640 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40ce40 0x55818f40d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 -- 172.21.15.57:0/2713575571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 msgr2=0x55818f40e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f40e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 -- 172.21.15.57:0/2713575571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40ce40 msgr2=0x55818f40d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40ce40 0x55818f40d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 -- 172.21.15.57:0/2713575571 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55818f0842d0 con 0x55818f40d830 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2caed9d640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2c980a3bb0 0x7f2c980a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009270 tx=0x7f2ca80139c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2cae59c640 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f40dcb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0098d00 tx=0x7f2ca006f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdc6b58e0a7f1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2c9cff9640 1 -- 172.21.15.57:0/2713575571 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00a0c50 con 0x55818f40d830 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2c9cff9640 1 -- 172.21.15.57:0/2713575571 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ca00a5510 con 0x55818f40d830 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.418+0000 7f2c9cff9640 1 -- 172.21.15.57:0/2713575571 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000fb90 con 0x55818f40d830 2024-04-21T08:44:50.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 -- 172.21.15.57:0/2713575571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 msgr2=0x55818f40dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f40dcb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0098d00 tx=0x7f2ca006f3e0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 -- 172.21.15.57:0/2713575571 shutdown_connections 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f40e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f40dcb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2713575571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40ce40 0x55818f40d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 -- 172.21.15.57:0/2713575571 >> 172.21.15.57:0/2713575571 conn(0x55818f408f40 msgr2=0x55818f407490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 -- 172.21.15.57:0/2713575571 shutdown_connections 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 -- 172.21.15.57:0/2713575571 wait complete. 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.419+0000 7f2cb7455740 1 -- start start 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 Processor -- start 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f41b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40d830 0x55818f41b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 0x55818f4227e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55818f4198b0 con 0x55818f40ce40 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55818f2fa190 con 0x55818f40e2d0 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55818f419a30 con 0x55818f40d830 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 10 client.0 _ll_register_callbacks cb 0x55818f2f58b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f41b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f41b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33688/0 (socket says 172.21.15.57:33688) 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 -- 172.21.15.57:0/198415314 learned_addr learned my addr 172.21.15.57:0/198415314 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40d830 0x55818f41b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 -- 172.21.15.57:0/198415314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40d830 msgr2=0x55818f41b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40d830 0x55818f41b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 -- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 msgr2=0x55818f4227e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 0x55818f4227e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 -- 172.21.15.57:0/198415314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55818f14e6b0 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cae59c640 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f41b260 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0087040 tx=0x7f2ca009adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae0c9f58c56d4f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2c95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2c95ffb640 1 -- 172.21.15.57:0/198415314 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00a0c50 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2c95ffb640 1 -- 172.21.15.57:0/198415314 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f2ca009dbd0 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.420+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ca00a5510 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2c95ffb640 1 -- 172.21.15.57:0/198415314 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2ca00ba050 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 10 client.15210 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55818f425b50 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2c95ffb640 1 -- 172.21.15.57:0/198415314 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ca00a5510 con 0x55818f40ce40 2024-04-21T08:44:50.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2c95ffb640 1 -- 172.21.15.57:0/198415314 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2ca00baef0 con 0x55818f40ce40 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 10 client.15210 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 10 client.15210 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 20 client.15210 populate_metadata read hostname 'smithi057' 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f427b60 0x55818f428fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 4 client.15210 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55818f37f530 con 0x55818f427b60 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.421+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f427b60 0x55818f428fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.422+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f427b60 0x55818f428fd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3523c3d5a98d9605 server_cookie=cbf7bff5bcdb7f2c in_seq=0 out_seq=0 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.422+0000 7f2c95ffb640 10 client.15210 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.422+0000 7f2c95ffb640 1 -- 172.21.15.57:0/198415314 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55818f37f530 con 0x55818f427b60 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.422+0000 7f2c95ffb640 10 client.15210 handle_command_reply: tid=0 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 client.15210 shutdown 2024-04-21T08:44:50.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 msgr2=0x55818f41b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f41b260 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0087040 tx=0x7f2ca009adb0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 shutdown_connections 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f427b60 0x55818f428fd0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 0x55818f4227e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40d830 0x55818f41b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 --2- 172.21.15.57:0/198415314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f41b260 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2c95ffb640 0 client.15210 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 >> 172.21.15.57:0/198415314 conn(0x55818f408f40 msgr2=0x55818f406a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 shutdown_connections 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 1 -- 172.21.15.57:0/198415314 wait complete. 2024-04-21T08:44:50.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.423+0000 7f2cb7455740 20 client.15210 trim_cache size 0 max 16384 2024-04-21T08:44:50.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 Processor -- start 2024-04-21T08:44:50.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 -- start start 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f419790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40d830 0x55818f422770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f402800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55818f427310 con 0x55818f40ce40 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55818f4279e0 con 0x55818f40d830 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cb7455740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55818f4280b0 con 0x55818f40e2d0 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f419790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f419790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33700/0 (socket says 172.21.15.57:33700) 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 -- 172.21.15.57:0/1615311531 learned_addr learned my addr 172.21.15.57:0/1615311531 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40d830 0x55818f422770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2caed9d640 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f402800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 -- 172.21.15.57:0/1615311531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 msgr2=0x55818f402800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f402800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 -- 172.21.15.57:0/1615311531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40d830 msgr2=0x55818f422770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40d830 0x55818f422770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 -- 172.21.15.57:0/1615311531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55818f40e0f0 con 0x55818f40ce40 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.442+0000 7f2cae59c640 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f419790 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2ca009c830 tx=0x7f2ca000a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26e259931179190 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2c97fff640 1 -- 172.21.15.57:0/1615311531 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00a0c50 con 0x55818f40ce40 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2c97fff640 1 -- 172.21.15.57:0/1615311531 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ca00a5510 con 0x55818f40ce40 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2c97fff640 1 -- 172.21.15.57:0/1615311531 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00bae50 con 0x55818f40ce40 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 -- 172.21.15.57:0/1615311531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 msgr2=0x55818f419790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f419790 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2ca009c830 tx=0x7f2ca000a560 comp rx=0 tx=0).stop 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 -- 172.21.15.57:0/1615311531 shutdown_connections 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40e2d0 0x55818f402800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40d830 0x55818f422770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 --2- 172.21.15.57:0/1615311531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40ce40 0x55818f419790 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 -- 172.21.15.57:0/1615311531 >> 172.21.15.57:0/1615311531 conn(0x55818f408f40 msgr2=0x55818f419dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 -- 172.21.15.57:0/1615311531 shutdown_connections 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 -- 172.21.15.57:0/1615311531 wait complete. 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.443+0000 7f2cb7455740 1 -- start start 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 Processor -- start 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40ce40 0x55818f417fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f422700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 0x55818f4168c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55818f428e50 con 0x55818f40d830 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55818f428780 con 0x55818f40e2d0 2024-04-21T08:44:50.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55818f429520 con 0x55818f40ce40 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 10 client.0 _ll_register_callbacks cb 0x7f2ca009c200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f422700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f422700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33706/0 (socket says 172.21.15.57:33706) 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 -- 172.21.15.57:0/2728494329 learned_addr learned my addr 172.21.15.57:0/2728494329 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cae59c640 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40ce40 0x55818f417fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 -- 172.21.15.57:0/2728494329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40ce40 msgr2=0x55818f417fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40ce40 0x55818f417fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 -- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 msgr2=0x55818f4168c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 0x55818f4168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 -- 172.21.15.57:0/2728494329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55818f416e00 con 0x55818f40d830 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cadd9b640 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f422700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0100bb0 tx=0x7f2ca402ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915f53e74d801814 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2c95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:44:50.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2c95ffb640 1 -- 172.21.15.57:0/2728494329 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4004450 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ca00a5510 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.444+0000 7f2c95ffb640 1 -- 172.21.15.57:0/2728494329 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f2ca4005040 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2c95ffb640 1 -- 172.21.15.57:0/2728494329 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2ca4055050 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 10 client.15216 fetch_fsmap learned FSMap version 21 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55818f48e3b0 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2c95ffb640 1 -- 172.21.15.57:0/2728494329 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ca00a5510 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2c95ffb640 1 -- 172.21.15.57:0/2728494329 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2ca40035f0 con 0x55818f40d830 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 10 client.15216 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 10 client.15216 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 20 client.15216 populate_metadata read hostname 'smithi057' 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f493020 0x55818f491d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 4 client.15216 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "status"}) -- 0x55818f3793a0 con 0x55818f493020 2024-04-21T08:44:50.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.445+0000 7f2cae59c640 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f493020 0x55818f491d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cae59c640 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f493020 0x55818f491d30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=939fd3adf5879d44 server_cookie=a02e08f1dbe4dbcd in_seq=0 out_seq=0 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2c95ffb640 10 client.15216 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2c95ffb640 1 -- 172.21.15.57:0/2728494329 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55818f3793a0 con 0x55818f493020 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2c95ffb640 10 client.15216 handle_command_reply: tid=0 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 client.15216 shutdown 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 msgr2=0x55818f422700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f422700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0100bb0 tx=0x7f2ca402ccc0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 shutdown_connections 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55818f493020 0x55818f491d30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55818f40e2d0 0x55818f4168c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55818f40d830 0x55818f422700 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2c95ffb640 0 client.15216 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 --2- 172.21.15.57:0/2728494329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55818f40ce40 0x55818f417fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 >> 172.21.15.57:0/2728494329 conn(0x55818f408f40 msgr2=0x55818f402800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 shutdown_connections 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 1 -- 172.21.15.57:0/2728494329 wait complete. 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.446+0000 7f2cb7455740 20 client.15216 trim_cache size 0 max 16384 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 24503, 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2024-04-21T08:44:50.453 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 112.824749184, 2024-04-21T08:44:50.454 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 21, 2024-04-21T08:44:50.454 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 93, 2024-04-21T08:44:50.454 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:44:50.454 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 119.12758888899999 2024-04-21T08:44:50.454 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2c980a3bb0 msgr2=0x7f2c980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2c980a3bb0 0x7f2c980a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009270 tx=0x7f2ca80139c0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 msgr2=0x7f2cb0197080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 0x7f2cb0197080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000c920 tx=0x7f2ca000cde0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 shutdown_connections 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f2c980a3bb0 0x7f2c980a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb01029a0 0x7f2cb019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cb0101fe0 0x7f2cb01975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 --2- 172.21.15.57:0/875792755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2cb0100e90 0x7f2cb0197080 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 >> 172.21.15.57:0/875792755 conn(0x7f2cb00fc400 msgr2=0x7f2cb0105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:50.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 shutdown_connections 2024-04-21T08:44:50.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:50.447+0000 7f2cb485c640 1 -- 172.21.15.57:0/875792755 wait complete. 2024-04-21T08:44:50.457 INFO:tasks.cephadm:Setup complete, yielding 2024-04-21T08:44:50.458 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:44:50.468 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:44:50.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-21T08:44:50.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:50 smithi129 ceph-mon[31272]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:50.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:50 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1136426072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:50 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2198149882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:50 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2994913063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:50 smithi057 ceph-mon[27968]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1136426072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2198149882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2994913063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:50 smithi136 ceph-mon[31791]: pgmap v355: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:50 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1136426072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:50 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2198149882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:50 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2994913063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:44:50.716 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:52.474 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:52 smithi057 ceph-mon[27968]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T08:44:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 -- 172.21.15.57:0/1951330781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 msgr2=0x7f492c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 --2- 172.21.15.57:0/1951330781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c101dc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f49140099c0 tx=0x7f4914009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 -- 172.21.15.57:0/1951330781 shutdown_connections 2024-04-21T08:44:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 --2- 172.21.15.57:0/1951330781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f492c103540 0x7f492c109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 --2- 172.21.15.57:0/1951330781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c101dc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 --2- 172.21.15.57:0/1951330781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f492c102ba0 0x7f492c103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 -- 172.21.15.57:0/1951330781 >> 172.21.15.57:0/1951330781 conn(0x7f492c0fcf40 msgr2=0x7f492c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 -- 172.21.15.57:0/1951330781 shutdown_connections 2024-04-21T08:44:52.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.585+0000 7f49323ec700 1 -- 172.21.15.57:0/1951330781 wait complete. 2024-04-21T08:44:52.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.586+0000 7f49323ec700 1 Processor -- start 2024-04-21T08:44:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.586+0000 7f49323ec700 1 -- start start 2024-04-21T08:44:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f49323ec700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c06b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f49323ec700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f492c102ba0 0x7f492c19d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f49323ec700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f492c103540 0x7f492c19ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f49323ec700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f492c10d460 con 0x7f492c1019a0 2024-04-21T08:44:52.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f49323ec700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f492c10d2e0 con 0x7f492c102ba0 2024-04-21T08:44:52.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f49323ec700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f492c10d5e0 con 0x7f492c103540 2024-04-21T08:44:52.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c06b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c06b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33720/0 (socket says 172.21.15.57:33720) 2024-04-21T08:44:52.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 -- 172.21.15.57:0/375664219 learned_addr learned my addr 172.21.15.57:0/375664219 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:52.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492b7fe700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f492c102ba0 0x7f492c19d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f4930989700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f492c103540 0x7f492c19ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 -- 172.21.15.57:0/375664219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f492c103540 msgr2=0x7f492c19ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f492c103540 0x7f492c19ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 -- 172.21.15.57:0/375664219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f492c102ba0 msgr2=0x7f492c19d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f492c102ba0 0x7f492c19d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 -- 172.21.15.57:0/375664219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c1a07d0 con 0x7f492c1019a0 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.587+0000 7f492bfff700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c06b2a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4914009f50 tx=0x7f4914003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.588+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914027070 con 0x7f492c1019a0 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.588+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4914004190 con 0x7f492c1019a0 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.588+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492c1a0a60 con 0x7f492c1019a0 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.588+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491400f460 con 0x7f492c1019a0 2024-04-21T08:44:52.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.588+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492c1a0eb0 con 0x7f492c1019a0 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.589+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f492c02df80 con 0x7f492c1019a0 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.590+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f491402a070 con 0x7f492c1019a0 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.590+0000 7f49297fa700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f49180a3c40 0x7f49180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.591+0000 7f492b7fe700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f49180a3c40 0x7f49180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:52.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.591+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f49140d1d80 con 0x7f492c1019a0 2024-04-21T08:44:52.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.591+0000 7f492b7fe700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f49180a3c40 0x7f49180a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f492c06b800 tx=0x7f491c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:52.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.594+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4914031070 con 0x7f492c1019a0 2024-04-21T08:44:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:52 smithi136 ceph-mon[31791]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T08:44:52.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.822+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f492c062e80 con 0x7f49180a3c40 2024-04-21T08:44:52.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.823+0000 7f49297fa700 1 -- 172.21.15.57:0/375664219 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f492c062e80 con 0x7f49180a3c40 2024-04-21T08:44:52.823 INFO:teuthology.orchestra.run.smithi057.stdout:Backend: cephadm 2024-04-21T08:44:52.823 INFO:teuthology.orchestra.run.smithi057.stdout:Available: Yes 2024-04-21T08:44:52.823 INFO:teuthology.orchestra.run.smithi057.stdout:Paused: No 2024-04-21T08:44:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:52 smithi129 ceph-mon[31272]: pgmap v356: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f49180a3c40 msgr2=0x7f49180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f49180a3c40 0x7f49180a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f492c06b800 tx=0x7f491c00a380 comp rx=0 tx=0).stop 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 msgr2=0x7f492c06b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c06b2a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4914009f50 tx=0x7f4914003750 comp rx=0 tx=0).stop 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 shutdown_connections 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f49180a3c40 0x7f49180a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f492c103540 0x7f492c19ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f492c1019a0 0x7f492c06b2a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 --2- 172.21.15.57:0/375664219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f492c102ba0 0x7f492c19d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:52.827 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 >> 172.21.15.57:0/375664219 conn(0x7f492c0fcf40 msgr2=0x7f492c105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:52.827 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 shutdown_connections 2024-04-21T08:44:52.827 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:52.826+0000 7f49323ec700 1 -- 172.21.15.57:0/375664219 wait complete. 2024-04-21T08:44:53.470 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-21T08:44:53.728 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:54 smithi057 ceph-mon[27968]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:54 smithi057 ceph-mon[27968]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:54 smithi136 ceph-mon[31791]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:54 smithi136 ceph-mon[31791]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:54 smithi129 ceph-mon[31272]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-21T08:44:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:54 smithi129 ceph-mon[31272]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:55.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 -- 172.21.15.57:0/581585190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4103d40 msgr2=0x7f8fd41041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 --2- 172.21.15.57:0/581585190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4103d40 0x7f8fd41041a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009990 tx=0x7f8fc8009ca0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 -- 172.21.15.57:0/581585190 shutdown_connections 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 --2- 172.21.15.57:0/581585190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fd41046e0 0x7f8fd410af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 --2- 172.21.15.57:0/581585190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd4100d60 0x7f8fd4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 --2- 172.21.15.57:0/581585190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4103d40 0x7f8fd41041a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 -- 172.21.15.57:0/581585190 >> 172.21.15.57:0/581585190 conn(0x7f8fd40fc6a0 msgr2=0x7f8fd40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 -- 172.21.15.57:0/581585190 shutdown_connections 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.484+0000 7f8fd8848700 1 -- 172.21.15.57:0/581585190 wait complete. 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 Processor -- start 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 -- start start 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4100d60 0x7f8fd41036a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fd4103d40 0x7f8fd4101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 0x7f8fd4102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd410e520 con 0x7f8fd4100d60 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd410e3a0 con 0x7f8fd4103d40 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd8848700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd410e6a0 con 0x7f8fd41046e0 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 0x7f8fd4102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4100d60 0x7f8fd41036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd2d9d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 0x7f8fd4102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:50470/0 (socket says 172.21.15.57:50470) 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd2d9d700 1 -- 172.21.15.57:0/572864634 learned_addr learned my addr 172.21.15.57:0/572864634 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd1d9b700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fd4103d40 0x7f8fd4101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd259c700 1 -- 172.21.15.57:0/572864634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 msgr2=0x7f8fd4102230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd259c700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 0x7f8fd4102230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd259c700 1 -- 172.21.15.57:0/572864634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fd4103d40 msgr2=0x7f8fd4101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd259c700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fd4103d40 0x7f8fd4101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd259c700 1 -- 172.21.15.57:0/572864634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd4102ae0 con 0x7f8fd4100d60 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.485+0000 7f8fd2d9d700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 0x7f8fd4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.486+0000 7f8fd259c700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4100d60 0x7f8fd41036a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400d720 tx=0x7f8fc400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.486+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4016070 con 0x7f8fd4100d60 2024-04-21T08:44:55.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.486+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd41a2d60 con 0x7f8fd4100d60 2024-04-21T08:44:55.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.486+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fc4005020 con 0x7f8fd4100d60 2024-04-21T08:44:55.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.486+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400faf0 con 0x7f8fd4100d60 2024-04-21T08:44:55.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.486+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd41a31c0 con 0x7f8fd4100d60 2024-04-21T08:44:55.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.487+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd402df80 con 0x7f8fd4100d60 2024-04-21T08:44:55.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.488+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8fc40051d0 con 0x7f8fd4100d60 2024-04-21T08:44:55.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.489+0000 7f8fc37fe700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8fbc0a3c40 0x7f8fbc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:55.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.489+0000 7f8fd1d9b700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8fbc0a3c40 0x7f8fbc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:55.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.489+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f8fc40d01c0 con 0x7f8fd4100d60 2024-04-21T08:44:55.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.489+0000 7f8fd1d9b700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8fbc0a3c40 0x7f8fbc0a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800b5c0 tx=0x7f8fc8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:55.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.493+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8fc409c7c0 con 0x7f8fd4100d60 2024-04-21T08:44:55.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.711+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f8fd4062e80 con 0x7f8fbc0a3c40 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.725+0000 7f8fc37fe700 1 -- 172.21.15.57:0/572864634 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f8fd4062e80 con 0x7f8fbc0a3c40 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.a smithi057 running (2m) 21s ago 2m 15.3M - 19.0.0-3154-ga630765c b45a994450e2 d9b3e4ca368b 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.b smithi129 running (2m) 57s ago 2m 25.6M - 19.0.0-3154-ga630765c b45a994450e2 302607c3a1b7 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.c smithi136 running (2m) 57s ago 2m 16.1M - 19.0.0-3154-ga630765c b45a994450e2 ea94fe2f9854 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.d smithi057 running (2m) 21s ago 2m 15.5M - 19.0.0-3154-ga630765c b45a994450e2 daf41631cee2 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.e smithi129 running (2m) 57s ago 2m 15.3M - 19.0.0-3154-ga630765c b45a994450e2 0317caf43338 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.f smithi136 running (2m) 57s ago 2m 13.0M - 19.0.0-3154-ga630765c b45a994450e2 445152d84157 2024-04-21T08:44:55.725 INFO:teuthology.orchestra.run.smithi057.stdout:mds.g smithi057 running (2m) 21s ago 2m 15.1M - 19.0.0-3154-ga630765c b45a994450e2 d8db5d41b5b7 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mds.h smithi129 running (2m) 57s ago 2m 15.3M - 19.0.0-3154-ga630765c b45a994450e2 aa35b7b8ebba 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mds.i smithi136 running (2m) 57s ago 2m 17.2M - 19.0.0-3154-ga630765c b45a994450e2 d8780213deb4 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mds.j smithi057 running (2m) 21s ago 2m 14.0M - 19.0.0-3154-ga630765c b45a994450e2 4519deced7b1 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mds.k smithi129 running (2m) 57s ago 2m 16.4M - 19.0.0-3154-ga630765c b45a994450e2 2f532aef29f7 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mds.l smithi136 running (2m) 57s ago 2m 13.1M - 19.0.0-3154-ga630765c b45a994450e2 e954924f42a4 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mgr.x smithi057 *:9283,8765 running (9m) 21s ago 9m 515M - 19.0.0-3154-ga630765c b45a994450e2 c79614f5e3d1 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mgr.y smithi129 *:8443,8765 running (7m) 57s ago 7m 452M - 19.0.0-3154-ga630765c b45a994450e2 b3b453020d02 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mgr.z smithi136 *:8443,8765 running (8m) 57s ago 8m 451M - 19.0.0-3154-ga630765c b45a994450e2 b1b5b8418733 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mon.a smithi057 running (10m) 21s ago 10m 52.5M 2048M 19.0.0-3154-ga630765c b45a994450e2 585910ff514b 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mon.b smithi129 running (8m) 57s ago 8m 41.8M 2048M 19.0.0-3154-ga630765c b45a994450e2 a066933c3cdf 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:mon.c smithi136 running (8m) 57s ago 8m 39.5M 2048M 19.0.0-3154-ga630765c b45a994450e2 830ca3f1e599 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 smithi057 running (7m) 21s ago 7m 45.5M 4273M 19.0.0-3154-ga630765c b45a994450e2 788dcec6d3f7 2024-04-21T08:44:55.726 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 smithi129 running (7m) 57s ago 7m 59.2M 4272M 19.0.0-3154-ga630765c b45a994450e2 7b0cf388cccd 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 smithi136 running (6m) 57s ago 6m 47.5M 4272M 19.0.0-3154-ga630765c b45a994450e2 bcf0f6e7c212 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 smithi057 running (6m) 21s ago 6m 46.7M 4273M 19.0.0-3154-ga630765c b45a994450e2 be3da817f3a4 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 smithi129 running (6m) 57s ago 6m 42.9M 4272M 19.0.0-3154-ga630765c b45a994450e2 3ca255951bdf 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 smithi136 running (5m) 57s ago 5m 44.8M 4272M 19.0.0-3154-ga630765c b45a994450e2 80023ee2d437 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 smithi057 running (5m) 21s ago 5m 53.3M 4273M 19.0.0-3154-ga630765c b45a994450e2 056ebc46522a 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 smithi129 running (4m) 57s ago 4m 43.9M 4272M 19.0.0-3154-ga630765c b45a994450e2 cc7504d19fd8 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.8 smithi136 running (4m) 57s ago 4m 52.5M 4272M 19.0.0-3154-ga630765c b45a994450e2 b68aead64152 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.9 smithi057 running (3m) 21s ago 3m 56.7M 4273M 19.0.0-3154-ga630765c b45a994450e2 108cc624b3d9 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.10 smithi129 running (3m) 57s ago 3m 42.3M 4272M 19.0.0-3154-ga630765c b45a994450e2 826f930b8a73 2024-04-21T08:44:55.727 INFO:teuthology.orchestra.run.smithi057.stdout:osd.11 smithi136 running (3m) 57s ago 3m 53.1M 4272M 19.0.0-3154-ga630765c b45a994450e2 0539d2f76ea9 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.727+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8fbc0a3c40 msgr2=0x7f8fbc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.727+0000 7f8fd8848700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8fbc0a3c40 0x7f8fbc0a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800b5c0 tx=0x7f8fc8005fb0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.727+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4100d60 msgr2=0x7f8fd41036a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.727+0000 7f8fd8848700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4100d60 0x7f8fd41036a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400d720 tx=0x7f8fc400da30 comp rx=0 tx=0).stop 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.727+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 shutdown_connections 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.727+0000 7f8fd8848700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8fbc0a3c40 0x7f8fbc0a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.728+0000 7f8fd8848700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fd41046e0 0x7f8fd4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.728+0000 7f8fd8848700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fd4100d60 0x7f8fd41036a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.728+0000 7f8fd8848700 1 --2- 172.21.15.57:0/572864634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fd4103d40 0x7f8fd4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:55.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.728+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 >> 172.21.15.57:0/572864634 conn(0x7f8fd40fc6a0 msgr2=0x7f8fd4107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:55.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.728+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 shutdown_connections 2024-04-21T08:44:55.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:55.728+0000 7f8fd8848700 1 -- 172.21.15.57:0/572864634 wait complete. 2024-04-21T08:44:56.486 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:56 smithi057 ceph-mon[27968]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 614 B/s rd, 1 op/s 2024-04-21T08:44:56.522 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-21T08:44:56.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:56 smithi136 ceph-mon[31791]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 614 B/s rd, 1 op/s 2024-04-21T08:44:56.778 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:44:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:56 smithi129 ceph-mon[31272]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 614 B/s rd, 1 op/s 2024-04-21T08:44:57.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:57 smithi057 ceph-mon[27968]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:57.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:57 smithi136 ceph-mon[31791]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:57.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:57 smithi129 ceph-mon[31272]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:44:58.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:44:58 smithi057 ceph-mon[27968]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 -- 172.21.15.57:0/3131999781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c102ac0 msgr2=0x7fa59c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 --2- 172.21.15.57:0/3131999781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c102ac0 0x7fa59c102ee0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa5940099c0 tx=0x7fa594009cd0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 -- 172.21.15.57:0/3131999781 shutdown_connections 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 --2- 172.21.15.57:0/3131999781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa59c104660 0x7fa59c10af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 --2- 172.21.15.57:0/3131999781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c102ac0 0x7fa59c102ee0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 --2- 172.21.15.57:0/3131999781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa59c103cc0 0x7fa59c104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 -- 172.21.15.57:0/3131999781 >> 172.21.15.57:0/3131999781 conn(0x7fa59c0fe060 msgr2=0x7fa59c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 -- 172.21.15.57:0/3131999781 shutdown_connections 2024-04-21T08:44:58.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.590+0000 7fa5a1795700 1 -- 172.21.15.57:0/3131999781 wait complete. 2024-04-21T08:44:58.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.591+0000 7fa5a1795700 1 Processor -- start 2024-04-21T08:44:58.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.591+0000 7fa5a1795700 1 -- start start 2024-04-21T08:44:58.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa5a1795700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa59c102ac0 0x7fa59c198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:58.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa5a1795700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa59c103cc0 0x7fa59c199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa5a1795700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 0x7fa59c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa5a1795700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c10e5a0 con 0x7fa59c104660 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa5a1795700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c10e420 con 0x7fa59c102ac0 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa5a1795700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa59c10e720 con 0x7fa59c103cc0 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 0x7fa59c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 0x7fa59c1a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41990/0 (socket says 172.21.15.57:41990) 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 -- 172.21.15.57:0/222579328 learned_addr learned my addr 172.21.15.57:0/222579328 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59a7fc700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa59c103cc0 0x7fa59c199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59affd700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa59c102ac0 0x7fa59c198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 -- 172.21.15.57:0/222579328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa59c103cc0 msgr2=0x7fa59c199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa59c103cc0 0x7fa59c199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 -- 172.21.15.57:0/222579328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa59c102ac0 msgr2=0x7fa59c198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:58.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa59c102ac0 0x7fa59c198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.592+0000 7fa59b7fe700 1 -- 172.21.15.57:0/222579328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59c1a0840 con 0x7fa59c104660 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.593+0000 7fa59b7fe700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 0x7fa59c1a01a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa58c007da0 tx=0x7fa58c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.593+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c016020 con 0x7fa59c104660 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.593+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59c1a0b30 con 0x7fa59c104660 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.593+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa58c00ac50 con 0x7fa59c104660 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.593+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0115e0 con 0x7fa59c104660 2024-04-21T08:44:58.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.593+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59c1a1070 con 0x7fa59c104660 2024-04-21T08:44:58.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.594+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa59c069730 con 0x7fa59c104660 2024-04-21T08:44:58.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.595+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa58c019020 con 0x7fa59c104660 2024-04-21T08:44:58.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.595+0000 7fa593fff700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5880a3c40 0x7fa5880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:44:58.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.596+0000 7fa59affd700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5880a3c40 0x7fa5880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:44:58.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.596+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa58c0d0b90 con 0x7fa59c104660 2024-04-21T08:44:58.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.596+0000 7fa59affd700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5880a3c40 0x7fa5880a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa594009f50 tx=0x7fa59400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:44:58.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.599+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa58c020020 con 0x7fa59c104660 2024-04-21T08:44:58.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:44:58 smithi136 ceph-mon[31791]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:58.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:44:58 smithi129 ceph-mon[31272]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-21T08:44:58.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.816+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fa59c062e80 con 0x7fa5880a3c40 2024-04-21T08:44:58.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.822+0000 7fa593fff700 1 -- 172.21.15.57:0/222579328 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fa59c062e80 con 0x7fa5880a3c40 2024-04-21T08:44:58.822 INFO:teuthology.orchestra.run.smithi057.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-21T08:44:58.822 INFO:teuthology.orchestra.run.smithi057.stdout:mds.all 12/12 60s ago 2m smithi057=a;smithi057=d;smithi057=g;smithi057=j;smithi129=b;smithi129=e;smithi129=h;smithi129=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-21T08:44:58.822 INFO:teuthology.orchestra.run.smithi057.stdout:mgr 3/3 60s ago 8m smithi057=x;smithi129=y;smithi136=z;count:3 2024-04-21T08:44:58.822 INFO:teuthology.orchestra.run.smithi057.stdout:mon 3/3 60s ago 8m smithi057:172.21.15.57=a;smithi129:172.21.15.129=b;smithi136:172.21.15.136=c;count:3 2024-04-21T08:44:58.822 INFO:teuthology.orchestra.run.smithi057.stdout:osd 12 60s ago - 2024-04-21T08:44:58.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5880a3c40 msgr2=0x7fa5880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:58.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5880a3c40 0x7fa5880a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa594009f50 tx=0x7fa59400b540 comp rx=0 tx=0).stop 2024-04-21T08:44:58.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 msgr2=0x7fa59c1a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:44:58.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 0x7fa59c1a01a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fa58c007da0 tx=0x7fa58c00a340 comp rx=0 tx=0).stop 2024-04-21T08:44:58.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 shutdown_connections 2024-04-21T08:44:58.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5880a3c40 0x7fa5880a6100 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa59c102ac0 0x7fa59c198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa59c104660 0x7fa59c1a01a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 --2- 172.21.15.57:0/222579328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa59c103cc0 0x7fa59c199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:44:58.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 >> 172.21.15.57:0/222579328 conn(0x7fa59c0fe060 msgr2=0x7fa59c106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:44:58.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 shutdown_connections 2024-04-21T08:44:58.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:44:58.825+0000 7fa5a1795700 1 -- 172.21.15.57:0/222579328 wait complete. 2024-04-21T08:44:59.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-21T08:44:59.747 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:00 smithi057 ceph-mon[27968]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 904 B/s rd, 1 op/s 2024-04-21T08:45:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:00 smithi057 ceph-mon[27968]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:00 smithi136 ceph-mon[31791]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 904 B/s rd, 1 op/s 2024-04-21T08:45:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:00 smithi136 ceph-mon[31791]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:00 smithi129 ceph-mon[31272]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 904 B/s rd, 1 op/s 2024-04-21T08:45:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:00 smithi129 ceph-mon[31272]: from='client.15234 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:01.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 -- 172.21.15.57:0/1131805537 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 msgr2=0x7fa54c1011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1131805537 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c1011b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa53c009990 tx=0x7fa53c009ca0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 -- 172.21.15.57:0/1131805537 shutdown_connections 2024-04-21T08:45:01.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1131805537 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c104710 0x7fa54c10afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1131805537 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c1011b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1131805537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54c103d70 0x7fa54c1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 -- 172.21.15.57:0/1131805537 >> 172.21.15.57:0/1131805537 conn(0x7fa54c0fc6f0 msgr2=0x7fa54c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 -- 172.21.15.57:0/1131805537 shutdown_connections 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 -- 172.21.15.57:0/1131805537 wait complete. 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 Processor -- start 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.506+0000 7fa5531a6700 1 -- start start 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c103d70 0x7fa54c199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54c104710 0x7fa54c1a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa54c10e5b0 con 0x7fa54c100d90 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa54c10e430 con 0x7fa54c103d70 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa54c10e730 con 0x7fa54c104710 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41998/0 (socket says 172.21.15.57:41998) 2024-04-21T08:45:01.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 -- 172.21.15.57:0/1692824994 learned_addr learned my addr 172.21.15.57:0/1692824994 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 -- 172.21.15.57:0/1692824994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54c104710 msgr2=0x7fa54c1a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa551743700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54c104710 0x7fa54c1a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54c104710 0x7fa54c1a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa54bfff700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c103d70 0x7fa54c199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 -- 172.21.15.57:0/1692824994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c103d70 msgr2=0x7fa54c199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c103d70 0x7fa54c199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 -- 172.21.15.57:0/1692824994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54c1a0860 con 0x7fa54c100d90 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa54bfff700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c103d70 0x7fa54c199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa550f42700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c198c10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00b5c0 tx=0x7fa53c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c027070 con 0x7fa54c100d90 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa53c022470 con 0x7fa54c100d90 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa54c1a0af0 con 0x7fa54c100d90 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.507+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c00f6a0 con 0x7fa54c100d90 2024-04-21T08:45:01.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.508+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa54c1a0fc0 con 0x7fa54c100d90 2024-04-21T08:45:01.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.509+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa54c069730 con 0x7fa54c100d90 2024-04-21T08:45:01.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.510+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa53c02a070 con 0x7fa54c100d90 2024-04-21T08:45:01.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.510+0000 7fa549ffb700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5340a3c40 0x7fa5340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:01.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.510+0000 7fa54bfff700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5340a3c40 0x7fa5340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:01.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.510+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa53c0d1650 con 0x7fa54c100d90 2024-04-21T08:45:01.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.511+0000 7fa54bfff700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5340a3c40 0x7fa5340a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa540007900 tx=0x7fa540008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:01.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.515+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa53c013070 con 0x7fa54c100d90 2024-04-21T08:45:01.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.732+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fa54c062e80 con 0x7fa5340a3c40 2024-04-21T08:45:01.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.734+0000 7fa549ffb700 1 -- 172.21.15.57:0/1692824994 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fa54c062e80 con 0x7fa5340a3c40 2024-04-21T08:45:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:HOST ADDR LABELS STATUS 2024-04-21T08:45:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 172.21.15.57 2024-04-21T08:45:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:smithi129 172.21.15.129 2024-04-21T08:45:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:smithi136 172.21.15.136 2024-04-21T08:45:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:3 hosts in cluster 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5340a3c40 msgr2=0x7fa5340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5340a3c40 0x7fa5340a6100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa540007900 tx=0x7fa540008040 comp rx=0 tx=0).stop 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 msgr2=0x7fa54c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c198c10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00b5c0 tx=0x7fa53c009f90 comp rx=0 tx=0).stop 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 shutdown_connections 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa5340a3c40 0x7fa5340a6100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa54c104710 0x7fa54c1a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa54c100d90 0x7fa54c198c10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 --2- 172.21.15.57:0/1692824994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa54c103d70 0x7fa54c199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 >> 172.21.15.57:0/1692824994 conn(0x7fa54c0fc6f0 msgr2=0x7fa54c107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 shutdown_connections 2024-04-21T08:45:01.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:01.736+0000 7fa5531a6700 1 -- 172.21.15.57:0/1692824994 wait complete. 2024-04-21T08:45:02.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-21T08:45:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:02 smithi136 ceph-mon[31791]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:02.667 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:02.789 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:02 smithi057 ceph-mon[27968]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:02 smithi129 ceph-mon[31272]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:03.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:03 smithi136 ceph-mon[31791]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:03.816 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:03 smithi057 ceph-mon[27968]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:03 smithi129 ceph-mon[31272]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 -- 172.21.15.57:0/497613025 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70102aa0 msgr2=0x7f0a70102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 --2- 172.21.15.57:0/497613025 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70102aa0 0x7f0a70102ec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0a600099c0 tx=0x7f0a60009cd0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 -- 172.21.15.57:0/497613025 shutdown_connections 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 --2- 172.21.15.57:0/497613025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a70104640 0x7f0a7010aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 --2- 172.21.15.57:0/497613025 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70102aa0 0x7f0a70102ec0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 --2- 172.21.15.57:0/497613025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a70103ca0 0x7f0a70104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 -- 172.21.15.57:0/497613025 >> 172.21.15.57:0/497613025 conn(0x7f0a700fe040 msgr2=0x7f0a70100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 -- 172.21.15.57:0/497613025 shutdown_connections 2024-04-21T08:45:04.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.435+0000 7f0a761b7700 1 -- 172.21.15.57:0/497613025 wait complete. 2024-04-21T08:45:04.436 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:04 smithi057 ceph-mon[27968]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 Processor -- start 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 -- start start 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a70102aa0 0x7f0a70198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 0x7f0a70199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a70104640 0x7f0a701a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010e510 con 0x7f0a70103ca0 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010e390 con 0x7f0a70102aa0 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a761b7700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010e690 con 0x7f0a70104640 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a6effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 0x7f0a70199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a6effd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 0x7f0a70199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44706/0 (socket says 172.21.15.57:44706) 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a6effd700 1 -- 172.21.15.57:0/2707334650 learned_addr learned my addr 172.21.15.57:0/2707334650 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:04.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.436+0000 7f0a6f7fe700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a70102aa0 0x7f0a70198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6ffff700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a70104640 0x7f0a701a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6effd700 1 -- 172.21.15.57:0/2707334650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a70104640 msgr2=0x7f0a701a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6effd700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a70104640 0x7f0a701a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6effd700 1 -- 172.21.15.57:0/2707334650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a70102aa0 msgr2=0x7f0a70198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6effd700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a70102aa0 0x7f0a70198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6effd700 1 -- 172.21.15.57:0/2707334650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a701a0750 con 0x7f0a70103ca0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6effd700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 0x7f0a70199030 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c9e0 tx=0x7f0a6400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64007ac0 con 0x7f0a70103ca0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a64007c60 con 0x7f0a70103ca0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a701a0a40 con 0x7f0a70103ca0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64018660 con 0x7f0a70103ca0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.437+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a701a0f80 con 0x7f0a70103ca0 2024-04-21T08:45:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.438+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a70069730 con 0x7f0a70103ca0 2024-04-21T08:45:04.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.440+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0a6400f450 con 0x7f0a70103ca0 2024-04-21T08:45:04.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.441+0000 7f0a6cff9700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a580a3c40 0x7f0a580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:04.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.441+0000 7f0a6f7fe700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a580a3c40 0x7f0a580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:04.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.441+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f0a640d09d0 con 0x7f0a70103ca0 2024-04-21T08:45:04.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.442+0000 7f0a6f7fe700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a580a3c40 0x7f0a580a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a60006010 tx=0x7f0a60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:04.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.445+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a64014030 con 0x7f0a70103ca0 2024-04-21T08:45:04.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:04 smithi136 ceph-mon[31791]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:04.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.663+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 --> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f0a70062e80 con 0x7f0a580a3c40 2024-04-21T08:45:04.664 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.664+0000 7f0a6cff9700 1 -- 172.21.15.57:0/2707334650 <== mgr.14152 v2:172.21.15.57:6800/3002879345 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f0a70062e80 con 0x7f0a580a3c40 2024-04-21T08:45:04.664 INFO:teuthology.orchestra.run.smithi057.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-21T08:45:04.664 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE7361049H4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:45:04.664 INFO:teuthology.orchestra.run.smithi057.stdout:smithi129 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300HY400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:45:04.664 INFO:teuthology.orchestra.run.smithi057.stdout:smithi136 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD514100122P0SGN 1863G No 3m ago Has a FileSystem, LVM detected 2024-04-21T08:45:04.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.666+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a580a3c40 msgr2=0x7f0a580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:04.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.666+0000 7f0a761b7700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a580a3c40 0x7f0a580a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a60006010 tx=0x7f0a60000bc0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.666+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 msgr2=0x7f0a70199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.666+0000 7f0a761b7700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 0x7f0a70199030 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c9e0 tx=0x7f0a6400ccf0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 shutdown_connections 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0a580a3c40 0x7f0a580a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a70102aa0 0x7f0a70198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a70104640 0x7f0a701a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 --2- 172.21.15.57:0/2707334650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a70103ca0 0x7f0a70199030 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 >> 172.21.15.57:0/2707334650 conn(0x7f0a700fe040 msgr2=0x7f0a70106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 shutdown_connections 2024-04-21T08:45:04.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:04.667+0000 7f0a761b7700 1 -- 172.21.15.57:0/2707334650 wait complete. 2024-04-21T08:45:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:04 smithi129 ceph-mon[31272]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:05.386 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-21T08:45:05.395 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:45:05.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-21T08:45:05.652 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:06 smithi136 ceph-mon[31791]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:06 smithi136 ceph-mon[31791]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:06.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:06 smithi057 ceph-mon[27968]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:06.636 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:06 smithi057 ceph-mon[27968]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:06 smithi129 ceph-mon[31272]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:06 smithi129 ceph-mon[31272]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.547+0000 7fbbc2aed700 1 -- 172.21.15.57:0/3401435864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1019c0 msgr2=0x7fbbbc101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.547+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/3401435864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1019c0 0x7fbbbc101de0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00099c0 tx=0x7fbbb0009cd0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- 172.21.15.57:0/3401435864 shutdown_connections 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/3401435864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbbc103560 0x7fbbbc109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/3401435864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1019c0 0x7fbbbc101de0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/3401435864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbbc102bc0 0x7fbbbc103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- 172.21.15.57:0/3401435864 >> 172.21.15.57:0/3401435864 conn(0x7fbbbc0fcfa0 msgr2=0x7fbbbc0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- 172.21.15.57:0/3401435864 shutdown_connections 2024-04-21T08:45:07.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- 172.21.15.57:0/3401435864 wait complete. 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 Processor -- start 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- start start 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbbc1019c0 0x7fbbbc198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbbc102bc0 0x7fbbbc199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 0x7fbbbc1a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10d580 con 0x7fbbbc103560 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10d400 con 0x7fbbbc102bc0 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc2aed700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10d700 con 0x7fbbbc1019c0 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.548+0000 7fbbc108a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 0x7fbbbc1a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 0x7fbbbc1a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44730/0 (socket says 172.21.15.57:44730) 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 -- 172.21.15.57:0/1833470144 learned_addr learned my addr 172.21.15.57:0/1833470144 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:07.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbbbfff700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbbc102bc0 0x7fbbbc199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc0889700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbbc1019c0 0x7fbbbc198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 -- 172.21.15.57:0/1833470144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbbc1019c0 msgr2=0x7fbbbc198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbbc1019c0 0x7fbbbc198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 -- 172.21.15.57:0/1833470144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbbc102bc0 msgr2=0x7fbbbc199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbbc102bc0 0x7fbbbc199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 -- 172.21.15.57:0/1833470144 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc1a0960 con 0x7fbbbc103560 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc108a700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 0x7fbbbc1a0290 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400b970 tx=0x7fbbb400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb400d860 con 0x7fbbbc103560 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc1a0c50 con 0x7fbbbc103560 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbb400e430 con 0x7fbbbc103560 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.549+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4017660 con 0x7fbbbc103560 2024-04-21T08:45:07.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.550+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1a1160 con 0x7fbbbc103560 2024-04-21T08:45:07.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.550+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc069730 con 0x7fbbbc103560 2024-04-21T08:45:07.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.552+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbbb400da00 con 0x7fbbbc103560 2024-04-21T08:45:07.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.552+0000 7fbbb9ffb700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbbac0a3c40 0x7fbbac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:07.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.552+0000 7fbbc0889700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbbac0a3c40 0x7fbbac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:07.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.552+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbbb40d0340 con 0x7fbbbc103560 2024-04-21T08:45:07.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.553+0000 7fbbc0889700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbbac0a3c40 0x7fbbac0a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009f50 tx=0x7fbbb000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:07.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.557+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbbb4013030 con 0x7fbbbc103560 2024-04-21T08:45:07.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.810+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fbbbc1a1af0 con 0x7fbbbc103560 2024-04-21T08:45:07.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.810+0000 7fbbb9ffb700 1 -- 172.21.15.57:0/1833470144 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) ==== 76+0+3016 (secure 0 0 0) 0x7fbbb409c9a0 con 0x7fbbbc103560 2024-04-21T08:45:07.810 INFO:teuthology.orchestra.run.smithi057.stdout:e21 2024-04-21T08:45:07.810 INFO:teuthology.orchestra.run.smithi057.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:legacy client fscid: 1 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:Filesystem 'cephfs' (1) 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:fs_name cephfs 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 21 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:created 2024-04-21T08:42:56.988323+0000 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:modified 2024-04-21T08:43:22.134494+0000 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:tableserver 0 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:root 0 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:session_timeout 300 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:session_autoclose 300 2024-04-21T08:45:07.811 INFO:teuthology.orchestra.run.smithi057.stdout:max_file_size 1099511627776 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:max_xattr_size 65536 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:required_client_features {} 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:last_failure 0 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:last_failure_osd_epoch 0 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:max_mds 1 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:in 0 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:up {0=24503} 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:failed 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:damaged 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:stopped 2024-04-21T08:45:07.812 INFO:teuthology.orchestra.run.smithi057.stdout:data_pools [3,4] 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:metadata_pool 2 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:inline_data disabled 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:balancer 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:bal_rank_mask -1 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:standby_count_wanted 0 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:qdb_cluster leader: 24503 members: 24503 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b{0:24503} state up:active seq 3 addr [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.i{0:24496} state up:standby-replay seq 1 addr [v2:172.21.15.136:6838/4293224778,v1:172.21.15.136:6839/4293224778] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:Standby daemons: 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.57:6834/2211232297,v1:172.21.15.57:6835/2211232297] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.813 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.57:6836/2986531851,v1:172.21.15.57:6837/2986531851] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.g{-1:14613} state up:standby seq 1 addr [v2:172.21.15.57:6838/1393580593,v1:172.21.15.57:6839/1393580593] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a{-1:14625} state up:standby seq 1 addr [v2:172.21.15.57:6840/177564568,v1:172.21.15.57:6841/177564568] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.k{-1:24416} state up:standby seq 1 addr [v2:172.21.15.129:6832/3055010748,v1:172.21.15.129:6833/3055010748] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.l{-1:24421} state up:standby seq 1 addr [v2:172.21.15.136:6832/3199760269,v1:172.21.15.136:6833/3199760269] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.e{-1:24440} state up:standby seq 1 addr [v2:172.21.15.129:6834/2158348740,v1:172.21.15.129:6835/2158348740] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.h{-1:24464} state up:standby seq 1 addr [v2:172.21.15.129:6836/1005091659,v1:172.21.15.129:6837/1005091659] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.f{-1:24466} state up:standby seq 1 addr [v2:172.21.15.136:6834/1669978943,v1:172.21.15.136:6835/1669978943] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.814 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c{-1:24478} state up:standby seq 1 addr [v2:172.21.15.136:6836/1005985303,v1:172.21.15.136:6837/1005985303] compat {c=[1],r=[1],i=[1fff]}] 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbbac0a3c40 msgr2=0x7fbbac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbbac0a3c40 0x7fbbac0a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009f50 tx=0x7fbbb000b540 comp rx=0 tx=0).stop 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 msgr2=0x7fbbbc1a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 0x7fbbbc1a0290 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400b970 tx=0x7fbbb400bc80 comp rx=0 tx=0).stop 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 shutdown_connections 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fbbac0a3c40 0x7fbbac0a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc103560 0x7fbbbc1a0290 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbbbc1019c0 0x7fbbbc198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 --2- 172.21.15.57:0/1833470144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbbc102bc0 0x7fbbbc199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 >> 172.21.15.57:0/1833470144 conn(0x7fbbbc0fcfa0 msgr2=0x7fbbbc0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 shutdown_connections 2024-04-21T08:45:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:07.813+0000 7fbbc2aed700 1 -- 172.21.15.57:0/1833470144 wait complete. 2024-04-21T08:45:07.816 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:08 smithi057 ceph-mon[27968]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:08 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1833470144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:45:08.688 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-21T08:45:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:08 smithi129 ceph-mon[31272]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:08 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1833470144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:45:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:08 smithi136 ceph-mon[31791]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:08 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1833470144' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-21T08:45:08.942 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:10.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:10 smithi057 ceph-mon[27968]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 -- 172.21.15.57:0/2379535765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 msgr2=0x7fa40c10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 --2- 172.21.15.57:0/2379535765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c10ac00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa40400b0a0 tx=0x7fa40400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 -- 172.21.15.57:0/2379535765 shutdown_connections 2024-04-21T08:45:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 --2- 172.21.15.57:0/2379535765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40c103530 0x7fa40c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 --2- 172.21.15.57:0/2379535765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c10ac00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 --2- 172.21.15.57:0/2379535765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa40c105e60 0x7fa40c108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 -- 172.21.15.57:0/2379535765 >> 172.21.15.57:0/2379535765 conn(0x7fa40c0fcf40 msgr2=0x7fa40c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 -- 172.21.15.57:0/2379535765 shutdown_connections 2024-04-21T08:45:10.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 -- 172.21.15.57:0/2379535765 wait complete. 2024-04-21T08:45:10.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.714+0000 7fa412278700 1 Processor -- start 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 -- start start 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40c103530 0x7fa40c102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa40c105e60 0x7fa40c1014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c10e500 con 0x7fa40c108790 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c10e380 con 0x7fa40c103530 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa412278700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa40c10e680 con 0x7fa40c105e60 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44748/0 (socket says 172.21.15.57:44748) 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa40b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa40c105e60 0x7fa40c1014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 -- 172.21.15.57:0/3932851405 learned_addr learned my addr 172.21.15.57:0/3932851405 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa40bfff700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40c103530 0x7fa40c102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:10.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 -- 172.21.15.57:0/3932851405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa40c105e60 msgr2=0x7fa40c1014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa40c105e60 0x7fa40c1014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 -- 172.21.15.57:0/3932851405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40c103530 msgr2=0x7fa40c102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40c103530 0x7fa40c102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.715+0000 7fa410815700 1 -- 172.21.15.57:0/3932851405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40c1022d0 con 0x7fa40c108790 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.716+0000 7fa410815700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c101a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa4040062a0 tx=0x7fa404004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.716+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404021070 con 0x7fa40c108790 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.716+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa404003c00 con 0x7fa40c108790 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.716+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40c1a2c50 con 0x7fa40c108790 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.716+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa404004820 con 0x7fa40c108790 2024-04-21T08:45:10.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.716+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40c1a30b0 con 0x7fa40c108790 2024-04-21T08:45:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.717+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40c069730 con 0x7fa40c108790 2024-04-21T08:45:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.718+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa404003db0 con 0x7fa40c108790 2024-04-21T08:45:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.718+0000 7fa4097fa700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f40a3c40 0x7fa3f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.719+0000 7fa40bfff700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f40a3c40 0x7fa3f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:10.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.719+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa40400f020 con 0x7fa40c108790 2024-04-21T08:45:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.719+0000 7fa40bfff700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f40a3c40 0x7fa3f40a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc005950 tx=0x7fa3fc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:10.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.722+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa40402a020 con 0x7fa40c108790 2024-04-21T08:45:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:10 smithi129 ceph-mon[31272]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:10.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:10 smithi136 ceph-mon[31791]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:10.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.930+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fa40c02df80 con 0x7fa40c108790 2024-04-21T08:45:10.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.931+0000 7fa4097fa700 1 -- 172.21.15.57:0/3932851405 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5625 (secure 0 0 0) 0x7fa40409e7b0 con 0x7fa40c108790 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 93 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:created 2024-04-21T08:34:49.550181+0000 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:modified 2024-04-21T08:44:47.297032+0000 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:crush_version 27 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:full_ratio 0.95 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:backfillfull_ratio 0.9 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:nearfull_ratio 0.85 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:require_min_compat_client luminous 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:min_compat_client luminous 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:require_osd_release squid 2024-04-21T08:45:10.931 INFO:teuthology.orchestra.run.smithi057.stdout:stretch_mode_enabled false 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:max_osd 12 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.57:6802/1351441903,v1:172.21.15.57:6803/1351441903] [v2:172.21.15.57:6804/1351441903,v1:172.21.15.57:6805/1351441903] exists,up 773b7403-49ea-4952-8d1e-d30974987af6 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6800/3548590037,v1:172.21.15.129:6801/3548590037] [v2:172.21.15.129:6802/3548590037,v1:172.21.15.129:6803/3548590037] exists,up ee1d7ace-7312-4242-aa99-11fcad20dd62 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/1481395579,v1:172.21.15.136:6801/1481395579] [v2:172.21.15.136:6802/1481395579,v1:172.21.15.136:6803/1481395579] exists,up 1e30fb84-6e42-4857-9e01-fecda24f1074 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.57:6810/3492557165,v1:172.21.15.57:6811/3492557165] [v2:172.21.15.57:6812/3492557165,v1:172.21.15.57:6813/3492557165] exists,up bcf84be5-1210-4e75-86ed-4aac7a0d4dd4 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6808/757493638,v1:172.21.15.129:6809/757493638] [v2:172.21.15.129:6810/757493638,v1:172.21.15.129:6811/757493638] exists,up d2797f86-8e4f-4d66-ba17-dcd90bb6869e 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6808/564097587,v1:172.21.15.136:6809/564097587] [v2:172.21.15.136:6810/564097587,v1:172.21.15.136:6811/564097587] exists,up f4b316af-e19a-41f1-b552-3e6aa0310bb8 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.57:6818/961629147,v1:172.21.15.57:6819/961629147] [v2:172.21.15.57:6820/961629147,v1:172.21.15.57:6821/961629147] exists,up 09e7ec78-6803-43af-9a9d-234a5c7d4f4f 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6816/1910048774,v1:172.21.15.129:6817/1910048774] [v2:172.21.15.129:6818/1910048774,v1:172.21.15.129:6819/1910048774] exists,up 6d8a8522-645a-4a69-a6c2-d38b04198b36 2024-04-21T08:45:10.932 INFO:teuthology.orchestra.run.smithi057.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/94445430,v1:172.21.15.136:6817/94445430] [v2:172.21.15.136:6818/94445430,v1:172.21.15.136:6819/94445430] exists,up 5a0b1a0a-5937-4691-ac89-25c328fabf77 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.57:6826/4116200547,v1:172.21.15.57:6827/4116200547] [v2:172.21.15.57:6828/4116200547,v1:172.21.15.57:6829/4116200547] exists,up ed5fefc0-93b7-4cb0-9b64-ccd879d2f230 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6824/1559005411,v1:172.21.15.129:6825/1559005411] [v2:172.21.15.129:6826/1559005411,v1:172.21.15.129:6827/1559005411] exists,up 4319594b-8e0c-4120-a5d9-40c9e1536cac 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/4048591529,v1:172.21.15.136:6825/4048591529] [v2:172.21.15.136:6826/4048591529,v1:172.21.15.136:6827/4048591529] exists,up b88b4078-cb27-4bfd-a17a-bf0a05dcc291 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 2.b [8,11] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 2.13 [2,11] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 2.1b [2,4] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 3.e [4,8] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 3.10 [5,9] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 3.21 [4,6] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 3.24 [4,9] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 3.38 [3,9] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 4.e [9,11] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 4.f [9,8] 2024-04-21T08:45:10.933 INFO:teuthology.orchestra.run.smithi057.stdout:pg_upmap_items 4.1b [9,8,5,10] 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:0/3363889148 expires 2024-04-22T08:35:44.517439+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:0/316357528 expires 2024-04-22T08:35:44.517439+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:6801/2928520021 expires 2024-04-22T08:35:44.517439+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:6800/2928520021 expires 2024-04-22T08:35:44.517439+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:0/2818769380 expires 2024-04-22T08:35:17.446046+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:0/898712319 expires 2024-04-22T08:35:17.446046+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:0/2910544903 expires 2024-04-22T08:35:17.446046+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:6800/3383203302 expires 2024-04-22T08:35:17.446046+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:0/3448624546 expires 2024-04-22T08:35:44.517439+0000 2024-04-21T08:45:10.934 INFO:teuthology.orchestra.run.smithi057.stdout:blocklist 172.21.15.57:6801/3383203302 expires 2024-04-22T08:35:17.446046+0000 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f40a3c40 msgr2=0x7fa3f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f40a3c40 0x7fa3f40a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc005950 tx=0x7fa3fc00b410 comp rx=0 tx=0).stop 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 msgr2=0x7fa40c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c101a20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa4040062a0 tx=0x7fa404004e40 comp rx=0 tx=0).stop 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 shutdown_connections 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa3f40a3c40 0x7fa3f40a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa40c103530 0x7fa40c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa40c108790 0x7fa40c101a20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 --2- 172.21.15.57:0/3932851405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa40c105e60 0x7fa40c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.933+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 >> 172.21.15.57:0/3932851405 conn(0x7fa40c0fcf40 msgr2=0x7fa40c103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.934+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 shutdown_connections 2024-04-21T08:45:10.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:10.934+0000 7fa412278700 1 -- 172.21.15.57:0/3932851405 wait complete. 2024-04-21T08:45:11.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:11 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3932851405' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:45:11.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:11 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3932851405' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:45:11.878 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-21T08:45:11.890 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:11.890 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:11 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3932851405' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 -- 172.21.15.57:0/3232444641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede0101140 msgr2=0x7fede01015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 --2- 172.21.15.57:0/3232444641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede0101140 0x7fede01015a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009900 tx=0x7fedd002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 -- 172.21.15.57:0/3232444641 shutdown_connections 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 --2- 172.21.15.57:0/3232444641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101ae0 0x7fede0101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 --2- 172.21.15.57:0/3232444641 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede0101140 0x7fede01015a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 --2- 172.21.15.57:0/3232444641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede01007e0 0x7fede0100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 -- 172.21.15.57:0/3232444641 >> 172.21.15.57:0/3232444641 conn(0x7fede00fc3e0 msgr2=0x7fede00fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 -- 172.21.15.57:0/3232444641 shutdown_connections 2024-04-21T08:45:12.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.001+0000 7fede7a4a640 1 -- 172.21.15.57:0/3232444641 wait complete. 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 Processor -- start 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 -- start start 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 0x7fede019a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101140 0x7fede019a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede0101ae0 0x7fede01a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fede010fcc0 con 0x7fede01007e0 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fede010fb40 con 0x7fede0101140 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede7a4a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fede010fe40 con 0x7fede0101ae0 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 0x7fede019a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 0x7fede019a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44772/0 (socket says 172.21.15.57:44772) 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 -- 172.21.15.57:0/2375833499 learned_addr learned my addr 172.21.15.57:0/2375833499 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede5fe7640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede0101ae0 0x7fede01a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede4fe5640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101140 0x7fede019a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 -- 172.21.15.57:0/2375833499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede0101ae0 msgr2=0x7fede01a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede0101ae0 0x7fede01a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 -- 172.21.15.57:0/2375833499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101140 msgr2=0x7fede019a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101140 0x7fede019a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 -- 172.21.15.57:0/2375833499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede01a1fb0 con 0x7fede01007e0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede4fe5640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101140 0x7fede019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.002+0000 7fede57e6640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 0x7fede019a390 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7feddc00d950 tx=0x7feddc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.003+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc017070 con 0x7fede01007e0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.003+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede01a22a0 con 0x7fede01007e0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.003+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc009c30 con 0x7fede01007e0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.003+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc00f620 con 0x7fede01007e0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.003+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede01a27f0 con 0x7fede01007e0 2024-04-21T08:45:12.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.004+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede010b7b0 con 0x7fede01007e0 2024-04-21T08:45:12.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.005+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7feddc010460 con 0x7fede01007e0 2024-04-21T08:45:12.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.005+0000 7fedd67fc640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fedcc0a3bb0 0x7fedcc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.005+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7feddc0d0460 con 0x7fede01007e0 2024-04-21T08:45:12.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.005+0000 7fede4fe5640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fedcc0a3bb0 0x7fedcc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.006+0000 7fede4fe5640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fedcc0a3bb0 0x7fedcc0a6070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fedd00096c0 tx=0x7fedd0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.009 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.010+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feddc09cac0 con 0x7fede01007e0 2024-04-21T08:45:12.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.196+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fede0062b50 con 0x7fede01007e0 2024-04-21T08:45:12.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.198+0000 7fedd67fc640 1 -- 172.21.15.57:0/2375833499 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7feddc009de0 con 0x7fede01007e0 2024-04-21T08:45:12.198 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:12.198 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fedcc0a3bb0 msgr2=0x7fedcc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fedcc0a3bb0 0x7fedcc0a6070 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fedd00096c0 tx=0x7fedd0000bc0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 msgr2=0x7fede019a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 0x7fede019a390 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7feddc00d950 tx=0x7feddc00de10 comp rx=0 tx=0).stop 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 shutdown_connections 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fedcc0a3bb0 0x7fedcc0a6070 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede0101ae0 0x7fede01a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fede0101140 0x7fede019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 --2- 172.21.15.57:0/2375833499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fede01007e0 0x7fede019a390 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 >> 172.21.15.57:0/2375833499 conn(0x7fede00fc3e0 msgr2=0x7fede00fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 shutdown_connections 2024-04-21T08:45:12.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.200+0000 7fede7a4a640 1 -- 172.21.15.57:0/2375833499 wait complete. 2024-04-21T08:45:12.211 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:12.212 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:12.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 -- 172.21.15.57:0/82737699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c102a10 msgr2=0x7f433c104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 --2- 172.21.15.57:0/82737699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c102a10 0x7f433c104e00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4330009900 tx=0x7f433002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 -- 172.21.15.57:0/82737699 shutdown_connections 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 --2- 172.21.15.57:0/82737699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c107c70 0x7f433c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 --2- 172.21.15.57:0/82737699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 0x7f433c107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 --2- 172.21.15.57:0/82737699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c102a10 0x7f433c104e00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 -- 172.21.15.57:0/82737699 >> 172.21.15.57:0/82737699 conn(0x7f433c0fc400 msgr2=0x7f433c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 -- 172.21.15.57:0/82737699 shutdown_connections 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.318+0000 7f4342aba640 1 -- 172.21.15.57:0/82737699 wait complete. 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 Processor -- start 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 -- start start 2024-04-21T08:45:12.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c102a10 0x7f433c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 0x7f433c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 0x7f433c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f433c10db20 con 0x7f433c107c70 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f433c10d9a0 con 0x7f433c102a10 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4342aba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f433c10dca0 con 0x7f433c105340 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 0x7f433c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 0x7f433c19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44776/0 (socket says 172.21.15.57:44776) 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 -- 172.21.15.57:0/283674361 learned_addr learned my addr 172.21.15.57:0/283674361 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f433bfff640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 0x7f433c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4340856640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c102a10 0x7f433c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 -- 172.21.15.57:0/283674361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 msgr2=0x7f433c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 0x7f433c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 -- 172.21.15.57:0/283674361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c102a10 msgr2=0x7f433c198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c102a10 0x7f433c198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 -- 172.21.15.57:0/283674361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433c19fe70 con 0x7f433c107c70 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4340856640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c102a10 0x7f433c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f433bfff640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 0x7f433c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.319+0000 7f4341057640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 0x7f433c19f800 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f433400cad0 tx=0x7f433400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.320+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433400ae20 con 0x7f433c107c70 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.320+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4334005020 con 0x7f433c107c70 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.320+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f433c1a0160 con 0x7f433c107c70 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.320+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f433400f6b0 con 0x7f433c107c70 2024-04-21T08:45:12.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.320+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f433c1a06b0 con 0x7f433c107c70 2024-04-21T08:45:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.321+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f433c0fdef0 con 0x7f433c107c70 2024-04-21T08:45:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.322+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f43340051d0 con 0x7f433c107c70 2024-04-21T08:45:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.323+0000 7f4339ffb640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f432c0a3c30 0x7f432c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.323+0000 7f4340856640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f432c0a3c30 0x7f432c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.323+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f43340d08d0 con 0x7f433c107c70 2024-04-21T08:45:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.323+0000 7f4340856640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f432c0a3c30 0x7f432c0a60f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4330036040 tx=0x7f4330005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.327+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4334014030 con 0x7f433c107c70 2024-04-21T08:45:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.517+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f433c062b50 con 0x7f433c107c70 2024-04-21T08:45:12.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.518+0000 7f4339ffb640 1 -- 172.21.15.57:0/283674361 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f433409ced0 con 0x7f433c107c70 2024-04-21T08:45:12.517 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f432c0a3c30 msgr2=0x7f432c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f432c0a3c30 0x7f432c0a60f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4330036040 tx=0x7f4330005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 msgr2=0x7f433c19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 0x7f433c19f800 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f433400cad0 tx=0x7f433400cf90 comp rx=0 tx=0).stop 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 shutdown_connections 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f432c0a3c30 0x7f432c0a60f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f433c107c70 0x7f433c19f800 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f433c105340 0x7f433c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 --2- 172.21.15.57:0/283674361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f433c102a10 0x7f433c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 >> 172.21.15.57:0/283674361 conn(0x7f433c0fc400 msgr2=0x7f433c103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 shutdown_connections 2024-04-21T08:45:12.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.520+0000 7f4342aba640 1 -- 172.21.15.57:0/283674361 wait complete. 2024-04-21T08:45:12.530 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:12 smithi057 ceph-mon[27968]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:12.530 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2375833499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:12.531 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:12.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 -- 172.21.15.57:0/2737461301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 msgr2=0x7f94cc1077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 --2- 172.21.15.57:0/2737461301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc1077b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f94bc009900 tx=0x7f94bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 -- 172.21.15.57:0/2737461301 shutdown_connections 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 --2- 172.21.15.57:0/2737461301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc107cf0 0x7f94cc10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 --2- 172.21.15.57:0/2737461301 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc1077b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 --2- 172.21.15.57:0/2737461301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94cc102a90 0x7f94cc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 -- 172.21.15.57:0/2737461301 >> 172.21.15.57:0/2737461301 conn(0x7f94cc0fc400 msgr2=0x7f94cc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 -- 172.21.15.57:0/2737461301 shutdown_connections 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 -- 172.21.15.57:0/2737461301 wait complete. 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 Processor -- start 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.637+0000 7f94d354a640 1 -- start start 2024-04-21T08:45:12.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d354a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc102a90 0x7f94cc102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d354a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d354a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94cc107cf0 0x7f94cc100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d354a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc10dac0 con 0x7f94cc1053c0 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d354a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc10d940 con 0x7f94cc102a90 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d354a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc10dc40 con 0x7f94cc107cf0 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d0ae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d0ae5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44794/0 (socket says 172.21.15.57:44794) 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d0ae5640 1 -- 172.21.15.57:0/3779711511 learned_addr learned my addr 172.21.15.57:0/3779711511 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc102a90 0x7f94cc102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d1ae7640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94cc107cf0 0x7f94cc100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 -- 172.21.15.57:0/3779711511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94cc107cf0 msgr2=0x7f94cc100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94cc107cf0 0x7f94cc100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 -- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 msgr2=0x7f94cc100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 -- 172.21.15.57:0/3779711511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94cc101810 con 0x7f94cc102a90 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.638+0000 7f94d12e6640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc102a90 0x7f94cc102400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94c800d720 tx=0x7f94c800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.639+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c8017070 con 0x7f94cc102a90 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.639+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94cc1a2380 con 0x7f94cc102a90 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.639+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94c800de30 con 0x7f94cc102a90 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.639+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c80047a0 con 0x7f94cc102a90 2024-04-21T08:45:12.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.639+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94cc1a28d0 con 0x7f94cc102a90 2024-04-21T08:45:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.640+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94cc0fdef0 con 0x7f94cc102a90 2024-04-21T08:45:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.641+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f94c80099d0 con 0x7f94cc102a90 2024-04-21T08:45:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.642+0000 7f94c27fc640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f94b80a3bb0 0x7f94b80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.642+0000 7f94d0ae5640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f94b80a3bb0 0x7f94b80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.642+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f94c80d0ad0 con 0x7f94cc102a90 2024-04-21T08:45:12.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.642+0000 7f94d0ae5640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f94b80a3bb0 0x7f94b80a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f94cc102170 tx=0x7f94bc0095a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.646+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94c8020020 con 0x7f94cc102a90 2024-04-21T08:45:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:12 smithi129 ceph-mon[31272]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:12 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2375833499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:12.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.833+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f94cc02d5c0 con 0x7f94cc102a90 2024-04-21T08:45:12.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.834+0000 7f94c27fc640 1 -- 172.21.15.57:0/3779711511 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f94c809d0d0 con 0x7f94cc102a90 2024-04-21T08:45:12.834 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f94b80a3bb0 msgr2=0x7f94b80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f94b80a3bb0 0x7f94b80a6070 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f94cc102170 tx=0x7f94bc0095a0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc102a90 msgr2=0x7f94cc102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc102a90 0x7f94cc102400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94c800d720 tx=0x7f94c800dbe0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 shutdown_connections 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f94b80a3bb0 0x7f94b80a6070 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94cc107cf0 0x7f94cc100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f94cc1053c0 0x7f94cc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 --2- 172.21.15.57:0/3779711511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94cc102a90 0x7f94cc102400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 >> 172.21.15.57:0/3779711511 conn(0x7f94cc0fc400 msgr2=0x7f94cc104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 shutdown_connections 2024-04-21T08:45:12.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.836+0000 7f94d354a640 1 -- 172.21.15.57:0/3779711511 wait complete. 2024-04-21T08:45:12.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:12 smithi136 ceph-mon[31791]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:12 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2375833499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 -- 172.21.15.57:0/2023845865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 msgr2=0x7f6bc0107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2023845865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc0107750 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6bac009900 tx=0x7f6bac02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 -- 172.21.15.57:0/2023845865 shutdown_connections 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2023845865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0107c90 0x7f6bc010a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2023845865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc0107750 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2023845865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bc0102a30 0x7f6bc0104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 -- 172.21.15.57:0/2023845865 >> 172.21.15.57:0/2023845865 conn(0x7f6bc00fc440 msgr2=0x7f6bc00fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 -- 172.21.15.57:0/2023845865 shutdown_connections 2024-04-21T08:45:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.952+0000 7f6bc7031640 1 -- 172.21.15.57:0/2023845865 wait complete. 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 Processor -- start 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 -- start start 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bc0102a30 0x7f6bc01982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc01987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0107c90 0x7f6bc019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc010db30 con 0x7f6bc0105360 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc010d9b0 con 0x7f6bc0107c90 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc7031640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc010dcb0 con 0x7f6bc0102a30 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc01987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc01987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45886/0 (socket says 172.21.15.57:45886) 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 -- 172.21.15.57:0/2926545171 learned_addr learned my addr 172.21.15.57:0/2926545171 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc4dcd640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bc0102a30 0x7f6bc01982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bc55ce640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0107c90 0x7f6bc019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 -- 172.21.15.57:0/2926545171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bc0102a30 msgr2=0x7f6bc01982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bc0102a30 0x7f6bc01982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 -- 172.21.15.57:0/2926545171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0107c90 msgr2=0x7f6bc019f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0107c90 0x7f6bc019f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.953+0000 7f6bbffff640 1 -- 172.21.15.57:0/2926545171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc019fec0 con 0x7f6bc0105360 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.954+0000 7f6bbffff640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc01987e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6bac000c00 tx=0x7f6bac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.954+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac043020 con 0x7f6bc0105360 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.954+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bac004aa0 con 0x7f6bc0105360 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.954+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac003bf0 con 0x7f6bc0105360 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.954+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc01a0150 con 0x7f6bc0105360 2024-04-21T08:45:12.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.954+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc01a0770 con 0x7f6bc0105360 2024-04-21T08:45:12.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.955+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc00fdf30 con 0x7f6bc0105360 2024-04-21T08:45:12.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.956+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6bac046020 con 0x7f6bc0105360 2024-04-21T08:45:12.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.957+0000 7f6bbdffb640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6bb00a3c30 0x7f6bb00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:12.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.957+0000 7f6bc4dcd640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6bb00a3c30 0x7f6bb00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:12.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.957+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f6bac0f2a90 con 0x7f6bc0105360 2024-04-21T08:45:12.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.957+0000 7f6bc4dcd640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6bb00a3c30 0x7f6bb00a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009990 tx=0x7f6bb4009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:12.960 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:12.961+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bac04d020 con 0x7f6bc0105360 2024-04-21T08:45:13.144 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.145+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6bc0062b50 con 0x7f6bc0105360 2024-04-21T08:45:13.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.146+0000 7f6bbdffb640 1 -- 172.21.15.57:0/2926545171 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6bac03c350 con 0x7f6bc0105360 2024-04-21T08:45:13.145 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6bb00a3c30 msgr2=0x7f6bb00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6bb00a3c30 0x7f6bb00a60f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009990 tx=0x7f6bb4009110 comp rx=0 tx=0).stop 2024-04-21T08:45:13.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 msgr2=0x7f6bc01987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc01987e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6bac000c00 tx=0x7f6bac004270 comp rx=0 tx=0).stop 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 shutdown_connections 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f6bb00a3c30 0x7f6bb00a60f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bc0107c90 0x7f6bc019f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6bc0105360 0x7f6bc01987e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 --2- 172.21.15.57:0/2926545171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bc0102a30 0x7f6bc01982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 >> 172.21.15.57:0/2926545171 conn(0x7f6bc00fc440 msgr2=0x7f6bc01035b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 shutdown_connections 2024-04-21T08:45:13.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.148+0000 7f6bc7031640 1 -- 172.21.15.57:0/2926545171 wait complete. 2024-04-21T08:45:13.158 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 -- 172.21.15.57:0/1280198999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48100e90 msgr2=0x7f9d481012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/1280198999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48100e90 0x7f9d481012b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d3802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 -- 172.21.15.57:0/1280198999 shutdown_connections 2024-04-21T08:45:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/1280198999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d481029a0 0x7f9d481092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/1280198999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d48101fe0 0x7f9d48102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/1280198999 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48100e90 0x7f9d481012b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 -- 172.21.15.57:0/1280198999 >> 172.21.15.57:0/1280198999 conn(0x7f9d480fc400 msgr2=0x7f9d480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 -- 172.21.15.57:0/1280198999 shutdown_connections 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 -- 172.21.15.57:0/1280198999 wait complete. 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 Processor -- start 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.264+0000 7f9d4d83f640 1 -- start start 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d48100e90 0x7f9d48069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 0x7f9d4819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d481029a0 0x7f9d4819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4810c9c0 con 0x7f9d48101fe0 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4810c840 con 0x7f9d481029a0 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4810cb40 con 0x7f9d48100e90 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 0x7f9d4819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 0x7f9d4819cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45900/0 (socket says 172.21.15.57:45900) 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 -- 172.21.15.57:0/2803934462 learned_addr learned my addr 172.21.15.57:0/2803934462 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d477fe640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d481029a0 0x7f9d4819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d46ffd640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d48100e90 0x7f9d48069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 -- 172.21.15.57:0/2803934462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d48100e90 msgr2=0x7f9d48069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d48100e90 0x7f9d48069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 -- 172.21.15.57:0/2803934462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d481029a0 msgr2=0x7f9d4819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d481029a0 0x7f9d4819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 -- 172.21.15.57:0/2803934462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4819fea0 con 0x7f9d48101fe0 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d46ffd640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d48100e90 0x7f9d48069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:13.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d467fc640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 0x7f9d4819cf70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00d770 tx=0x7f9d3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:13.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c017070 con 0x7f9d48101fe0 2024-04-21T08:45:13.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3c009c30 con 0x7f9d48101fe0 2024-04-21T08:45:13.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d481a0130 con 0x7f9d48101fe0 2024-04-21T08:45:13.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c003cc0 con 0x7f9d48101fe0 2024-04-21T08:45:13.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.265+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d481a0680 con 0x7f9d48101fe0 2024-04-21T08:45:13.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.266+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d481082e0 con 0x7f9d48101fe0 2024-04-21T08:45:13.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.267+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9d3c003e60 con 0x7f9d48101fe0 2024-04-21T08:45:13.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.268+0000 7f9d4c83d640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9d300a3bb0 0x7f9d300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.268+0000 7f9d46ffd640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9d300a3bb0 0x7f9d300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.268+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f9d3c0d0110 con 0x7f9d48101fe0 2024-04-21T08:45:13.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.269+0000 7f9d46ffd640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9d300a3bb0 0x7f9d300a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d38006010 tx=0x7f9d3802c7e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:13.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.273+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d3c09c770 con 0x7f9d48101fe0 2024-04-21T08:45:13.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.465+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9d48062b50 con 0x7f9d48101fe0 2024-04-21T08:45:13.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.466+0000 7f9d4c83d640 1 -- 172.21.15.57:0/2803934462 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f9d3c009de0 con 0x7f9d48101fe0 2024-04-21T08:45:13.465 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:13.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.467+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9d300a3bb0 msgr2=0x7f9d300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.467+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9d300a3bb0 0x7f9d300a6070 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9d38006010 tx=0x7f9d3802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.467+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 msgr2=0x7f9d4819cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.467+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 0x7f9d4819cf70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00d770 tx=0x7f9d3c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 shutdown_connections 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f9d300a3bb0 0x7f9d300a6070 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d481029a0 0x7f9d4819f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9d48101fe0 0x7f9d4819cf70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 --2- 172.21.15.57:0/2803934462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d48100e90 0x7f9d48069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 >> 172.21.15.57:0/2803934462 conn(0x7f9d480fc400 msgr2=0x7f9d481054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 shutdown_connections 2024-04-21T08:45:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.468+0000 7f9d4d83f640 1 -- 172.21.15.57:0/2803934462 wait complete. 2024-04-21T08:45:13.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/283674361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3779711511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.478 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2926545171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.479 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:13.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 -- 172.21.15.57:0/4126904240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63141053d0 msgr2=0x7f63141077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 --2- 172.21.15.57:0/4126904240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63141053d0 0x7f63141077c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6304009900 tx=0x7f630402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:13.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 -- 172.21.15.57:0/4126904240 shutdown_connections 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 --2- 172.21.15.57:0/4126904240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6314107d00 0x7f631410a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 --2- 172.21.15.57:0/4126904240 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63141053d0 0x7f63141077c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 --2- 172.21.15.57:0/4126904240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6314102aa0 0x7f6314104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 -- 172.21.15.57:0/4126904240 >> 172.21.15.57:0/4126904240 conn(0x7f63140fc470 msgr2=0x7f63140fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 -- 172.21.15.57:0/4126904240 shutdown_connections 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.581+0000 7f631b2d2640 1 -- 172.21.15.57:0/4126904240 wait complete. 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 Processor -- start 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 -- start start 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 0x7f6314198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63141053d0 0x7f63141986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6314107d00 0x7f631419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f631410db00 con 0x7f6314102aa0 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f631410d980 con 0x7f63141053d0 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631b2d2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f631410dc80 con 0x7f6314107d00 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 0x7f6314198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 0x7f6314198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45918/0 (socket says 172.21.15.57:45918) 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 -- 172.21.15.57:0/3530487774 learned_addr learned my addr 172.21.15.57:0/3530487774 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631886d640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63141053d0 0x7f63141986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631986f640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6314107d00 0x7f631419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 -- 172.21.15.57:0/3530487774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6314107d00 msgr2=0x7f631419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6314107d00 0x7f631419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 -- 172.21.15.57:0/3530487774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63141053d0 msgr2=0x7f63141986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63141053d0 0x7f63141986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 -- 172.21.15.57:0/3530487774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631419fdb0 con 0x7f6314102aa0 2024-04-21T08:45:13.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.582+0000 7f631906e640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 0x7f6314198190 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f631000d950 tx=0x7f631000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.583+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310017070 con 0x7f6314102aa0 2024-04-21T08:45:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.583+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6310009c30 con 0x7f6314102aa0 2024-04-21T08:45:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.583+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63141a00a0 con 0x7f6314102aa0 2024-04-21T08:45:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.583+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631000f620 con 0x7f6314102aa0 2024-04-21T08:45:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.583+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63141a05c0 con 0x7f6314102aa0 2024-04-21T08:45:13.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.584+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63140fdf60 con 0x7f6314102aa0 2024-04-21T08:45:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.585+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6310010460 con 0x7f6314102aa0 2024-04-21T08:45:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.586+0000 7f630a7fc640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f63000a3bb0 0x7f63000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.586+0000 7f631886d640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f63000a3bb0 0x7f63000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.586+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f63100d03f0 con 0x7f6314102aa0 2024-04-21T08:45:13.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.586+0000 7f631886d640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f63000a3bb0 0x7f63000a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6304005ab0 tx=0x7f6304005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:13.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.589+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f631009c9f0 con 0x7f6314102aa0 2024-04-21T08:45:13.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.815+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f631402d5c0 con 0x7f6314102aa0 2024-04-21T08:45:13.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.816+0000 7f630a7fc640 1 -- 172.21.15.57:0/3530487774 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f6310009de0 con 0x7f6314102aa0 2024-04-21T08:45:13.816 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:13.816 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:13.816 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:13.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.817+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f63000a3bb0 msgr2=0x7f63000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.817+0000 7f631b2d2640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f63000a3bb0 0x7f63000a6070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6304005ab0 tx=0x7f6304005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.817+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 msgr2=0x7f6314198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.817+0000 7f631b2d2640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 0x7f6314198190 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f631000d950 tx=0x7f631000de10 comp rx=0 tx=0).stop 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 shutdown_connections 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f63000a3bb0 0x7f63000a6070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6314107d00 0x7f631419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63141053d0 0x7f63141986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 --2- 172.21.15.57:0/3530487774 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6314102aa0 0x7f6314198190 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 >> 172.21.15.57:0/3530487774 conn(0x7f63140fc470 msgr2=0x7f63141033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:13.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 shutdown_connections 2024-04-21T08:45:13.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.818+0000 7f631b2d2640 1 -- 172.21.15.57:0/3530487774 wait complete. 2024-04-21T08:45:13.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/283674361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3779711511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2926545171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.829 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:13.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/283674361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3779711511' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2926545171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 -- 172.21.15.57:0/3368506442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8101fa0 msgr2=0x7fa0c81023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 --2- 172.21.15.57:0/3368506442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8101fa0 0x7fa0c81023c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009900 tx=0x7fa0b802bc50 comp rx=0 tx=0).stop 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 -- 172.21.15.57:0/3368506442 shutdown_connections 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 --2- 172.21.15.57:0/3368506442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c8103b50 0x7fa0c810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 --2- 172.21.15.57:0/3368506442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0c8103190 0x7fa0c8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 --2- 172.21.15.57:0/3368506442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8101fa0 0x7fa0c81023c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 -- 172.21.15.57:0/3368506442 >> 172.21.15.57:0/3368506442 conn(0x7fa0c80fd530 msgr2=0x7fa0c80ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 -- 172.21.15.57:0/3368506442 shutdown_connections 2024-04-21T08:45:13.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.931+0000 7fa0cd470640 1 -- 172.21.15.57:0/3368506442 wait complete. 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 Processor -- start 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 -- start start 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c8101fa0 0x7fa0c8198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 0x7fa0c81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0c8103b50 0x7fa0c819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810daf0 con 0x7fa0c8103190 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810d970 con 0x7fa0c8101fa0 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0cd470640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810dc70 con 0x7fa0c8103b50 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0c67fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 0x7fa0c81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0c67fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 0x7fa0c81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45942/0 (socket says 172.21.15.57:45942) 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0c67fc640 1 -- 172.21.15.57:0/191931733 learned_addr learned my addr 172.21.15.57:0/191931733 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0c6ffd640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c8101fa0 0x7fa0c8198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.932+0000 7fa0c67fc640 1 -- 172.21.15.57:0/191931733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0c8103b50 msgr2=0x7fa0c819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0c67fc640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0c8103b50 0x7fa0c819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0c67fc640 1 -- 172.21.15.57:0/191931733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c8101fa0 msgr2=0x7fa0c8198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0c67fc640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c8101fa0 0x7fa0c8198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0c67fc640 1 -- 172.21.15.57:0/191931733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c819fd80 con 0x7fa0c8103190 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0c67fc640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 0x7fa0c81986a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c920 tx=0x7fa0bc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc007c70 con 0x7fa0c8103190 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c81a0070 con 0x7fa0c8103190 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0bc007e10 con 0x7fa0c8103190 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc018820 con 0x7fa0c8103190 2024-04-21T08:45:13.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.933+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c81a0610 con 0x7fa0c8103190 2024-04-21T08:45:13.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.934+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c81091f0 con 0x7fa0c8103190 2024-04-21T08:45:13.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.935+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa0bc00f450 con 0x7fa0c8103190 2024-04-21T08:45:13.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.936+0000 7fa0affff640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0b00a3bb0 0x7fa0b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:13.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.936+0000 7fa0c6ffd640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0b00a3bb0 0x7fa0b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:13.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.936+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa0bc0d09a0 con 0x7fa0c8103190 2024-04-21T08:45:13.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.936+0000 7fa0c6ffd640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0b00a3bb0 0x7fa0b00a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80096c0 tx=0x7fa0b8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:13.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:13.940+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0bc014030 con 0x7fa0c8103190 2024-04-21T08:45:14.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.126+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa0c8062b50 con 0x7fa0c8103190 2024-04-21T08:45:14.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.128+0000 7fa0affff640 1 -- 172.21.15.57:0/191931733 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7fa0bc09cfa0 con 0x7fa0c8103190 2024-04-21T08:45:14.127 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:14.128 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:14.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0b00a3bb0 msgr2=0x7fa0b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0b00a3bb0 0x7fa0b00a6070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80096c0 tx=0x7fa0b8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 msgr2=0x7fa0c81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 0x7fa0c81986a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00c920 tx=0x7fa0bc00cde0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 shutdown_connections 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa0b00a3bb0 0x7fa0b00a6070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0c8103b50 0x7fa0c819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0c8103190 0x7fa0c81986a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 --2- 172.21.15.57:0/191931733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0c8101fa0 0x7fa0c8198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 >> 172.21.15.57:0/191931733 conn(0x7fa0c80fd530 msgr2=0x7fa0c81063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 shutdown_connections 2024-04-21T08:45:14.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.130+0000 7fa0cd470640 1 -- 172.21.15.57:0/191931733 wait complete. 2024-04-21T08:45:14.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:14.142 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:14.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 -- 172.21.15.57:0/390121481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 msgr2=0x7fa2d41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/390121481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d41092c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc009ca0 tx=0x7fa2cc02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 -- 172.21.15.57:0/390121481 shutdown_connections 2024-04-21T08:45:14.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/390121481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d41092c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/390121481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4101fe0 0x7fa2d4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/390121481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4100e90 0x7fa2d41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 -- 172.21.15.57:0/390121481 >> 172.21.15.57:0/390121481 conn(0x7fa2d40fc400 msgr2=0x7fa2d40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 -- 172.21.15.57:0/390121481 shutdown_connections 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 -- 172.21.15.57:0/390121481 wait complete. 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.248+0000 7fa2d9c43640 1 Processor -- start 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 -- start start 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4100e90 0x7fa2d4198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4101fe0 0x7fa2d4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d410ca60 con 0x7fa2d41029a0 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d410c8e0 con 0x7fa2d4101fe0 2024-04-21T08:45:14.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d9c43640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d410cbe0 con 0x7fa2d4100e90 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45950/0 (socket says 172.21.15.57:45950) 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 -- 172.21.15.57:0/3367645042 learned_addr learned my addr 172.21.15.57:0/3367645042 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d2ffd640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4101fe0 0x7fa2d4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d37fe640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4100e90 0x7fa2d4198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 -- 172.21.15.57:0/3367645042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4100e90 msgr2=0x7fa2d4198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4100e90 0x7fa2d4198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 -- 172.21.15.57:0/3367645042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4101fe0 msgr2=0x7fa2d4198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4101fe0 0x7fa2d4198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 -- 172.21.15.57:0/3367645042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d419fe70 con 0x7fa2d41029a0 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d37fe640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4100e90 0x7fa2d4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:14.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d2ffd640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4101fe0 0x7fa2d4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.249+0000 7fa2d3fff640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d419f800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc02c700 tx=0x7fa2cc034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.250+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc039040 con 0x7fa2d41029a0 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.250+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2cc044070 con 0x7fa2d41029a0 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.250+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d41a0100 con 0x7fa2d41029a0 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.250+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc0042a0 con 0x7fa2d41029a0 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.250+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d41a0640 con 0x7fa2d41029a0 2024-04-21T08:45:14.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.251+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d41a0960 con 0x7fa2d41029a0 2024-04-21T08:45:14.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.252+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa2cc003970 con 0x7fa2d41029a0 2024-04-21T08:45:14.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.252+0000 7fa2d0ff9640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa2bc0a3c30 0x7fa2bc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.253+0000 7fa2d37fe640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa2bc0a3c30 0x7fa2bc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.253+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa2cc0f29b0 con 0x7fa2d41029a0 2024-04-21T08:45:14.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.253+0000 7fa2d37fe640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa2bc0a3c30 0x7fa2bc0a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009b40 tx=0x7fa2c4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.257+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2cc04e020 con 0x7fa2d41029a0 2024-04-21T08:45:14.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.444+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa2d41a0cf0 con 0x7fa2d41029a0 2024-04-21T08:45:14.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.444+0000 7fa2d0ff9640 1 -- 172.21.15.57:0/3367645042 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fa2cc0bf010 con 0x7fa2d41029a0 2024-04-21T08:45:14.444 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa2bc0a3c30 msgr2=0x7fa2bc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa2bc0a3c30 0x7fa2bc0a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009b40 tx=0x7fa2c4005e50 comp rx=0 tx=0).stop 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 msgr2=0x7fa2d419f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d419f800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc02c700 tx=0x7fa2cc034760 comp rx=0 tx=0).stop 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 shutdown_connections 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fa2bc0a3c30 0x7fa2bc0a60f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2d41029a0 0x7fa2d419f800 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa2d4101fe0 0x7fa2d4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 --2- 172.21.15.57:0/3367645042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4100e90 0x7fa2d4198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 >> 172.21.15.57:0/3367645042 conn(0x7fa2d40fc400 msgr2=0x7fa2d40fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 shutdown_connections 2024-04-21T08:45:14.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.446+0000 7fa2d9c43640 1 -- 172.21.15.57:0/3367645042 wait complete. 2024-04-21T08:45:14.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:14.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 -- 172.21.15.57:0/2118278200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c101fa0 msgr2=0x7f3c0c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 --2- 172.21.15.57:0/2118278200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c1023c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009900 tx=0x7f3c0802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 -- 172.21.15.57:0/2118278200 shutdown_connections 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 --2- 172.21.15.57:0/2118278200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103b50 0x7f3c0c10a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 --2- 172.21.15.57:0/2118278200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c103190 0x7f3c0c103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 --2- 172.21.15.57:0/2118278200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c1023c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 -- 172.21.15.57:0/2118278200 >> 172.21.15.57:0/2118278200 conn(0x7f3c0c0fd530 msgr2=0x7f3c0c0ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 -- 172.21.15.57:0/2118278200 shutdown_connections 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.563+0000 7f3c12d52640 1 -- 172.21.15.57:0/2118278200 wait complete. 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 Processor -- start 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 -- start start 2024-04-21T08:45:14.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103190 0x7f3c0c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 0x7f3c0c19f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c10db70 con 0x7f3c0c103b50 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c10d9f0 con 0x7f3c0c101fa0 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c12d52640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c10dcf0 con 0x7f3c0c103190 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 0x7f3c0c19f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 0x7f3c0c19f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45964/0 (socket says 172.21.15.57:45964) 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 -- 172.21.15.57:0/544734605 learned_addr learned my addr 172.21.15.57:0/544734605 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c03fff640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103190 0x7f3c0c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c10aee640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 -- 172.21.15.57:0/544734605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103190 msgr2=0x7f3c0c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103190 0x7f3c0c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 -- 172.21.15.57:0/544734605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c101fa0 msgr2=0x7f3c0c198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c112ef640 1 -- 172.21.15.57:0/544734605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c19fe70 con 0x7f3c0c103b50 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c03fff640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103190 0x7f3c0c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.564+0000 7f3c10aee640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.565+0000 7f3c112ef640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 0x7f3c0c19f800 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400cce0 tx=0x7f3c0400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.565+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400f450 con 0x7f3c0c103b50 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.565+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0400faa0 con 0x7f3c0c103b50 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.565+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0c1a0160 con 0x7f3c0c103b50 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.565+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04018b70 con 0x7f3c0c103b50 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.565+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0c1a0730 con 0x7f3c0c103b50 2024-04-21T08:45:14.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.566+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c0c109340 con 0x7f3c0c103b50 2024-04-21T08:45:14.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.567+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3c04018d10 con 0x7f3c0c103b50 2024-04-21T08:45:14.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.568+0000 7f3c01ffb640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3bf40a3c30 0x7f3bf40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.568+0000 7f3c10aee640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3bf40a3c30 0x7f3bf40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.568+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f3c040d0bd0 con 0x7f3c0c103b50 2024-04-21T08:45:14.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.568+0000 7f3c10aee640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3bf40a3c30 0x7f3bf40a60f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3c08036040 tx=0x7f3c08005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.572+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c04014030 con 0x7f3c0c103b50 2024-04-21T08:45:14.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.757+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c0c062b50 con 0x7f3c0c103b50 2024-04-21T08:45:14.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.757+0000 7f3c01ffb640 1 -- 172.21.15.57:0/544734605 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f3c0409d230 con 0x7f3c0c103b50 2024-04-21T08:45:14.757 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3bf40a3c30 msgr2=0x7f3bf40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3bf40a3c30 0x7f3bf40a60f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3c08036040 tx=0x7f3c08005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 msgr2=0x7f3c0c19f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 0x7f3c0c19f800 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400cce0 tx=0x7f3c0400a620 comp rx=0 tx=0).stop 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 shutdown_connections 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3bf40a3c30 0x7f3bf40a60f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c0c103b50 0x7f3c0c19f800 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c0c103190 0x7f3c0c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 --2- 172.21.15.57:0/544734605 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c0c101fa0 0x7f3c0c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 >> 172.21.15.57:0/544734605 conn(0x7f3c0c0fd530 msgr2=0x7f3c0c106510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 shutdown_connections 2024-04-21T08:45:14.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.759+0000 7f3c12d52640 1 -- 172.21.15.57:0/544734605 wait complete. 2024-04-21T08:45:14.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:14 smithi057 ceph-mon[27968]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:14.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2803934462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:14.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3530487774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:14.769 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/191931733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:14.770 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:14 smithi129 ceph-mon[31272]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2803934462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3530487774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/191931733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:14.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.875+0000 7f613fb52640 1 -- 172.21.15.57:0/1076286052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 msgr2=0x7f6138102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.875+0000 7f613fb52640 1 --2- 172.21.15.57:0/1076286052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138102460 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6128009900 tx=0x7f612802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- 172.21.15.57:0/1076286052 shutdown_connections 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 --2- 172.21.15.57:0/1076286052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61381029a0 0x7f61381092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 --2- 172.21.15.57:0/1076286052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138102460 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 --2- 172.21.15.57:0/1076286052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6138100e90 0x7f61381012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- 172.21.15.57:0/1076286052 >> 172.21.15.57:0/1076286052 conn(0x7f61380fc400 msgr2=0x7f61380fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- 172.21.15.57:0/1076286052 shutdown_connections 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- 172.21.15.57:0/1076286052 wait complete. 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 Processor -- start 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- start start 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6138100e90 0x7f6138198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61381029a0 0x7f613819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f613810daf0 con 0x7f6138101fe0 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f613810d970 con 0x7f61381029a0 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613fb52640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f613810dc70 con 0x7f6138100e90 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613d0ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613d0ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45984/0 (socket says 172.21.15.57:45984) 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.876+0000 7f613d0ed640 1 -- 172.21.15.57:0/729875937 learned_addr learned my addr 172.21.15.57:0/729875937 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d8ee640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6138100e90 0x7f6138198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d0ed640 1 -- 172.21.15.57:0/729875937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6138100e90 msgr2=0x7f6138198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d0ed640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6138100e90 0x7f6138198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d0ed640 1 -- 172.21.15.57:0/729875937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61381029a0 msgr2=0x7f613819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613e0ef640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61381029a0 0x7f613819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d0ed640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61381029a0 0x7f613819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d0ed640 1 -- 172.21.15.57:0/729875937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613819fe20 con 0x7f6138101fe0 2024-04-21T08:45:14.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613d0ed640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138198740 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6128036040 tx=0x7f61280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128043020 con 0x7f6138101fe0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6128004aa0 con 0x7f6138101fe0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61381a00b0 con 0x7f6138101fe0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128003de0 con 0x7f6138101fe0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.877+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61381a0650 con 0x7f6138101fe0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.878+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61381081c0 con 0x7f6138101fe0 2024-04-21T08:45:14.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.879+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6128046020 con 0x7f6138101fe0 2024-04-21T08:45:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.880+0000 7f612effd640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f61240a3c30 0x7f61240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.880+0000 7f613d8ee640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f61240a3c30 0x7f61240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.880+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f61280f3010 con 0x7f6138101fe0 2024-04-21T08:45:14.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.880+0000 7f613d8ee640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f61240a3c30 0x7f61240a60f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6134005e10 tx=0x7f6134005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:14.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:14.883+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f612804d020 con 0x7f6138101fe0 2024-04-21T08:45:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:14 smithi136 ceph-mon[31791]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-21T08:45:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2803934462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3530487774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/191931733' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:15.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.068+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f613802d830 con 0x7f6138101fe0 2024-04-21T08:45:15.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.068+0000 7f612effd640 1 -- 172.21.15.57:0/729875937 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f61280bf610 con 0x7f6138101fe0 2024-04-21T08:45:15.068 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f61240a3c30 msgr2=0x7f61240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f61240a3c30 0x7f61240a60f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6134005e10 tx=0x7f6134005da0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 msgr2=0x7f6138198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138198740 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6128036040 tx=0x7f61280042f0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 shutdown_connections 2024-04-21T08:45:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f61240a3c30 0x7f61240a60f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61381029a0 0x7f613819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6138101fe0 0x7f6138198740 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 --2- 172.21.15.57:0/729875937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6138100e90 0x7f6138198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 >> 172.21.15.57:0/729875937 conn(0x7f61380fc400 msgr2=0x7f6138105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 shutdown_connections 2024-04-21T08:45:15.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.070+0000 7f613fb52640 1 -- 172.21.15.57:0/729875937 wait complete. 2024-04-21T08:45:15.081 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 -- 172.21.15.57:0/1873483478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08102a00 msgr2=0x7fce08109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 --2- 172.21.15.57:0/1873483478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08102a00 0x7fce08109320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fce00009ca0 tx=0x7fce0002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 -- 172.21.15.57:0/1873483478 shutdown_connections 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 --2- 172.21.15.57:0/1873483478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08102a00 0x7fce08109320 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 --2- 172.21.15.57:0/1873483478 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce08102040 0x7fce081024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 --2- 172.21.15.57:0/1873483478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08100e60 0x7fce08101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 -- 172.21.15.57:0/1873483478 >> 172.21.15.57:0/1873483478 conn(0x7fce080fc440 msgr2=0x7fce080fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 -- 172.21.15.57:0/1873483478 shutdown_connections 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 -- 172.21.15.57:0/1873483478 wait complete. 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.186+0000 7fce0d481640 1 Processor -- start 2024-04-21T08:45:15.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 -- start start 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 0x7fce08198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce08102040 0x7fce081987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08102a00 0x7fce0819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fce0810ca70 con 0x7fce08100e60 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fce0810c8f0 con 0x7fce08102a00 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce0d481640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fce0810cbf0 con 0x7fce08102040 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 0x7fce08198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 0x7fce08198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45996/0 (socket says 172.21.15.57:45996) 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 -- 172.21.15.57:0/3767642990 learned_addr learned my addr 172.21.15.57:0/3767642990 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce067fc640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce08102040 0x7fce081987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce077fe640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08102a00 0x7fce0819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 -- 172.21.15.57:0/3767642990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce08102040 msgr2=0x7fce081987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce08102040 0x7fce081987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 -- 172.21.15.57:0/3767642990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08102a00 msgr2=0x7fce0819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08102a00 0x7fce0819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 -- 172.21.15.57:0/3767642990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0819fe40 con 0x7fce08100e60 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce077fe640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08102a00 0x7fce0819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.187+0000 7fce06ffd640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 0x7fce08198290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800b450 tx=0x7fcdf800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.188+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8015020 con 0x7fce08100e60 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.188+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdf8011d40 con 0x7fce08100e60 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.188+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce081a0130 con 0x7fce08100e60 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.188+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf800f340 con 0x7fce08100e60 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.188+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce081a0680 con 0x7fce08100e60 2024-04-21T08:45:15.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.189+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce081a0940 con 0x7fce08100e60 2024-04-21T08:45:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.190+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcdf8018020 con 0x7fce08100e60 2024-04-21T08:45:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.190+0000 7fcdeffff640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcdf00a3bb0 0x7fcdf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.191+0000 7fce067fc640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcdf00a3bb0 0x7fcdf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.191+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fcdf80d07c0 con 0x7fce08100e60 2024-04-21T08:45:15.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.191+0000 7fce067fc640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcdf00a3bb0 0x7fcdf00a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fce08199730 tx=0x7fcdfc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:15.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.194+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcdf801f020 con 0x7fce08100e60 2024-04-21T08:45:15.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.382+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fce081a0b10 con 0x7fce08100e60 2024-04-21T08:45:15.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.382+0000 7fcdeffff640 1 -- 172.21.15.57:0/3767642990 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fcdf800d040 con 0x7fce08100e60 2024-04-21T08:45:15.382 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcdf00a3bb0 msgr2=0x7fcdf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcdf00a3bb0 0x7fcdf00a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fce08199730 tx=0x7fcdfc00b390 comp rx=0 tx=0).stop 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 msgr2=0x7fce08198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 0x7fce08198290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800b450 tx=0x7fcdf800b910 comp rx=0 tx=0).stop 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 shutdown_connections 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fcdf00a3bb0 0x7fcdf00a6070 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce08102a00 0x7fce0819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce08102040 0x7fce081987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 --2- 172.21.15.57:0/3767642990 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce08100e60 0x7fce08198290 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 >> 172.21.15.57:0/3767642990 conn(0x7fce080fc440 msgr2=0x7fce080fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 shutdown_connections 2024-04-21T08:45:15.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.384+0000 7fce0d481640 1 -- 172.21.15.57:0/3767642990 wait complete. 2024-04-21T08:45:15.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 -- 172.21.15.57:0/2925177169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102050 msgr2=0x7f1c7c1024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/2925177169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102050 0x7f1c7c1024d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1c70009900 tx=0x7f1c7002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 -- 172.21.15.57:0/2925177169 shutdown_connections 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/2925177169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c7c102a10 0x7f1c7c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/2925177169 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102050 0x7f1c7c1024d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/2925177169 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c7c100e70 0x7f1c7c101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 -- 172.21.15.57:0/2925177169 >> 172.21.15.57:0/2925177169 conn(0x7f1c7c0fc470 msgr2=0x7f1c7c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 -- 172.21.15.57:0/2925177169 shutdown_connections 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.498+0000 7f1c81ee0640 1 -- 172.21.15.57:0/2925177169 wait complete. 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 Processor -- start 2024-04-21T08:45:15.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 -- start start 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c7c100e70 0x7f1c7c06b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c7c102050 0x7f1c7c19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 0x7f1c7c19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c10ca80 con 0x7f1c7c102a10 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c10c900 con 0x7f1c7c102050 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c81ee0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c10cc00 con 0x7f1c7c100e70 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c7bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 0x7f1c7c19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c7bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 0x7f1c7c19f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46016/0 (socket says 172.21.15.57:46016) 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c7bfff640 1 -- 172.21.15.57:0/4183602271 learned_addr learned my addr 172.21.15.57:0/4183602271 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c7affd640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c7c102050 0x7f1c7c19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.499+0000 7f1c7b7fe640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c7c100e70 0x7f1c7c06b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c7bfff640 1 -- 172.21.15.57:0/4183602271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c7c100e70 msgr2=0x7f1c7c06b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c7bfff640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c7c100e70 0x7f1c7c06b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c7bfff640 1 -- 172.21.15.57:0/4183602271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c7c102050 msgr2=0x7f1c7c19cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c7bfff640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c7c102050 0x7f1c7c19cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c7bfff640 1 -- 172.21.15.57:0/4183602271 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7c19fea0 con 0x7f1c7c102a10 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c7bfff640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 0x7f1c7c19f590 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400cad0 tx=0x7f1c7400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7400ae20 con 0x7f1c7c102a10 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c74005020 con 0x7f1c7c102a10 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7c1a0130 con 0x7f1c7c102a10 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7400f6b0 con 0x7f1c7c102a10 2024-04-21T08:45:15.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.500+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c7c1a0680 con 0x7f1c7c102a10 2024-04-21T08:45:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.501+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c7c108210 con 0x7f1c7c102a10 2024-04-21T08:45:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.503+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1c740051d0 con 0x7f1c7c102a10 2024-04-21T08:45:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.503+0000 7f1c78ff9640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c640a3bb0 0x7f1c640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.503+0000 7f1c7b7fe640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c640a3bb0 0x7f1c640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.503+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f1c740d0bd0 con 0x7f1c7c102a10 2024-04-21T08:45:15.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.504+0000 7f1c7b7fe640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c640a3bb0 0x7f1c640a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c101bc0 tx=0x7f1c6c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:15.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.508+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c74014030 con 0x7f1c7c102a10 2024-04-21T08:45:15.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.731+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c7c02d5c0 con 0x7f1c7c102a10 2024-04-21T08:45:15.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.732+0000 7f1c78ff9640 1 -- 172.21.15.57:0/4183602271 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f1c7409d230 con 0x7f1c7c102a10 2024-04-21T08:45:15.732 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:15.732 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:15.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.734+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c640a3bb0 msgr2=0x7f1c640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.734+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c640a3bb0 0x7f1c640a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c101bc0 tx=0x7f1c6c008040 comp rx=0 tx=0).stop 2024-04-21T08:45:15.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.734+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 msgr2=0x7f1c7c19f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.734+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 0x7f1c7c19f590 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400cad0 tx=0x7f1c7400cf90 comp rx=0 tx=0).stop 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 shutdown_connections 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f1c640a3bb0 0x7f1c640a6070 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1c7c102a10 0x7f1c7c19f590 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1c7c102050 0x7f1c7c19cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 --2- 172.21.15.57:0/4183602271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c7c100e70 0x7f1c7c06b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 >> 172.21.15.57:0/4183602271 conn(0x7f1c7c0fc470 msgr2=0x7f1c7c1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 shutdown_connections 2024-04-21T08:45:15.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.735+0000 7f1c81ee0640 1 -- 172.21.15.57:0/4183602271 wait complete. 2024-04-21T08:45:15.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3367645042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/544734605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/729875937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.745 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3767642990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:15.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3367645042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/544734605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/729875937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3767642990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 -- 172.21.15.57:0/3327655443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103b50 msgr2=0x7f615410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 --2- 172.21.15.57:0/3327655443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103b50 0x7f615410a3e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6144009900 tx=0x7f614402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:15.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 -- 172.21.15.57:0/3327655443 shutdown_connections 2024-04-21T08:45:15.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 --2- 172.21.15.57:0/3327655443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103b50 0x7f615410a3e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 --2- 172.21.15.57:0/3327655443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6154103190 0x7f6154103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 --2- 172.21.15.57:0/3327655443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154101fa0 0x7f61541023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 -- 172.21.15.57:0/3327655443 >> 172.21.15.57:0/3327655443 conn(0x7f61540fd530 msgr2=0x7f61540ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 -- 172.21.15.57:0/3327655443 shutdown_connections 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.848+0000 7f615bf23640 1 -- 172.21.15.57:0/3327655443 wait complete. 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 Processor -- start 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 -- start start 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6154101fa0 0x7f6154198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 0x7f6154198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154103b50 0x7f615419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f615410dae0 con 0x7f6154103190 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f615410d960 con 0x7f6154103b50 2024-04-21T08:45:15.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615bf23640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f615410dc60 con 0x7f6154101fa0 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 0x7f6154198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 0x7f6154198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46034/0 (socket says 172.21.15.57:46034) 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 -- 172.21.15.57:0/1778097561 learned_addr learned my addr 172.21.15.57:0/1778097561 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 -- 172.21.15.57:0/1778097561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6154101fa0 msgr2=0x7f6154198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f6159cbf640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6154101fa0 0x7f6154198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6154101fa0 0x7f6154198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615a4c0640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154103b50 0x7f615419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 -- 172.21.15.57:0/1778097561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154103b50 msgr2=0x7f615419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154103b50 0x7f615419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:15.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 -- 172.21.15.57:0/1778097561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615419fd50 con 0x7f6154103190 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f615a4c0640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154103b50 0x7f615419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.849+0000 7f61594be640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 0x7f6154198670 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f614800d770 tx=0x7f614800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.850+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148017070 con 0x7f6154103190 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.850+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61480054d0 con 0x7f6154103190 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.850+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148004c30 con 0x7f6154103190 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.850+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615419ffe0 con 0x7f6154103190 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.850+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61541a0500 con 0x7f6154103190 2024-04-21T08:45:15.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.851+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61541092c0 con 0x7f6154103190 2024-04-21T08:45:15.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.852+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6148004e10 con 0x7f6154103190 2024-04-21T08:45:15.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.852+0000 7f6152ffd640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f613c0a3c30 0x7f613c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:15.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.853+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f61480d0240 con 0x7f6154103190 2024-04-21T08:45:15.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.853+0000 7f6159cbf640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f613c0a3c30 0x7f613c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:15.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.853+0000 7f6159cbf640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f613c0a3c30 0x7f613c0a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f614c005c40 tx=0x7f614c005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:15.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:15.857+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f614809c8a0 con 0x7f6154103190 2024-04-21T08:45:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3367645042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/544734605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/729875937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3767642990' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:16.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.044+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6154062bf0 con 0x7f6154103190 2024-04-21T08:45:16.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.045+0000 7f6152ffd640 1 -- 172.21.15.57:0/1778097561 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7f614809c1a0 con 0x7f6154103190 2024-04-21T08:45:16.045 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:16.045 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:16.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f613c0a3c30 msgr2=0x7f613c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f613c0a3c30 0x7f613c0a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f614c005c40 tx=0x7f614c005bd0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 msgr2=0x7f6154198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 0x7f6154198670 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f614800d770 tx=0x7f614800dc30 comp rx=0 tx=0).stop 2024-04-21T08:45:16.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 shutdown_connections 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f613c0a3c30 0x7f613c0a60f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6154103b50 0x7f615419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6154103190 0x7f6154198670 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 --2- 172.21.15.57:0/1778097561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6154101fa0 0x7f6154198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.047+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 >> 172.21.15.57:0/1778097561 conn(0x7f61540fd530 msgr2=0x7f6154106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.048+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 shutdown_connections 2024-04-21T08:45:16.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.048+0000 7f615bf23640 1 -- 172.21.15.57:0/1778097561 wait complete. 2024-04-21T08:45:16.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:42:56.988323+0000', 'modified': '2024-04-21T08:43:22.134494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6839/4293224778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 4293224778}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 4293224778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6839/123336774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 123336774}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 123336774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24503]} 2024-04-21T08:45:16.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:45:16.059 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:45:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/4160254936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 msgr2=0x7f0ff410a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/4160254936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff410a3d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0fec009ca0 tx=0x7f0fec02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/4160254936 shutdown_connections 2024-04-21T08:45:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/4160254936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff410a3d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/4160254936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ff4103180 0x7f0ff4103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/4160254936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4101f90 0x7f0ff41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/4160254936 >> 172.21.15.57:0/4160254936 conn(0x7f0ff40fd500 msgr2=0x7f0ff40ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/4160254936 shutdown_connections 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.167+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/4160254936 wait complete. 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 Processor -- start 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 -- start start 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ff4101f90 0x7f0ff4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4103180 0x7f0ff41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff410dad0 con 0x7f0ff4103b40 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff410d950 con 0x7f0ff4103180 2024-04-21T08:45:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff9f4b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff410dc50 con 0x7f0ff4101f90 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46050/0 (socket says 172.21.15.57:46050) 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.168+0000 7f0ff3fff640 1 -- 172.21.15.57:0/659999150 learned_addr learned my addr 172.21.15.57:0/659999150 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ff4101f90 0x7f0ff4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4103180 0x7f0ff41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff3fff640 1 -- 172.21.15.57:0/659999150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ff4101f90 msgr2=0x7f0ff4198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff3fff640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ff4101f90 0x7f0ff4198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff3fff640 1 -- 172.21.15.57:0/659999150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4103180 msgr2=0x7f0ff41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff3fff640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4103180 0x7f0ff41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff3fff640 1 -- 172.21.15.57:0/659999150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff419fda0 con 0x7f0ff4103b40 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4103180 0x7f0ff41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff3fff640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff419f730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0fec02c700 tx=0x7f0fec0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff0ff9640 1 -- 172.21.15.57:0/659999150 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec041020 con 0x7f0ff4103b40 2024-04-21T08:45:16.171 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff0ff9640 1 -- 172.21.15.57:0/659999150 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fec034ca0 con 0x7f0ff4103b40 2024-04-21T08:45:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff41a0030 con 0x7f0ff4103b40 2024-04-21T08:45:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff0ff9640 1 -- 172.21.15.57:0/659999150 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fec039040 con 0x7f0ff4103b40 2024-04-21T08:45:16.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.169+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff41a0520 con 0x7f0ff4103b40 2024-04-21T08:45:16.178 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ff0ff9640 1 -- 172.21.15.57:0/659999150 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0fec044020 con 0x7f0ff4103b40 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 Processor -- start 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 -- start start 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80aae40 0x5653b80ab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 0x5653b80abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ac2d0 0x5653b80b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5653b800fb10 con 0x5653b80ac2d0 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5653b7fad240 con 0x5653b80aae40 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.172+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5653b800bad0 con 0x5653b80ab830 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ac2d0 0x5653b80b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 0x5653b80abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 0x5653b80abcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42022/0 (socket says 172.21.15.57:42022) 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 -- 172.21.15.57:0/1412558873 learned_addr learned my addr 172.21.15.57:0/1412558873 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80aae40 0x5653b80ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff0ff9640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0fdc0a3bb0 0x7f0fdc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 -- 172.21.15.57:0/1412558873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80aae40 msgr2=0x5653b80ab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80aae40 0x5653b80ab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff3fff640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0fdc0a3bb0 0x7f0fdc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 -- 172.21.15.57:0/1412558873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ac2d0 msgr2=0x5653b80b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ac2d0 0x5653b80b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 -- 172.21.15.57:0/1412558873 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653b7d222d0 con 0x5653b80ab830 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff0ff9640 1 -- 172.21.15.57:0/659999150 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f0fec0f2500 con 0x7f0ff4103b40 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 0x5653b80abcb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009ca0 tx=0x7f0fe4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=169c08686168408d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0fd9ffb640 1 -- 172.21.15.57:0/1412558873 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4004220 con 0x5653b80ab830 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0ff3fff640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0fdc0a3bb0 0x7f0fdc0a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0be3f0 tx=0x7f0fec0b0040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0fd9ffb640 1 -- 172.21.15.57:0/1412558873 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fe40043c0 con 0x5653b80ab830 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.173+0000 7f0fd9ffb640 1 -- 172.21.15.57:0/1412558873 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe40197a0 con 0x5653b80ab830 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 -- 172.21.15.57:0/1412558873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 msgr2=0x5653b80abcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 0x5653b80abcb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009ca0 tx=0x7f0fe4011810 comp rx=0 tx=0).stop 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 -- 172.21.15.57:0/1412558873 shutdown_connections 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ac2d0 0x5653b80b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ab830 0x5653b80abcb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/1412558873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80aae40 0x5653b80ab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 -- 172.21.15.57:0/1412558873 >> 172.21.15.57:0/1412558873 conn(0x5653b80a6f40 msgr2=0x5653b80a5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 -- 172.21.15.57:0/1412558873 shutdown_connections 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.174+0000 7f0ffcc14740 1 -- 172.21.15.57:0/1412558873 wait complete. 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 -- start start 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 Processor -- start 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5653b80b34b0 con 0x5653b80ab830 2024-04-21T08:45:16.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5653b7f98190 con 0x5653b80ac2d0 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5653b80b3630 con 0x5653b80aae40 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ffcc14740 10 client.0 _ll_register_callbacks cb 0x5653b7f938b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46064/0 (socket says 172.21.15.57:46064) 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 -- 172.21.15.57:0/3381409308 learned_addr learned my addr 172.21.15.57:0/3381409308 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 -- 172.21.15.57:0/3381409308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 msgr2=0x5653b80b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 -- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 msgr2=0x5653b80bc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.175+0000 7f0ff37fe640 1 -- 172.21.15.57:0/3381409308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653b7dec6b0 con 0x5653b80ab830 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80b53f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009f50 tx=0x7f0fe4003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66253bf29473a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0fd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/3381409308 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4028070 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/3381409308 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f0fe4005250 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fe40043c0 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/3381409308 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0fe4034050 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 10 client.15348 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5653b80bf7a0 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/3381409308 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fe40043c0 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/3381409308 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0fe4004b70 con 0x5653b80ab830 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 10 client.15348 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 10 client.15348 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 20 client.15348 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b80c1c30 0x5653b80c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 4 client.15348 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:16.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.176+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653b801d530 con 0x5653b80c1c30 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.177+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b80c1c30 0x5653b80c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.177+0000 7f0fd2ffd640 10 client.15348 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.177+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b80c1c30 0x5653b80c30a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba9e5e76fc210003 server_cookie=a5916aebc5c22198 in_seq=0 out_seq=0 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/3381409308 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653b801d530 con 0x5653b80c1c30 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0fd2ffd640 10 client.15348 handle_command_reply: tid=0 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 client.15348 shutdown 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 msgr2=0x5653b80b53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80b53f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009f50 tx=0x7f0fe4003880 comp rx=0 tx=0).stop 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 shutdown_connections 2024-04-21T08:45:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b80c1c30 0x5653b80c30a0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80b53f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/3381409308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0fd2ffd640 0 client.15348 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 >> 172.21.15.57:0/3381409308 conn(0x5653b80a6f40 msgr2=0x5653b80a4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 shutdown_connections 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 1 -- 172.21.15.57:0/3381409308 wait complete. 2024-04-21T08:45:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.178+0000 7f0ffcc14740 20 client.15348 trim_cache size 0 max 16384 2024-04-21T08:45:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 Processor -- start 2024-04-21T08:45:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 -- start start 2024-04-21T08:45:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 0x5653b80a5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ab830 0x5653b80a5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ac2d0 0x5653b80bc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5653b80c13e0 con 0x5653b80aae40 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5653b80c1ab0 con 0x5653b80ab830 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5653b80c2180 con 0x5653b80ac2d0 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 0x5653b80a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 0x5653b80a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46068/0 (socket says 172.21.15.57:46068) 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 -- 172.21.15.57:0/811337329 learned_addr learned my addr 172.21.15.57:0/811337329 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 -- 172.21.15.57:0/811337329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ac2d0 msgr2=0x5653b80bc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ac2d0 0x5653b80bc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 -- 172.21.15.57:0/811337329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ab830 msgr2=0x5653b80a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ab830 0x5653b80a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 -- 172.21.15.57:0/811337329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653b80b2ac0 con 0x5653b80aae40 2024-04-21T08:45:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.197+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 0x5653b80a5400 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x5653b80b44a0 tx=0x7f0fe8014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbed312332b82cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0fd97fa640 1 -- 172.21.15.57:0/811337329 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8026e60 con 0x5653b80aae40 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0fd97fa640 1 -- 172.21.15.57:0/811337329 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fe802b070 con 0x5653b80aae40 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0fd97fa640 1 -- 172.21.15.57:0/811337329 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe8004b90 con 0x5653b80aae40 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 -- 172.21.15.57:0/811337329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 msgr2=0x5653b80a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 0x5653b80a5400 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x5653b80b44a0 tx=0x7f0fe8014040 comp rx=0 tx=0).stop 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 -- 172.21.15.57:0/811337329 shutdown_connections 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80ac2d0 0x5653b80bc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ab830 0x5653b80a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/811337329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80aae40 0x5653b80a5400 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 -- 172.21.15.57:0/811337329 >> 172.21.15.57:0/811337329 conn(0x5653b80a6f40 msgr2=0x5653b80af840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 -- 172.21.15.57:0/811337329 shutdown_connections 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.198+0000 7f0ffcc14740 1 -- 172.21.15.57:0/811337329 wait complete. 2024-04-21T08:45:16.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 -- start start 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 Processor -- start 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5653b80c2f20 con 0x5653b80ab830 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5653b80c2850 con 0x5653b80ac2d0 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5653b80c35f0 con 0x5653b80aae40 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ffcc14740 10 client.0 _ll_register_callbacks cb 0x7f0fe8009a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80ba920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46070/0 (socket says 172.21.15.57:46070) 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 -- 172.21.15.57:0/139658356 learned_addr learned my addr 172.21.15.57:0/139658356 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 -- 172.21.15.57:0/139658356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 msgr2=0x5653b80b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 -- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 msgr2=0x5653b80bae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff37fe640 1 -- 172.21.15.57:0/139658356 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653b80bb670 con 0x5653b80ab830 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.199+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ff37fe640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80ba920 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009f50 tx=0x7f0fe4009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2055a6c12d52132 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0fd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/139658356 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4031020 con 0x5653b80ab830 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/139658356 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f0fe4005450 con 0x5653b80ab830 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fe802b070 con 0x5653b80ab830 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/139658356 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0fe4028070 con 0x5653b80ab830 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 10 client.15354 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:16.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5653b8127be0 con 0x5653b80ab830 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/139658356 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fe802b070 con 0x5653b80ab830 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/139658356 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0fe40039c0 con 0x5653b80ab830 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 10 client.15354 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 10 client.15354 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 20 client.15354 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b812c760 0x5653b812b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 4 client.15354 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.200+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "status"}) -- 0x5653b80173a0 con 0x5653b812c760 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b812c760 0x5653b812b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ff2ffd640 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b812c760 0x5653b812b470 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a514438d1685721 server_cookie=a6bb4b2a099df70 in_seq=0 out_seq=0 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0fd2ffd640 10 client.15354 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0fd2ffd640 1 -- 172.21.15.57:0/139658356 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5653b80173a0 con 0x5653b812c760 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0fd2ffd640 10 client.15354 handle_command_reply: tid=0 2024-04-21T08:45:16.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 client.15354 shutdown 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 msgr2=0x5653b80ba920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80ba920 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009f50 tx=0x7f0fe4009fa0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 shutdown_connections 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5653b812c760 0x5653b812b470 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5653b80ac2d0 0x5653b80bae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0fd2ffd640 0 client.15354 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5653b80ab830 0x5653b80ba920 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.201+0000 7f0ffcc14740 1 --2- 172.21.15.57:0/139658356 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5653b80aae40 0x5653b80b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 >> 172.21.15.57:0/139658356 conn(0x5653b80a6f40 msgr2=0x5653b80a4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 shutdown_connections 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ffcc14740 1 -- 172.21.15.57:0/139658356 wait complete. 2024-04-21T08:45:16.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ffcc14740 20 client.15354 trim_cache size 0 max 16384 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 24503, 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 138.58014150899999, 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 21, 2024-04-21T08:45:16.209 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 93, 2024-04-21T08:45:16.210 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:45:16.210 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 144.882994967 2024-04-21T08:45:16.210 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0fdc0a3bb0 msgr2=0x7f0fdc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0fdc0a3bb0 0x7f0fdc0a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0fec0be3f0 tx=0x7f0fec0b0040 comp rx=0 tx=0).stop 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 msgr2=0x7f0ff419f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff419f730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0fec02c700 tx=0x7f0fec0346d0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 shutdown_connections 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0fdc0a3bb0 0x7f0fdc0a6070 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ff4103b40 0x7f0ff419f730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ff4103180 0x7f0ff41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.212 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 --2- 172.21.15.57:0/659999150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ff4101f90 0x7f0ff4198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.212 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 >> 172.21.15.57:0/659999150 conn(0x7f0ff40fd500 msgr2=0x7f0ff41064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.212 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 shutdown_connections 2024-04-21T08:45:16.212 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.202+0000 7f0ff9f4b640 1 -- 172.21.15.57:0/659999150 wait complete. 2024-04-21T08:45:16.213 INFO:teuthology.run_tasks:Running task sequential... 2024-04-21T08:45:16.222 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-21T08:45:16.222 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-21T08:45:16.223 INFO:teuthology.task.print:Enabling mgr modules 2024-04-21T08:45:16.223 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-21T08:45:16.223 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:45:16.224 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:45:16.224 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-21T08:45:16.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 -- 172.21.15.57:0/2346025978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102040 msgr2=0x7f92041024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 --2- 172.21.15.57:0/2346025978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102040 0x7f92041024c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f91f0009900 tx=0x7f91f002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 -- 172.21.15.57:0/2346025978 shutdown_connections 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 --2- 172.21.15.57:0/2346025978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102a00 0x7f9204109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 --2- 172.21.15.57:0/2346025978 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102040 0x7f92041024c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 --2- 172.21.15.57:0/2346025978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9204100e60 0x7f9204101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 -- 172.21.15.57:0/2346025978 >> 172.21.15.57:0/2346025978 conn(0x7f92040fc420 msgr2=0x7f92040fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 -- 172.21.15.57:0/2346025978 shutdown_connections 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 -- 172.21.15.57:0/2346025978 wait complete. 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 Processor -- start 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 -- start start 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.326+0000 7f920b0c7640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9204100e60 0x7f9204198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f920b0c7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102040 0x7f92041986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f920b0c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 0x7f920419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f920b0c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f920410ca40 con 0x7f9204102a00 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f920b0c7640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f920410c8c0 con 0x7f9204100e60 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f920b0c7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f920410cbc0 con 0x7f9204102040 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 0x7f920419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 0x7f920419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46092/0 (socket says 172.21.15.57:46092) 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 -- 172.21.15.57:0/3544482108 learned_addr learned my addr 172.21.15.57:0/3544482108 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9203fff640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102040 0x7f92041986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9208e63640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9204100e60 0x7f9204198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 -- 172.21.15.57:0/3544482108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102040 msgr2=0x7f92041986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102040 0x7f92041986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 -- 172.21.15.57:0/3544482108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9204100e60 msgr2=0x7f9204198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9204100e60 0x7f9204198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 -- 172.21.15.57:0/3544482108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f920419fdb0 con 0x7f9204102a00 2024-04-21T08:45:16.329 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9203fff640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102040 0x7f92041986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:16.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9209664640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 0x7f920419f740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009fd0 tx=0x7f91fc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc017070 con 0x7f9204102a00 2024-04-21T08:45:16.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91fc003e20 con 0x7f9204102a00 2024-04-21T08:45:16.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92041a00a0 con 0x7f9204102a00 2024-04-21T08:45:16.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.327+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc010460 con 0x7f9204102a00 2024-04-21T08:45:16.330 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.328+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92041a05c0 con 0x7f9204102a00 2024-04-21T08:45:16.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.329+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92041a0820 con 0x7f9204102a00 2024-04-21T08:45:16.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.330+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f91fc003970 con 0x7f9204102a00 2024-04-21T08:45:16.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.330+0000 7f9201ffb640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f91f40a3cd0 0x7f91f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:16.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.331+0000 7f9208e63640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f91f40a3cd0 0x7f91f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:16.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.331+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f91fc0d0570 con 0x7f9204102a00 2024-04-21T08:45:16.331 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.331+0000 7f9208e63640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f91f40a3cd0 0x7f91f40a6190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9204101bb0 tx=0x7f91f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:16.334 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.335+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91fc09cbd0 con 0x7f9204102a00 2024-04-21T08:45:16.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:16.575+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f92041a0bb0 con 0x7f9204102a00 2024-04-21T08:45:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:16 smithi129 ceph-mon[31272]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:16 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4183602271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:16 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1778097561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:16 smithi057 ceph-mon[27968]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:16 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4183602271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:16 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1778097561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:16 smithi136 ceph-mon[31791]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:45:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:16 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4183602271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:16 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1778097561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:17.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.434+0000 7f9201ffb640 1 -- 172.21.15.57:0/3544482108 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f91fc09c4d0 con 0x7f9204102a00 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f91f40a3cd0 msgr2=0x7f91f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f91f40a3cd0 0x7f91f40a6190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9204101bb0 tx=0x7f91f8005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 msgr2=0x7f920419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 0x7f920419f740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f91fc009fd0 tx=0x7f91fc00b260 comp rx=0 tx=0).stop 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 shutdown_connections 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f91f40a3cd0 0x7f91f40a6190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9204102a00 0x7f920419f740 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9204102040 0x7f92041986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 --2- 172.21.15.57:0/3544482108 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9204100e60 0x7f9204198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 >> 172.21.15.57:0/3544482108 conn(0x7f92040fc420 msgr2=0x7f9204069600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 shutdown_connections 2024-04-21T08:45:17.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.437+0000 7f920b0c7640 1 -- 172.21.15.57:0/3544482108 wait complete. 2024-04-21T08:45:17.457 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-21T08:45:17.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 -- 172.21.15.57:0/2347283905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c107d30 msgr2=0x7ff81c10a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 --2- 172.21.15.57:0/2347283905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c107d30 0x7ff81c10a120 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff814009ca0 tx=0x7ff81402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 -- 172.21.15.57:0/2347283905 shutdown_connections 2024-04-21T08:45:17.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 --2- 172.21.15.57:0/2347283905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c107d30 0x7ff81c10a120 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 --2- 172.21.15.57:0/2347283905 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff81c105400 0x7ff81c1077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 --2- 172.21.15.57:0/2347283905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 0x7ff81c104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.579+0000 7ff822cd2640 1 -- 172.21.15.57:0/2347283905 >> 172.21.15.57:0/2347283905 conn(0x7ff81c0fc420 msgr2=0x7ff81c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 -- 172.21.15.57:0/2347283905 shutdown_connections 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 -- 172.21.15.57:0/2347283905 wait complete. 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 Processor -- start 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 -- start start 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 0x7ff81c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 0x7ff81c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff81c107d30 0x7ff81c19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c10db10 con 0x7ff81c105400 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c10d990 con 0x7ff81c107d30 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff822cd2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff81c10dc90 con 0x7ff81c102ad0 2024-04-21T08:45:17.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff81bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 0x7ff81c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff81bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 0x7ff81c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46120/0 (socket says 172.21.15.57:46120) 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.580+0000 7ff81bfff640 1 -- 172.21.15.57:0/3778557611 learned_addr learned my addr 172.21.15.57:0/3778557611 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff82126f640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff81c107d30 0x7ff81c19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff820a6e640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 0x7ff81c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff81bfff640 1 -- 172.21.15.57:0/3778557611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 msgr2=0x7ff81c198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff81bfff640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 0x7ff81c198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff81bfff640 1 -- 172.21.15.57:0/3778557611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff81c107d30 msgr2=0x7ff81c19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff81bfff640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff81c107d30 0x7ff81c19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff81bfff640 1 -- 172.21.15.57:0/3778557611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81c19fd50 con 0x7ff81c105400 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff820a6e640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 0x7ff81c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff81bfff640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 0x7ff81c198670 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff80800e9c0 tx=0x7ff80800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80800cd80 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8080054d0 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81c19ffe0 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808010430 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.581+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81c1a0500 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.582+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff81c0fdf10 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.583+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7ff808004fd0 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.584+0000 7ff819ffb640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 0x7ff80c0a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.584+0000 7ff820a6e640 1 -- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 msgr2=0x7ff80c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.584+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7ff8080d0c60 con 0x7ff81c105400 2024-04-21T08:45:17.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.584+0000 7ff820a6e640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 0x7ff80c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:17.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.589+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff80809d2c0 con 0x7ff81c105400 2024-04-21T08:45:17.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.783+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7ff81c062bf0 con 0x7ff81c105400 2024-04-21T08:45:17.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.785+0000 7ff820a6e640 1 -- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 msgr2=0x7ff80c0a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:17.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.785+0000 7ff820a6e640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 0x7ff80c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:45:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.788+0000 7ff819ffb640 1 -- 172.21.15.57:0/3778557611 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7ff80809cbc0 con 0x7ff81c105400 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 msgr2=0x7ff80c0a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 0x7ff80c0a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 msgr2=0x7ff81c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 0x7ff81c198670 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff80800e9c0 tx=0x7ff80800ee80 comp rx=0 tx=0).stop 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 shutdown_connections 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7ff80c0a3ca0 0x7ff80c0a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff81c107d30 0x7ff81c19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff81c105400 0x7ff81c198670 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 --2- 172.21.15.57:0/3778557611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff81c102ad0 0x7ff81c198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 >> 172.21.15.57:0/3778557611 conn(0x7ff81c0fc420 msgr2=0x7ff81c103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 shutdown_connections 2024-04-21T08:45:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.790+0000 7ff822cd2640 1 -- 172.21.15.57:0/3778557611 wait complete. 2024-04-21T08:45:17.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3544482108' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:45:17.797 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: ignoring --setuser ceph since I am not root 2024-04-21T08:45:17.797 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: ignoring --setgroup ceph since I am not root 2024-04-21T08:45:17.798 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-mgr[33648]: -- 172.21.15.136:0/3226247850 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c36bab85a0 con 0x55c36ba8cc00 2024-04-21T08:45:17.798 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:17.612+0000 7f2bca73d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:45:17.798 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:17.797+0000 7f2bca73d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:45:17.800 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:17 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3544482108' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:45:17.800 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: ignoring --setuser ceph since I am not root 2024-04-21T08:45:17.800 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: ignoring --setgroup ceph since I am not root 2024-04-21T08:45:17.800 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:17.641+0000 7fa300c7d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:45:17.801 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-21T08:45:17.803 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3544482108' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-21T08:45:17.804 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: ignoring --setuser ceph since I am not root 2024-04-21T08:45:17.804 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: ignoring --setgroup ceph since I am not root 2024-04-21T08:45:17.804 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-mgr[32671]: -- 172.21.15.129:0/1332409141 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a6897385a0 con 0x55a68970c800 2024-04-21T08:45:17.804 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:17.615+0000 7f3adc01e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-21T08:45:17.804 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:17.804+0000 7f3adc01e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:45:17.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 -- 172.21.15.57:0/258932392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 msgr2=0x7f3f2c10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 --2- 172.21.15.57:0/258932392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c10a3f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009ca0 tx=0x7f3f2402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 -- 172.21.15.57:0/258932392 shutdown_connections 2024-04-21T08:45:17.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 --2- 172.21.15.57:0/258932392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c10a3f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 --2- 172.21.15.57:0/258932392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c1031a0 0x7f3f2c103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 --2- 172.21.15.57:0/258932392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f2c101fb0 0x7f3f2c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 -- 172.21.15.57:0/258932392 >> 172.21.15.57:0/258932392 conn(0x7f3f2c0fd540 msgr2=0x7f3f2c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 -- 172.21.15.57:0/258932392 shutdown_connections 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 -- 172.21.15.57:0/258932392 wait complete. 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.905+0000 7f3f31515640 1 Processor -- start 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 -- start start 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c101fb0 0x7f3f2c075d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f2c1031a0 0x7f3f2c074340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c10daf0 con 0x7f3f2c103b60 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c10d970 con 0x7f3f2c101fb0 2024-04-21T08:45:17.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f31515640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c10dc70 con 0x7f3f2c1031a0 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c101fb0 0x7f3f2c075d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c074880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46132/0 (socket says 172.21.15.57:46132) 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/2040665052 learned_addr learned my addr 172.21.15.57:0/2040665052 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2a7fc640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f2c1031a0 0x7f3f2c074340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/2040665052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f2c1031a0 msgr2=0x7f3f2c074340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f2c1031a0 0x7f3f2c074340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/2040665052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c101fb0 msgr2=0x7f3f2c075d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c101fb0 0x7f3f2c075d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2b7fe640 1 -- 172.21.15.57:0/2040665052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2c075110 con 0x7f3f2c103b60 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.906+0000 7f3f2affd640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c101fb0 0x7f3f2c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:17.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.907+0000 7f3f2b7fe640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c074880 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3f2402c700 tx=0x7f3f24034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:17.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.907+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24039040 con 0x7f3f2c103b60 2024-04-21T08:45:17.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.907+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f24044070 con 0x7f3f2c103b60 2024-04-21T08:45:17.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.907+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2c1a2150 con 0x7f3f2c103b60 2024-04-21T08:45:17.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.907+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f240042a0 con 0x7f3f2c103b60 2024-04-21T08:45:17.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.907+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2c1a26a0 con 0x7f3f2c103b60 2024-04-21T08:45:17.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.908+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f2c1092d0 con 0x7f3f2c103b60 2024-04-21T08:45:17.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.909+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f3f24003970 con 0x7f3f2c103b60 2024-04-21T08:45:17.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.909+0000 7f3f13fff640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3f140a3ca0 0x7f3f140a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:17.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.910+0000 7f3f2affd640 1 -- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3f140a3ca0 msgr2=0x7f3f140a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:17.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.910+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f3f240f2b10 con 0x7f3f2c103b60 2024-04-21T08:45:17.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.910+0000 7f3f2affd640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3f140a3ca0 0x7f3f140a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:17.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:17.914+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f240bf170 con 0x7f3f2c103b60 2024-04-21T08:45:18.073 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:17.907+0000 7f3adc01e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:45:18.088 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:17.840+0000 7fa300c7d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:45:18.089 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:17.953+0000 7fa300c7d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:45:18.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.102+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f3f2c062b50 con 0x7f3f2c103b60 2024-04-21T08:45:18.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.107+0000 7f3f13fff640 1 -- 172.21.15.57:0/2040665052 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7f3f240bea70 con 0x7f3f2c103b60 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3f140a3ca0 msgr2=0x7f3f140a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3f140a3ca0 0x7f3f140a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 msgr2=0x7f3f2c074880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c074880 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3f2402c700 tx=0x7f3f24034760 comp rx=0 tx=0).stop 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 shutdown_connections 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f3f140a3ca0 0x7f3f140a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f2c103b60 0x7f3f2c074880 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f2c1031a0 0x7f3f2c074340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 --2- 172.21.15.57:0/2040665052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f2c101fb0 0x7f3f2c075d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 >> 172.21.15.57:0/2040665052 conn(0x7f3f2c0fd540 msgr2=0x7f3f2c1064a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:18.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 shutdown_connections 2024-04-21T08:45:18.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:18.109+0000 7f3f31515640 1 -- 172.21.15.57:0/2040665052 wait complete. 2024-04-21T08:45:18.120 INFO:teuthology.run_tasks:Running task kclient... 2024-04-21T08:45:18.134 INFO:tasks.kclient:Mounting kernel clients... 2024-04-21T08:45:18.134 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-21T08:45:18.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:18.134 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2024-04-21T08:45:18.135 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:17.900+0000 7f2bca73d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-21T08:45:18.150 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-21T08:45:18.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:18.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-21T08:45:18.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:18.216 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2024-04-21T08:45:18.338 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-21T08:45:18.338 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-21T08:45:18.338 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:18.338 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph fs ls 2024-04-21T08:45:18.385 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:18.290+0000 7fa300c7d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:45:18.385 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:18.385+0000 7fa300c7d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:45:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:18 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3544482108' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:45:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:18 smithi129 ceph-mon[31272]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:45:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:18 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3778557611' entity='client.admin' 2024-04-21T08:45:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:18 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2040665052' entity='client.admin' 2024-04-21T08:45:18.574 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:18.216+0000 7f3adc01e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:45:18.574 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:18.300+0000 7f3adc01e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:45:18.600 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:45:18.635 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:18 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:18.207+0000 7f2bca73d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-21T08:45:18.635 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:18 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:18.292+0000 7f2bca73d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-21T08:45:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:18 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3544482108' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:45:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:18 smithi136 ceph-mon[31791]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:45:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:18 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3778557611' entity='client.admin' 2024-04-21T08:45:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:18 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2040665052' entity='client.admin' 2024-04-21T08:45:18.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:18 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3544482108' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-21T08:45:18.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:18 smithi057 ceph-mon[27968]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-21T08:45:18.723 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:18 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3778557611' entity='client.admin' 2024-04-21T08:45:18.724 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:18 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2040665052' entity='client.admin' 2024-04-21T08:45:19.802 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:19.462+0000 7f2bca73d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:45:19.802 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:19.632+0000 7f2bca73d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:45:19.802 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:19.802+0000 7f2bca73d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:45:19.817 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:19.476+0000 7f3adc01e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:45:19.817 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:19.645+0000 7f3adc01e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:45:19.818 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:19.818+0000 7f3adc01e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:45:19.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:19.675+0000 7fa300c7d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-21T08:45:20.208 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:19.856+0000 7fa300c7d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-21T08:45:20.208 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:20.039+0000 7fa300c7d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-21T08:45:20.385 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:20.090+0000 7f2bca73d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:45:20.385 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:20.183+0000 7f2bca73d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:45:20.399 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:20.104+0000 7f3adc01e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:45:20.399 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:20.196+0000 7f3adc01e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:45:20.458 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:20.349+0000 7fa300c7d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-21T08:45:20.459 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:20.450+0000 7fa300c7d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-21T08:45:20.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 -- 172.21.15.57:0/3490017220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 msgr2=0x7f40ac10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/3490017220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac10f930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f40a400b0a0 tx=0x7f40a400b3b0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 -- 172.21.15.57:0/3490017220 shutdown_connections 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/3490017220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac10f930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/3490017220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac108a60 0x7f40ac108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/3490017220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac06b140 0x7f40ac06b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 -- 172.21.15.57:0/3490017220 >> 172.21.15.57:0/3490017220 conn(0x7f40ac078980 msgr2=0x7f40ac078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 -- 172.21.15.57:0/3490017220 shutdown_connections 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.456+0000 7f40b1ab5700 1 -- 172.21.15.57:0/3490017220 wait complete. 2024-04-21T08:45:20.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 Processor -- start 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 -- start start 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac06b140 0x7f40ac19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac108a60 0x7f40ac19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac112c00 con 0x7f40ac06bae0 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac112a80 con 0x7f40ac108a60 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40b1ab5700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f40ac112d80 con 0x7f40ac06b140 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac19b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46140/0 (socket says 172.21.15.57:46140) 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 -- 172.21.15.57:0/1969350514 learned_addr learned my addr 172.21.15.57:0/1969350514 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:20.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40ab7fe700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac06b140 0x7f40ac19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40abfff700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac108a60 0x7f40ac19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 -- 172.21.15.57:0/1969350514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac06b140 msgr2=0x7f40ac19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac06b140 0x7f40ac19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 -- 172.21.15.57:0/1969350514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac108a60 msgr2=0x7f40ac19fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.457+0000 7f40aaffd700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac108a60 0x7f40ac19fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40aaffd700 1 -- 172.21.15.57:0/1969350514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40ac1a02a0 con 0x7f40ac06bae0 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40ab7fe700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac06b140 0x7f40ac19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40abfff700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac108a60 0x7f40ac19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40aaffd700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac19b7d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e9a0 tx=0x7f40a000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a000ca10 con 0x7f40ac06bae0 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40ac1a0590 con 0x7f40ac06bae0 2024-04-21T08:45:20.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40a000cbb0 con 0x7f40ac06bae0 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0018680 con 0x7f40ac06bae0 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.458+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40ac1a7e10 con 0x7f40ac06bae0 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.459+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40ac1041a0 con 0x7f40ac06bae0 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.460+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f40a0010430 con 0x7f40ac06bae0 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.461+0000 7f40a8ff9700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 0x7f40940a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.461+0000 7f40ab7fe700 1 -- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 msgr2=0x7f40940a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.462+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f40a00d0c70 con 0x7f40ac06bae0 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.462+0000 7f40ab7fe700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 0x7f40940a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:20.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.465+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40a009d270 con 0x7f40ac06bae0 2024-04-21T08:45:20.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.663+0000 7f40ab7fe700 1 -- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 msgr2=0x7f40940a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:20.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.663+0000 7f40ab7fe700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 0x7f40940a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:45:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.725+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f40ac02df80 con 0x7f40ac06bae0 2024-04-21T08:45:20.725 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:20.669+0000 7fa300c7d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:45:20.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.726+0000 7f40a8ff9700 1 -- 172.21.15.57:0/1969350514 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+88 (secure 0 0 0) 0x7f40a009cb70 con 0x7f40ac06bae0 2024-04-21T08:45:20.726 INFO:teuthology.orchestra.run.smithi057.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-21T08:45:20.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 msgr2=0x7f40940a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 0x7f40940a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 msgr2=0x7f40ac19b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac19b7d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f40a000e9a0 tx=0x7f40a000ecb0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 shutdown_connections 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f40940a3c90 0x7f40940a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40ac108a60 0x7f40ac19fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f40ac06b140 0x7f40ac19b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 --2- 172.21.15.57:0/1969350514 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f40ac06bae0 0x7f40ac19b7d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 >> 172.21.15.57:0/1969350514 conn(0x7f40ac078980 msgr2=0x7f40ac10de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 shutdown_connections 2024-04-21T08:45:20.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:20.728+0000 7f40b1ab5700 1 -- 172.21.15.57:0/1969350514 wait complete. 2024-04-21T08:45:20.781 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:20.400+0000 7f3adc01e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:45:20.782 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:20.593+0000 7f3adc01e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:45:20.782 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:20.387+0000 7f2bca73d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-21T08:45:20.782 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:20.582+0000 7f2bca73d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:45:21.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:20 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1969350514' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:45:21.088 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:20 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1969350514' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:45:21.089 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:20.885+0000 7fa300c7d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-21T08:45:21.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:20 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1969350514' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi057.front.sepia.ceph.com 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-21T08:45:21.475 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:45:21.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:21.476 DEBUG:teuthology.orchestra.run.smithi057:> ip addr 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: inet 127.0.0.1/8 scope host lo 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 ::1/128 scope host 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:86 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f0 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:45:21.492 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:6b:ff:87 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp7s0f1 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:62 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f0 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: inet 172.21.15.57/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft 42071sec preferred_lft 42071sec 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: inet6 fe80::ec4:7aff:febd:1562/64 scope link noprefixroute 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: valid_lft forever preferred_lft forever 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: link/ether 0c:c4:7a:bd:15:63 brd ff:ff:ff:ff:ff:ff 2024-04-21T08:45:21.493 INFO:teuthology.orchestra.run.smithi057.stdout: altname enp1s0f1 2024-04-21T08:45:21.494 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add name ceph-brx type bridge 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr flush dev ceph-brx 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set ceph-brx up 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-21T08:45:21.494 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:45:21.573 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:21.196+0000 7f3adc01e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:45:21.582 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:45:21.605 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:21.180+0000 7f2bca73d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:45:21.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:21.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:21.636 DEBUG:teuthology.orchestra.run.smithi057:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-21T08:45:21.661 INFO:teuthology.orchestra.run.smithi057.stdout:1 2024-04-21T08:45:21.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:21.663 DEBUG:teuthology.orchestra.run.smithi057:> route 2024-04-21T08:45:21.719 INFO:teuthology.orchestra.run.smithi057.stdout:Kernel IP routing table 2024-04-21T08:45:21.720 INFO:teuthology.orchestra.run.smithi057.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-21T08:45:21.720 INFO:teuthology.orchestra.run.smithi057.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-21T08:45:21.720 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-21T08:45:21.720 INFO:teuthology.orchestra.run.smithi057.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-21T08:45:21.720 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:45:21.720 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:45:21.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-21T08:45:21.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-21T08:45:21.721 DEBUG:teuthology.orchestra.run.smithi057:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-21T08:45:21.721 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:45:21.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:45:21.818 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:21.522+0000 7fa300c7d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-21T08:45:21.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:21.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:21.850 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2024-04-21T08:45:21.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:21.866 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list-id 2024-04-21T08:45:21.885 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:21.605+0000 7f2bca73d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:45:21.885 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:21.702+0000 7f2bca73d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:45:21.912 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:21.617+0000 7f3adc01e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:45:21.912 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:21.714+0000 7f3adc01e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:45:21.921 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:45:21.921 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:45:21.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:45:21.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-21T08:45:21.921 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:45:22.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:45:22.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.041 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-21T08:45:22.041 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:45:22.090 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:21.991+0000 7fa300c7d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-21T08:45:22.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:45:22.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.156 DEBUG:teuthology.orchestra.run.smithi057:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-21T08:45:22.156 DEBUG:teuthology.orchestra.run.smithi057:> set -e 2024-04-21T08:45:22.157 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set brx.0 up 2024-04-21T08:45:22.157 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link set dev brx.0 master ceph-brx 2024-04-21T08:45:22.157 DEBUG:teuthology.orchestra.run.smithi057:> ') 2024-04-21T08:45:22.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:45:22.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.235 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:45:22.235 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:22.251 INFO:teuthology.orchestra.run.smithi057.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-21T08:45:22.251 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-21T08:45:22.252 DEBUG:teuthology.orchestra.run.smithi057:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:22.309 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-21T08:45:22.309 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-21T08:45:22.310 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/0d190491-43b7-4ad2-b994-7f7858bb7cfc 2024-04-21T08:45:22.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:22.310 DEBUG:teuthology.orchestra.run.smithi057:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/0d190491-43b7-4ad2-b994-7f7858bb7cfc /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-21T08:45:22.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:21.913+0000 7f3adc01e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:45:22.370 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:22.098+0000 7fa300c7d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-21T08:45:22.370 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:22.329+0000 7fa300c7d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:45:22.384 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:21.903+0000 7f2bca73d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-21T08:45:22.391 INFO:teuthology.orchestra.run.smithi057.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-21T08:45:22.391 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-04-21T08:45:22.510 INFO:teuthology.orchestra.run.smithi057.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-04-21T08:45:22.510 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-04-21T08:45:22.510 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: resolved to: "172.21.15.57:3300,172.21.15.129:3300,172.21.15.136:3300" 2024-04-21T08:45:22.510 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: trying mount with new device syntax: 0@df0fb3ae-ffb9-11ee-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/0d190491-43b7-4ad2-b994-7f7858bb7cfc 2024-04-21T08:45:22.510 INFO:teuthology.orchestra.run.smithi057.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.57:3300/172.21.15.129:3300/172.21.15.136:3300" will pass to kernel 2024-04-21T08:45:22.511 INFO:teuthology.orchestra.run.smithi057.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-21T08:45:22.511 INFO:teuthology.orchestra.run.smithi057.stdout: You just mounted a file system that supports labels which does not 2024-04-21T08:45:22.511 INFO:teuthology.orchestra.run.smithi057.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-21T08:45:22.511 INFO:teuthology.orchestra.run.smithi057.stdout: applications will generate AVC messages and not be allowed access to 2024-04-21T08:45:22.511 INFO:teuthology.orchestra.run.smithi057.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-21T08:45:22.512 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-21T08:45:22.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:22.513 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:22.580 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:45:22.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:45:22.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.679 INFO:teuthology.orchestra.run.smithi057.stdout:client24979 2024-04-21T08:45:22.679 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:45:22.679 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:45:22.695 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:45:22.759 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:22.423+0000 7f2bca73d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:45:22.759 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:22.517+0000 7f2bca73d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:45:22.775 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:22.437+0000 7f3adc01e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:45:22.775 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:22.531+0000 7f3adc01e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:45:22.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:45:22.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.794 INFO:teuthology.orchestra.run.smithi057.stdout:client24979 2024-04-21T08:45:22.795 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-21T08:45:22.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-21T08:45:22.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.853 INFO:teuthology.orchestra.run.smithi057.stdout:client24979 2024-04-21T08:45:22.853 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968.client24979/status') 2024-04-21T08:45:22.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-21T08:45:22.912 INFO:teuthology.orchestra.run.smithi057.stdout:instance: client.24979 (3)172.21.15.57:0/1420736868 2024-04-21T08:45:22.913 INFO:teuthology.orchestra.run.smithi057.stdout:blocklisted: false 2024-04-21T08:45:22.913 INFO:teuthology.orchestra.run.smithi057.stderr:0+1 records in 2024-04-21T08:45:22.913 INFO:teuthology.orchestra.run.smithi057.stderr:0+1 records out 2024-04-21T08:45:22.913 INFO:teuthology.orchestra.run.smithi057.stderr:71 bytes copied, 2.4057e-05 s, 3.0 MB/s 2024-04-21T08:45:22.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-21T08:45:22.920 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-21T08:45:22.932 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-21T08:45:22.957 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/275041333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8105320 msgr2=0x7fd3e8107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/275041333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8105320 0x7fd3e8107710 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009900 tx=0x7fd3dc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/275041333 shutdown_connections 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/275041333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3e8107c50 0x7fd3e810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/275041333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8105320 0x7fd3e8107710 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/275041333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3e81029f0 0x7fd3e8104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/275041333 >> 172.21.15.57:0/275041333 conn(0x7fd3e80fc3e0 msgr2=0x7fd3e80fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/275041333 shutdown_connections 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.062+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/275041333 wait complete. 2024-04-21T08:45:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 Processor -- start 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 -- start start 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3e81029f0 0x7fd3e819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3e8105320 0x7fd3e819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 0x7fd3e81a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e810daf0 con 0x7fd3e8107c50 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e810d970 con 0x7fd3e8105320 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3ecfb2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e810dc70 con 0x7fd3e81029f0 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 0x7fd3e81a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 0x7fd3e81a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45876/0 (socket says 172.21.15.57:45876) 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 -- 172.21.15.57:0/559182880 learned_addr learned my addr 172.21.15.57:0/559182880 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e659c640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3e81029f0 0x7fd3e819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e5d9b640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3e8105320 0x7fd3e819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 -- 172.21.15.57:0/559182880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3e81029f0 msgr2=0x7fd3e819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3e81029f0 0x7fd3e819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 -- 172.21.15.57:0/559182880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3e8105320 msgr2=0x7fd3e819a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3e8105320 0x7fd3e819a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 -- 172.21.15.57:0/559182880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e81a2020 con 0x7fd3e8107c50 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.063+0000 7fd3e6d9d640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 0x7fd3e81a19b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000eb10 tx=0x7fd3e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.064+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0004b20 con 0x7fd3e8107c50 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.064+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3e0004cc0 con 0x7fd3e8107c50 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.064+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e81a2310 con 0x7fd3e8107c50 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.064+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0010670 con 0x7fd3e8107c50 2024-04-21T08:45:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.064+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e81a27e0 con 0x7fd3e8107c50 2024-04-21T08:45:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.065+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e81097a0 con 0x7fd3e8107c50 2024-04-21T08:45:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.066+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fd3e0005000 con 0x7fd3e8107c50 2024-04-21T08:45:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.067+0000 7fd3d77fe640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd3d00a3c00 0x7fd3d00a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.067+0000 7fd3e659c640 1 -- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd3d00a3c00 msgr2=0x7fd3d00a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.067+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fd3e00d0fd0 con 0x7fd3e8107c50 2024-04-21T08:45:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.067+0000 7fd3e659c640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd3d00a3c00 0x7fd3d00a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.071+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3e0053ff0 con 0x7fd3e8107c50 2024-04-21T08:45:23.073 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:22.776+0000 7f3adc01e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:45:23.073 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:22.908+0000 7f3adc01e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:45:23.073 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:23.002+0000 7f3adc01e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:45:23.100 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:22.759+0000 7f2bca73d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:45:23.100 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:22.891+0000 7f2bca73d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:45:23.100 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:22.985+0000 7f2bca73d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:45:23.100 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:23.100+0000 7f2bca73d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:45:23.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.260+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3e8062b50 con 0x7fd3e8107c50 2024-04-21T08:45:23.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.262+0000 7fd3d77fe640 1 -- 172.21.15.57:0/559182880 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7fd3e009d5d0 con 0x7fd3e8107c50 2024-04-21T08:45:23.262 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:23.262 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:23.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd3d00a3c00 msgr2=0x7fd3d00a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:23.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd3d00a3c00 0x7fd3d00a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 msgr2=0x7fd3e81a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 0x7fd3e81a19b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000eb10 tx=0x7fd3e000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 shutdown_connections 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fd3d00a3c00 0x7fd3d00a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3e8107c50 0x7fd3e81a19b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3e8105320 0x7fd3e819a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 --2- 172.21.15.57:0/559182880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3e81029f0 0x7fd3e819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 >> 172.21.15.57:0/559182880 conn(0x7fd3e80fc3e0 msgr2=0x7fd3e8102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 shutdown_connections 2024-04-21T08:45:23.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.264+0000 7fd3ecfb2640 1 -- 172.21.15.57:0/559182880 wait complete. 2024-04-21T08:45:23.266 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:22.899+0000 7fa300c7d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-21T08:45:23.266 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:23.001+0000 7fa300c7d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-21T08:45:23.266 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:23.265+0000 7fa300c7d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-21T08:45:23.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:23.277 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:23.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/559182880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/464997748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 msgr2=0x7f8eb81092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/464997748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb81092c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009ca0 tx=0x7f8eb002c140 comp rx=0 tx=0).stop 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/464997748 shutdown_connections 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/464997748 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb81092c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/464997748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8eb8101fe0 0x7f8eb8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/464997748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eb8100e90 0x7f8eb81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/464997748 >> 172.21.15.57:0/464997748 conn(0x7f8eb80fc400 msgr2=0x7f8eb80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:23.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/464997748 shutdown_connections 2024-04-21T08:45:23.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/464997748 wait complete. 2024-04-21T08:45:23.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 Processor -- start 2024-04-21T08:45:23.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.386+0000 7f8ebf4c8640 1 -- start start 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8eb8100e90 0x7f8eb8069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eb8101fe0 0x7f8eb819d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb810cb00 con 0x7f8eb81029a0 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb810c980 con 0x7f8eb8100e90 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb810cc80 con 0x7f8eb8101fe0 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45908/0 (socket says 172.21.15.57:45908) 2024-04-21T08:45:23.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 -- 172.21.15.57:0/312576263 learned_addr learned my addr 172.21.15.57:0/312576263 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebd264640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8eb8100e90 0x7f8eb8069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 -- 172.21.15.57:0/312576263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eb8101fe0 msgr2=0x7f8eb819d0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eb8101fe0 0x7f8eb819d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 -- 172.21.15.57:0/312576263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8eb8100e90 msgr2=0x7f8eb8069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8eb8100e90 0x7f8eb8069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 -- 172.21.15.57:0/312576263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb81a0000 con 0x7f8eb81029a0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebda65640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb819f6f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0005cc0 tx=0x7f8eb0035ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0039040 con 0x7f8eb81029a0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb81a0230 con 0x7f8eb81029a0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.387+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eb0044070 con 0x7f8eb81029a0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.388+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0004640 con 0x7f8eb81029a0 2024-04-21T08:45:23.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.388+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb81a0770 con 0x7f8eb81029a0 2024-04-21T08:45:23.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.388+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8eb8108330 con 0x7f8eb81029a0 2024-04-21T08:45:23.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.390+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f8eb0003970 con 0x7f8eb81029a0 2024-04-21T08:45:23.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.390+0000 7f8eae7fc640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8ea40a3c00 0x7f8ea40a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.391+0000 7f8ebd264640 1 -- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8ea40a3c00 msgr2=0x7f8ea40a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:23.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.391+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f8eb00f38c0 con 0x7f8eb81029a0 2024-04-21T08:45:23.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.391+0000 7f8ebd264640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8ea40a3c00 0x7f8ea40a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:23.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.398+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8eb00bff20 con 0x7f8eb81029a0 2024-04-21T08:45:23.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/559182880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:23.422 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:23.116+0000 7f3adc01e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:45:23.521 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/559182880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:23.522 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:23.418+0000 7fa300c7d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-21T08:45:23.522 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:23.522+0000 7fa300c7d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-21T08:45:23.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.586+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8eb802d810 con 0x7f8eb81029a0 2024-04-21T08:45:23.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.587+0000 7f8eae7fc640 1 -- 172.21.15.57:0/312576263 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f8eb00bf820 con 0x7f8eb81029a0 2024-04-21T08:45:23.586 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8ea40a3c00 msgr2=0x7f8ea40a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8ea40a3c00 0x7f8ea40a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 msgr2=0x7f8eb819f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb819f6f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0005cc0 tx=0x7f8eb0035ea0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 shutdown_connections 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f8ea40a3c00 0x7f8ea40a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8eb81029a0 0x7f8eb819f6f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8eb8101fe0 0x7f8eb819d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 --2- 172.21.15.57:0/312576263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8eb8100e90 0x7f8eb8069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 >> 172.21.15.57:0/312576263 conn(0x7f8eb80fc400 msgr2=0x7f8eb8105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:23.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 shutdown_connections 2024-04-21T08:45:23.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.589+0000 7f8ebf4c8640 1 -- 172.21.15.57:0/312576263 wait complete. 2024-04-21T08:45:23.601 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:23.710 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 -- 172.21.15.57:0/2097734176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 msgr2=0x7f0ce4107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.710 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/2097734176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4107730 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009900 tx=0x7f0ccc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 -- 172.21.15.57:0/2097734176 shutdown_connections 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/2097734176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ce4107c70 0x7f0ce410a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/2097734176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4107730 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/2097734176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 0x7f0ce4104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 -- 172.21.15.57:0/2097734176 >> 172.21.15.57:0/2097734176 conn(0x7f0ce40fc400 msgr2=0x7f0ce40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 -- 172.21.15.57:0/2097734176 shutdown_connections 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 -- 172.21.15.57:0/2097734176 wait complete. 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 Processor -- start 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.709+0000 7f0cea1ce640 1 -- start start 2024-04-21T08:45:23.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cea1ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 0x7f0ce4198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cea1ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cea1ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ce4107c70 0x7f0ce419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cea1ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce410dac0 con 0x7f0ce4105340 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cea1ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce410d940 con 0x7f0ce4107c70 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cea1ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce410dc40 con 0x7f0ce4102a10 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45934/0 (socket says 172.21.15.57:45934) 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 -- 172.21.15.57:0/339272208 learned_addr learned my addr 172.21.15.57:0/339272208 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0ce37fe640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 0x7f0ce4198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0ce3fff640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ce4107c70 0x7f0ce419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:23.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 -- 172.21.15.57:0/339272208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 msgr2=0x7f0ce4198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 0x7f0ce4198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 -- 172.21.15.57:0/339272208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ce4107c70 msgr2=0x7f0ce419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ce4107c70 0x7f0ce419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 -- 172.21.15.57:0/339272208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce419fe20 con 0x7f0ce4105340 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0ce37fe640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 0x7f0ce4198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.710+0000 7f0cdaffd640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4198740 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc000c00 tx=0x7f0ccc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.711+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc043020 con 0x7f0ce4105340 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.711+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce4100800 con 0x7f0ce4105340 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.711+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ccc004350 con 0x7f0ce4105340 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.711+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc003b80 con 0x7f0ce4105340 2024-04-21T08:45:23.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.711+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce4100d50 con 0x7f0ce4105340 2024-04-21T08:45:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.712+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce40fdef0 con 0x7f0ce4105340 2024-04-21T08:45:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.713+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f0ccc046020 con 0x7f0ce4105340 2024-04-21T08:45:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.713+0000 7f0ce17fa640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0cd00a3c00 0x7f0cd00a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:23.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.714+0000 7f0ce37fe640 1 -- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0cd00a3c00 msgr2=0x7f0cd00a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.714+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f0ccc0f2940 con 0x7f0ce4105340 2024-04-21T08:45:23.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.714+0000 7f0ce37fe640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0cd00a3c00 0x7f0cd00a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:23.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.718+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ccc04d020 con 0x7f0ce4105340 2024-04-21T08:45:23.823 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:23.423+0000 7f3adc01e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:45:23.838 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:23.651+0000 7fa300c7d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-21T08:45:23.884 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:23.404+0000 7f2bca73d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:45:23.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.904+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0ce41020a0 con 0x7f0ce4105340 2024-04-21T08:45:23.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.904+0000 7f0ce17fa640 1 -- 172.21.15.57:0/339272208 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f0ccc0bef40 con 0x7f0ce4105340 2024-04-21T08:45:23.904 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:23.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0cd00a3c00 msgr2=0x7f0cd00a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:23.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0cd00a3c00 0x7f0cd00a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 msgr2=0x7f0ce4198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:23.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4198740 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc000c00 tx=0x7f0ccc005dc0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 shutdown_connections 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f0cd00a3c00 0x7f0cd00a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ce4107c70 0x7f0ce419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ce4105340 0x7f0ce4198740 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 --2- 172.21.15.57:0/339272208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ce4102a10 0x7f0ce4198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 >> 172.21.15.57:0/339272208 conn(0x7f0ce40fc400 msgr2=0x7f0ce41048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 shutdown_connections 2024-04-21T08:45:23.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:23.906+0000 7f0cea1ce640 1 -- 172.21.15.57:0/339272208 wait complete. 2024-04-21T08:45:23.917 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 -- 172.21.15.57:0/2669654546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c105340 msgr2=0x7f061c107730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 --2- 172.21.15.57:0/2669654546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c105340 0x7f061c107730 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f060c009900 tx=0x7f060c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 -- 172.21.15.57:0/2669654546 shutdown_connections 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 --2- 172.21.15.57:0/2669654546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f061c107c70 0x7f061c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 --2- 172.21.15.57:0/2669654546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c105340 0x7f061c107730 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 --2- 172.21.15.57:0/2669654546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f061c102a10 0x7f061c104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 -- 172.21.15.57:0/2669654546 >> 172.21.15.57:0/2669654546 conn(0x7f061c0fc400 msgr2=0x7f061c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.024+0000 7f06245f4640 1 -- 172.21.15.57:0/2669654546 shutdown_connections 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 -- 172.21.15.57:0/2669654546 wait complete. 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 Processor -- start 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 -- start start 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f061c102a10 0x7f061c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f061c105340 0x7f061c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 0x7f061c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f061c10dac0 con 0x7f061c107c70 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f061c10d940 con 0x7f061c105340 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f06245f4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f061c10dc40 con 0x7f061c102a10 2024-04-21T08:45:24.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f0622b91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 0x7f061c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f0622390640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f061c102a10 0x7f061c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f0622b91640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 0x7f061c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45962/0 (socket says 172.21.15.57:45962) 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f0622b91640 1 -- 172.21.15.57:0/2341890843 learned_addr learned my addr 172.21.15.57:0/2341890843 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.025+0000 7f0621b8f640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f061c105340 0x7f061c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f0622b91640 1 -- 172.21.15.57:0/2341890843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f061c102a10 msgr2=0x7f061c198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f0622b91640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f061c102a10 0x7f061c198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f0622b91640 1 -- 172.21.15.57:0/2341890843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f061c105340 msgr2=0x7f061c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f0622b91640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f061c105340 0x7f061c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f0622b91640 1 -- 172.21.15.57:0/2341890843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c19fe20 con 0x7f061c107c70 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f0622b91640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 0x7f061c19f7b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f061400eb10 tx=0x7f061400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614004b20 con 0x7f061c107c70 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0614004cc0 con 0x7f061c107c70 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c1a0110 con 0x7f061c107c70 2024-04-21T08:45:24.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614010670 con 0x7f061c107c70 2024-04-21T08:45:24.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.026+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c1a05e0 con 0x7f061c107c70 2024-04-21T08:45:24.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.027+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061c0fdef0 con 0x7f061c107c70 2024-04-21T08:45:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.028+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f0614005020 con 0x7f061c107c70 2024-04-21T08:45:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.029+0000 7f06137fe640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 0x7f06080a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.029+0000 7f0622390640 1 -- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 msgr2=0x7f06080a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.029+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f06140d0d90 con 0x7f061c107c70 2024-04-21T08:45:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.029+0000 7f0622390640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 0x7f06080a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:24.033 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.033+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f061409d3f0 con 0x7f061c107c70 2024-04-21T08:45:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.230+0000 7f0622390640 1 -- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 msgr2=0x7f06080a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.230+0000 7f0622390640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 0x7f06080a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-21T08:45:24.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.247+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f061c062b50 con 0x7f061c107c70 2024-04-21T08:45:24.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.247+0000 7f06137fe640 1 -- 172.21.15.57:0/2341890843 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f061409ccf0 con 0x7f061c107c70 2024-04-21T08:45:24.247 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:24.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 msgr2=0x7f06080a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:24.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 0x7f06080a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 msgr2=0x7f061c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 0x7f061c19f7b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f061400eb10 tx=0x7f061400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 shutdown_connections 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f06080a3c00 0x7f06080a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f061c107c70 0x7f061c19f7b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f061c105340 0x7f061c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 --2- 172.21.15.57:0/2341890843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f061c102a10 0x7f061c198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.249+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 >> 172.21.15.57:0/2341890843 conn(0x7f061c0fc400 msgr2=0x7f061c1048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.250+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 shutdown_connections 2024-04-21T08:45:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.250+0000 7f06245f4640 1 -- 172.21.15.57:0/2341890843 wait complete. 2024-04-21T08:45:24.259 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:23.973+0000 7fa300c7d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-21T08:45:24.260 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:24.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:24.027+0000 7f3adc01e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:45:24.323 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y[32667]: 2024-04-21T08:45:24.241+0000 7f3adc01e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:45:24.332 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:24.005+0000 7f2bca73d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:45:24.332 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z[33644]: 2024-04-21T08:45:24.220+0000 7f2bca73d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 -- 172.21.15.57:0/129788026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6481029a0 msgr2=0x7fc6481092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/129788026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6481029a0 0x7fc6481092c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc640009900 tx=0x7fc64002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 -- 172.21.15.57:0/129788026 shutdown_connections 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/129788026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6481029a0 0x7fc6481092c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/129788026 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc648101fe0 0x7fc648102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/129788026 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 0x7fc6481012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 -- 172.21.15.57:0/129788026 >> 172.21.15.57:0/129788026 conn(0x7fc6480fc400 msgr2=0x7fc6480fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 -- 172.21.15.57:0/129788026 shutdown_connections 2024-04-21T08:45:24.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.369+0000 7fc64ce8c640 1 -- 172.21.15.57:0/129788026 wait complete. 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 Processor -- start 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 -- start start 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 0x7fc648069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 0x7fc64819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6481029a0 0x7fc6481a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc64810c9d0 con 0x7fc648101fe0 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc64810c850 con 0x7fc6481029a0 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64ce8c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc64810cb50 con 0x7fc648100e90 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc645d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 0x7fc64819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc645d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 0x7fc64819f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45972/0 (socket says 172.21.15.57:45972) 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc645d9b640 1 -- 172.21.15.57:0/1437000328 learned_addr learned my addr 172.21.15.57:0/1437000328 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.370+0000 7fc64659c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 0x7fc648069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc646d9d640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6481029a0 0x7fc6481a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc645d9b640 1 -- 172.21.15.57:0/1437000328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 msgr2=0x7fc648069c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc645d9b640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 0x7fc648069c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc645d9b640 1 -- 172.21.15.57:0/1437000328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6481029a0 msgr2=0x7fc6481a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc645d9b640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6481029a0 0x7fc6481a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc645d9b640 1 -- 172.21.15.57:0/1437000328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6481a2050 con 0x7fc648101fe0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc64659c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 0x7fc648069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc645d9b640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 0x7fc64819f310 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc63000e9c0 tx=0x7fc63000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63000cd80 con 0x7fc648101fe0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6300035f0 con 0x7fc648101fe0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6481a22e0 con 0x7fc648101fe0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630010690 con 0x7fc648101fe0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.371+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6481a2830 con 0x7fc648101fe0 2024-04-21T08:45:24.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.372+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc648108290 con 0x7fc648101fe0 2024-04-21T08:45:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.373+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7fc6300037a0 con 0x7fc648101fe0 2024-04-21T08:45:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.374+0000 7fc63f7fe640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc6340a3c30 0x7fc6340a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.374+0000 7fc64659c640 1 -- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc6340a3c30 msgr2=0x7fc6340a60f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.374+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc6300d0b00 con 0x7fc648101fe0 2024-04-21T08:45:24.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.374+0000 7fc64659c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc6340a3c30 0x7fc6340a60f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:24.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.379+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc63009d160 con 0x7fc648101fe0 2024-04-21T08:45:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.565+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc648062b80 con 0x7fc648101fe0 2024-04-21T08:45:24.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.565+0000 7fc63f7fe640 1 -- 172.21.15.57:0/1437000328 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fc63009ca60 con 0x7fc648101fe0 2024-04-21T08:45:24.565 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc6340a3c30 msgr2=0x7fc6340a60f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc6340a3c30 0x7fc6340a60f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 msgr2=0x7fc64819f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 0x7fc64819f310 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc63000e9c0 tx=0x7fc63000ee80 comp rx=0 tx=0).stop 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 shutdown_connections 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7fc6340a3c30 0x7fc6340a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6481029a0 0x7fc6481a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc648101fe0 0x7fc64819f310 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 --2- 172.21.15.57:0/1437000328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc648100e90 0x7fc648069c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 >> 172.21.15.57:0/1437000328 conn(0x7fc6480fc400 msgr2=0x7fc648105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 shutdown_connections 2024-04-21T08:45:24.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.567+0000 7fc64ce8c640 1 -- 172.21.15.57:0/1437000328 wait complete. 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/312576263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/339272208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: Standby manager daemon z restarted 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: Standby manager daemon z started 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:45:24.577 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: Standby manager daemon y restarted 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: Standby manager daemon y started 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:45:24.578 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2341890843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.579 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/312576263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/339272208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: Standby manager daemon z restarted 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: Standby manager daemon z started 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: Standby manager daemon y restarted 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: Standby manager daemon y started 2024-04-21T08:45:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:45:24.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:45:24.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:45:24.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:45:24.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:24 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2341890843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.683 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 -- 172.21.15.57:0/1451885852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850100e60 msgr2=0x7f8850101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1451885852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850100e60 0x7f8850101280 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8840009900 tx=0x7f884002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 -- 172.21.15.57:0/1451885852 shutdown_connections 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1451885852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850102a00 0x7f8850109320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1451885852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8850102040 0x7f88501024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1451885852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850100e60 0x7f8850101280 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.683+0000 7f8854ae4640 1 -- 172.21.15.57:0/1451885852 >> 172.21.15.57:0/1451885852 conn(0x7f88500fc420 msgr2=0x7f88500fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 -- 172.21.15.57:0/1451885852 shutdown_connections 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 -- 172.21.15.57:0/1451885852 wait complete. 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 Processor -- start 2024-04-21T08:45:24.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 -- start start 2024-04-21T08:45:24.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8850100e60 0x7f88501981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850102040 0x7f8850198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.684+0000 7f8854ae4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 0x7f885019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f8854ae4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f885010c9d0 con 0x7f8850102a00 2024-04-21T08:45:24.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f8854ae4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f885010c850 con 0x7f8850102040 2024-04-21T08:45:24.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f8854ae4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f885010cb50 con 0x7f8850100e60 2024-04-21T08:45:24.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 0x7f885019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 0x7f885019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45994/0 (socket says 172.21.15.57:45994) 2024-04-21T08:45:24.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 -- 172.21.15.57:0/1030061332 learned_addr learned my addr 172.21.15.57:0/1030061332 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:24.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884e59c640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8850100e60 0x7f88501981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884dd9b640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850102040 0x7f8850198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:24.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 -- 172.21.15.57:0/1030061332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8850100e60 msgr2=0x7f88501981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8850100e60 0x7f88501981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 -- 172.21.15.57:0/1030061332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850102040 msgr2=0x7f8850198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850102040 0x7f8850198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 -- 172.21.15.57:0/1030061332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885019fe00 con 0x7f8850102a00 2024-04-21T08:45:24.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f884ed9d640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 0x7f885019f790 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f884800eb10 tx=0x7f884800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:24.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848004b20 con 0x7f8850102a00 2024-04-21T08:45:24.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8848004cc0 con 0x7f8850102a00 2024-04-21T08:45:24.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88501a00f0 con 0x7f8850102a00 2024-04-21T08:45:24.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848010670 con 0x7f8850102a00 2024-04-21T08:45:24.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.685+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88501a05f0 con 0x7f8850102a00 2024-04-21T08:45:24.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.687+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88501a08b0 con 0x7f8850102a00 2024-04-21T08:45:24.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.689+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7f8848005020 con 0x7f8850102a00 2024-04-21T08:45:24.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.689+0000 7f883f7fe640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f88380a3c00 0x7f88380a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:24.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.689+0000 7f884e59c640 1 -- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f88380a3c00 msgr2=0x7f88380a60c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.57:6800/3002879345 2024-04-21T08:45:24.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.689+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f88480d0d80 con 0x7f8850102a00 2024-04-21T08:45:24.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.689+0000 7f884e59c640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f88380a3c00 0x7f88380a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-21T08:45:24.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.694+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f884809d3e0 con 0x7f8850102a00 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/312576263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/339272208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: Standby manager daemon z restarted 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: Standby manager daemon z started 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.136:0/1961656021' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: Standby manager daemon y restarted 2024-04-21T08:45:24.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: Standby manager daemon y started 2024-04-21T08:45:24.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-21T08:45:24.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-21T08:45:24.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-21T08:45:24.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='mgr.? 172.21.15.129:0/2335940328' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-21T08:45:24.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:24 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2341890843' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:24.838 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:24.659+0000 7fa300c7d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-21T08:45:24.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.881+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88501a0a80 con 0x7f8850102a00 2024-04-21T08:45:24.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.882+0000 7f883f7fe640 1 -- 172.21.15.57:0/1030061332 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24697 (secure 0 0 0) 0x7f884809cce0 con 0x7f8850102a00 2024-04-21T08:45:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:24.882 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":93,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:44:47.297032+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f88380a3c00 msgr2=0x7f88380a60c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f88380a3c00 0x7f88380a60c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 msgr2=0x7f885019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 0x7f885019f790 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f884800eb10 tx=0x7f884800c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 shutdown_connections 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:6800/3002879345,v1:172.21.15.57:6801/3002879345] conn(0x7f88380a3c00 0x7f88380a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8850102a00 0x7f885019f790 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850102040 0x7f8850198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 --2- 172.21.15.57:0/1030061332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8850100e60 0x7f88501981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 >> 172.21.15.57:0/1030061332 conn(0x7f88500fc420 msgr2=0x7f885006a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 shutdown_connections 2024-04-21T08:45:24.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:24.884+0000 7f8854ae4640 1 -- 172.21.15.57:0/1030061332 wait complete. 2024-04-21T08:45:24.895 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:24.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:25.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.016+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2976799037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8100610 msgr2=0x7fe1e8100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.016+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2976799037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8100610 0x7fe1e8100a30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009900 tx=0x7fe1d802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2976799037 shutdown_connections 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2976799037 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1e81021c0 0x7fe1e8108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2976799037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8101800 0x7fe1e8101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2976799037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8100610 0x7fe1e8100a30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2976799037 >> 172.21.15.57:0/2976799037 conn(0x7fe1e80fbb80 msgr2=0x7fe1e80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2976799037 shutdown_connections 2024-04-21T08:45:25.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2976799037 wait complete. 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.017+0000 7fe1edd2b640 1 Processor -- start 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 -- start start 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8100610 0x7fe1e8111300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8101800 0x7fe1e8111840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1e81021c0 0x7fe1e81168a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810afa0 con 0x7fe1e81021c0 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810ae20 con 0x7fe1e8101800 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1edd2b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810b120 con 0x7fe1e8100610 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1e77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8100610 0x7fe1e8111300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1e6ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8101800 0x7fe1e8111840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1e77fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8100610 0x7fe1e8111300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47084/0 (socket says 172.21.15.57:47084) 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1e77fe640 1 -- 172.21.15.57:0/2519024915 learned_addr learned my addr 172.21.15.57:0/2519024915 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.018+0000 7fe1e7fff640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1e81021c0 0x7fe1e81168a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e6ffd640 1 -- 172.21.15.57:0/2519024915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8100610 msgr2=0x7fe1e8111300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e6ffd640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8100610 0x7fe1e8111300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e6ffd640 1 -- 172.21.15.57:0/2519024915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1e81021c0 msgr2=0x7fe1e81168a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e6ffd640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1e81021c0 0x7fe1e81168a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e6ffd640 1 -- 172.21.15.57:0/2519024915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e8116f10 con 0x7fe1e8101800 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e6ffd640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8101800 0x7fe1e8111840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00e9f0 tx=0x7fe1dc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc00ccc0 con 0x7fe1e8101800 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e81aef70 con 0x7fe1e8101800 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.019+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1dc0108e0 con 0x7fe1e8101800 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.020+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc018840 con 0x7fe1e8101800 2024-04-21T08:45:25.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.020+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e81af480 con 0x7fe1e8101800 2024-04-21T08:45:25.022 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.021+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e8105850 con 0x7fe1e8101800 2024-04-21T08:45:25.022 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.022+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fe1dc010430 con 0x7fe1e8101800 2024-04-21T08:45:25.023 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.024+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe1dc0d04b0 con 0x7fe1e8101800 2024-04-21T08:45:25.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.028+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1dc09ce10 con 0x7fe1e8101800 2024-04-21T08:45:25.168 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:24 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:24.895+0000 7fa300c7d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-21T08:45:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.257+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1e802d5c0 con 0x7fe1e8101800 2024-04-21T08:45:25.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.258+0000 7fe1e4ff9640 1 -- 172.21.15.57:0/2519024915 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fe1dc09c710 con 0x7fe1e8101800 2024-04-21T08:45:25.258 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8101800 msgr2=0x7fe1e8111840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8101800 0x7fe1e8111840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00e9f0 tx=0x7fe1dc00edf0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 shutdown_connections 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1e81021c0 0x7fe1e81168a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1e8101800 0x7fe1e8111840 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 --2- 172.21.15.57:0/2519024915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1e8100610 0x7fe1e8111300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 >> 172.21.15.57:0/2519024915 conn(0x7fe1e80fbb80 msgr2=0x7fe1e8102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 shutdown_connections 2024-04-21T08:45:25.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.260+0000 7fe1edd2b640 1 -- 172.21.15.57:0/2519024915 wait complete. 2024-04-21T08:45:25.276 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1437000328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1030061332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: Active manager daemon x restarted 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: Activating manager daemon x 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: mgrmap e21: x(active, starting, since 0.0138851s), standbys: y, z 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:45:25.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:45:25.387 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:45:25.388 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:45:25.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: Manager daemon x is now available 2024-04-21T08:45:25.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:25.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:45:25.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:45:25.389 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2519024915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:25.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 -- 172.21.15.57:0/1112613093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c101b80 msgr2=0x7faa3c102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 --2- 172.21.15.57:0/1112613093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c101b80 0x7faa3c102000 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7faa34009ca0 tx=0x7faa3402c180 comp rx=0 tx=0).stop 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 -- 172.21.15.57:0/1112613093 shutdown_connections 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 --2- 172.21.15.57:0/1112613093 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c101b80 0x7faa3c102000 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 --2- 172.21.15.57:0/1112613093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa3c1011e0 0x7faa3c101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 --2- 172.21.15.57:0/1112613093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa3c100880 0x7faa3c100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 -- 172.21.15.57:0/1112613093 >> 172.21.15.57:0/1112613093 conn(0x7faa3c0fc400 msgr2=0x7faa3c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 -- 172.21.15.57:0/1112613093 shutdown_connections 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.388+0000 7faa40f59640 1 -- 172.21.15.57:0/1112613093 wait complete. 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 Processor -- start 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 -- start start 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa3c100880 0x7faa3c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 0x7faa3c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa3c101b80 0x7faa3c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10fcd0 con 0x7faa3c1011e0 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10fb50 con 0x7faa3c101b80 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa40f59640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c10fe50 con 0x7faa3c100880 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 0x7faa3c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 0x7faa3c19a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46158/0 (socket says 172.21.15.57:46158) 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 -- 172.21.15.57:0/1844843399 learned_addr learned my addr 172.21.15.57:0/1844843399 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa3a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa3c100880 0x7faa3c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa3ad9d640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa3c101b80 0x7faa3c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 -- 172.21.15.57:0/1844843399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa3c100880 msgr2=0x7faa3c19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa3c100880 0x7faa3c19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 -- 172.21.15.57:0/1844843399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa3c101b80 msgr2=0x7faa3c1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa3c101b80 0x7faa3c1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.389+0000 7faa39d9b640 1 -- 172.21.15.57:0/1844843399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c1a1fb0 con 0x7faa3c1011e0 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.390+0000 7faa39d9b640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 0x7faa3c19a8d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faa3000d950 tx=0x7faa3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:25.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.390+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30017070 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.390+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3c1a22a0 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.390+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa30009c30 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.390+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3c1a2840 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.390+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3000f620 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.391+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa3c10b800 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.392+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7faa30010460 con 0x7faa3c1011e0 2024-04-21T08:45:25.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.393+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7faa300cfea0 con 0x7faa3c1011e0 2024-04-21T08:45:25.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.401+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa30009de0 con 0x7faa3c1011e0 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1437000328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1030061332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: Active manager daemon x restarted 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: Activating manager daemon x 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:45:25.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: mgrmap e21: x(active, starting, since 0.0138851s), standbys: y, z 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:45:25.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:45:25.576 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: Manager daemon x is now available 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:45:25.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2519024915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:25.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.168+0000 7fa25fdf0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.168+0000 7fa25fdf0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.168+0000 7fa25fdf0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.168+0000 7fa25fdf0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.168+0000 7fa25fdf0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.589 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.170+0000 7fa262df6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.170+0000 7fa262df6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.170+0000 7fa262df6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.170+0000 7fa262df6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:45:25.170+0000 7fa262df6700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1437000328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1030061332' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:25.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: Active manager daemon x restarted 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: Activating manager daemon x 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: osdmap e94: 12 total, 12 up, 12 in 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: mgrmap e21: x(active, starting, since 0.0138851s), standbys: y, z 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-21T08:45:25.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-21T08:45:25.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: Manager daemon x is now available 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-21T08:45:25.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2519024915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:25.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.593+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faa3c02d5c0 con 0x7faa3c1011e0 2024-04-21T08:45:25.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.594+0000 7faa2b7fe640 1 -- 172.21.15.57:0/1844843399 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7faa3009c150 con 0x7faa3c1011e0 2024-04-21T08:45:25.594 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.596+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 msgr2=0x7faa3c19a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.596+0000 7faa40f59640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 0x7faa3c19a8d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7faa3000d950 tx=0x7faa3000de10 comp rx=0 tx=0).stop 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 shutdown_connections 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa3c101b80 0x7faa3c1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa3c1011e0 0x7faa3c19a8d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 --2- 172.21.15.57:0/1844843399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa3c100880 0x7faa3c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 >> 172.21.15.57:0/1844843399 conn(0x7faa3c0fc400 msgr2=0x7faa3c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:25.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 shutdown_connections 2024-04-21T08:45:25.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.597+0000 7faa40f59640 1 -- 172.21.15.57:0/1844843399 wait complete. 2024-04-21T08:45:25.611 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:25.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 -- 172.21.15.57:0/3543947184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 msgr2=0x7fa3a810a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 --2- 172.21.15.57:0/3543947184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a810a0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0009ca0 tx=0x7fa3a002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 -- 172.21.15.57:0/3543947184 shutdown_connections 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 --2- 172.21.15.57:0/3543947184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a810a0c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 --2- 172.21.15.57:0/3543947184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8105340 0x7fa3a8107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 --2- 172.21.15.57:0/3543947184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8102a10 0x7fa3a8104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 -- 172.21.15.57:0/3543947184 >> 172.21.15.57:0/3543947184 conn(0x7fa3a80fc400 msgr2=0x7fa3a80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 -- 172.21.15.57:0/3543947184 shutdown_connections 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 -- 172.21.15.57:0/3543947184 wait complete. 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 Processor -- start 2024-04-21T08:45:25.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.721+0000 7fa3aef52640 1 -- start start 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3aef52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 0x7fa3a81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3aef52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8105340 0x7fa3a8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3aef52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3aef52640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a810dad0 con 0x7fa3a8107c70 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3aef52640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a810d950 con 0x7fa3a8105340 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3aef52640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a810dc50 con 0x7fa3a8102a10 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3accee640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 0x7fa3a81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3accee640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 0x7fa3a81981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47182/0 (socket says 172.21.15.57:47182) 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3accee640 1 -- 172.21.15.57:0/1543644643 learned_addr learned my addr 172.21.15.57:0/1543644643 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3a7fff640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8105340 0x7fa3a8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 -- 172.21.15.57:0/1543644643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 msgr2=0x7fa3a81981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 0x7fa3a81981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 -- 172.21.15.57:0/1543644643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8105340 msgr2=0x7fa3a8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8105340 0x7fa3a8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 -- 172.21.15.57:0/1543644643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a819fdf0 con 0x7fa3a8107c70 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3accee640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 0x7fa3a81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3a7fff640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8105340 0x7fa3a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.722+0000 7fa3ad4ef640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a819f780 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002c630 tx=0x7fa3a00346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.723+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0041020 con 0x7fa3a8107c70 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.723+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3a0034ca0 con 0x7fa3a8107c70 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.723+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a81a0080 con 0x7fa3a8107c70 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.723+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0039040 con 0x7fa3a8107c70 2024-04-21T08:45:25.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.723+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a81a05c0 con 0x7fa3a8107c70 2024-04-21T08:45:25.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.724+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a80fdef0 con 0x7fa3a8107c70 2024-04-21T08:45:25.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.726+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7fa3a0044020 con 0x7fa3a8107c70 2024-04-21T08:45:25.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.726+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa3a00f22d0 con 0x7fa3a8107c70 2024-04-21T08:45:25.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.734+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3a004b020 con 0x7fa3a8107c70 2024-04-21T08:45:25.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.914+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fa3a00ba280 con 0x7fa3a8107c70 2024-04-21T08:45:25.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.915+0000 7fa3a5ffb640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa39801db50 0x7fa3980adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:25.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.921+0000 7fa3accee640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa39801db50 0x7fa3980adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:25.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.924+0000 7fa3accee640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa39801db50 0x7fa3980adea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa39c011550 tx=0x7fa39c005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:25.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.942+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa3a802d5c0 con 0x7fa3a8107c70 2024-04-21T08:45:25.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.943+0000 7fa3a5ffb640 1 -- 172.21.15.57:0/1543644643 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fa3a00bec30 con 0x7fa3a8107c70 2024-04-21T08:45:25.942 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa39801db50 msgr2=0x7fa3980adea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa39801db50 0x7fa3980adea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa39c011550 tx=0x7fa39c005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 msgr2=0x7fa3a819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a819f780 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa3a002c630 tx=0x7fa3a00346d0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 shutdown_connections 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa39801db50 0x7fa3980adea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3a8107c70 0x7fa3a819f780 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3a8105340 0x7fa3a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 --2- 172.21.15.57:0/1543644643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3a8102a10 0x7fa3a81981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:25.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 >> 172.21.15.57:0/1543644643 conn(0x7fa3a80fc400 msgr2=0x7fa3a81033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:25.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 shutdown_connections 2024-04-21T08:45:25.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:25.945+0000 7fa3aef52640 1 -- 172.21.15.57:0/1543644643 wait complete. 2024-04-21T08:45:25.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:26.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- 172.21.15.57:0/4109530056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 msgr2=0x7f80881012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- 172.21.15.57:0/4109530056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f80881012b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8078009900 tx=0x7f807802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- 172.21.15.57:0/4109530056 shutdown_connections 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- 172.21.15.57:0/4109530056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80881029a0 0x7f80881092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- 172.21.15.57:0/4109530056 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8088101fe0 0x7f8088102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- 172.21.15.57:0/4109530056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f80881012b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- 172.21.15.57:0/4109530056 >> 172.21.15.57:0/4109530056 conn(0x7f80880fc400 msgr2=0x7f80880fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- 172.21.15.57:0/4109530056 shutdown_connections 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- 172.21.15.57:0/4109530056 wait complete. 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 Processor -- start 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- start start 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f8088197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8088101fe0 0x7f8088197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80881029a0 0x7f808819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f808810c9c0 con 0x7f8088100e90 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f808810c840 con 0x7f80881029a0 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.072+0000 7f808f4c1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f808810cb40 con 0x7f8088101fe0 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f8088197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f8088197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46214/0 (socket says 172.21.15.57:46214) 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 -- 172.21.15.57:0/2124313086 learned_addr learned my addr 172.21.15.57:0/2124313086 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808ca5c640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8088101fe0 0x7f8088197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808da5e640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80881029a0 0x7f808819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 -- 172.21.15.57:0/2124313086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8088101fe0 msgr2=0x7f8088197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8088101fe0 0x7f8088197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 -- 172.21.15.57:0/2124313086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80881029a0 msgr2=0x7f808819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80881029a0 0x7f808819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 -- 172.21.15.57:0/2124313086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808819ec50 con 0x7f8088100e90 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808d25d640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f8088197030 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8078036040 tx=0x7f80780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:26.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780044e0 con 0x7f8088100e90 2024-04-21T08:45:26.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80780035f0 con 0x7f8088100e90 2024-04-21T08:45:26.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80880690e0 con 0x7f8088100e90 2024-04-21T08:45:26.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80780476e0 con 0x7f8088100e90 2024-04-21T08:45:26.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.073+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8088069630 con 0x7f8088100e90 2024-04-21T08:45:26.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.074+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8088068d70 con 0x7f8088100e90 2024-04-21T08:45:26.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.075+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f8078003a50 con 0x7f8088100e90 2024-04-21T08:45:26.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.076+0000 7f807e7fc640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80740a39b0 0x7f80740a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.076+0000 7f808ca5c640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80740a39b0 0x7f80740a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.076+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f80780f4c90 con 0x7f8088100e90 2024-04-21T08:45:26.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.077+0000 7f808ca5c640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80740a39b0 0x7f80740a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8088198540 tx=0x7f8084005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:26.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.080+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80780c14e0 con 0x7f8088100e90 2024-04-21T08:45:26.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.267+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8088107e20 con 0x7f8088100e90 2024-04-21T08:45:26.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.268+0000 7f807e7fc640 1 -- 172.21.15.57:0/2124313086 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f80780c0de0 con 0x7f8088100e90 2024-04-21T08:45:26.267 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:26.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80740a39b0 msgr2=0x7f80740a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80740a39b0 0x7f80740a5e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8088198540 tx=0x7f8084005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 msgr2=0x7f8088197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f8088197030 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8078036040 tx=0x7f80780042f0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 shutdown_connections 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80740a39b0 0x7f80740a5e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80881029a0 0x7f808819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8088101fe0 0x7f8088197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 --2- 172.21.15.57:0/2124313086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8088100e90 0x7f8088197030 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 >> 172.21.15.57:0/2124313086 conn(0x7f80880fc400 msgr2=0x7f80881053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 shutdown_connections 2024-04-21T08:45:26.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.270+0000 7f808f4c1640 1 -- 172.21.15.57:0/2124313086 wait complete. 2024-04-21T08:45:26.281 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-21T08:45:26.281 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 -- 172.21.15.57:0/621982564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c101b40 msgr2=0x7f818c101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 --2- 172.21.15.57:0/621982564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c101b40 0x7f818c101fc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8184009ca0 tx=0x7f818402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 -- 172.21.15.57:0/621982564 shutdown_connections 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 --2- 172.21.15.57:0/621982564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c101b40 0x7f818c101fc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 --2- 172.21.15.57:0/621982564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c1011a0 0x7f818c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 --2- 172.21.15.57:0/621982564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 -- 172.21.15.57:0/621982564 >> 172.21.15.57:0/621982564 conn(0x7f818c0fc3e0 msgr2=0x7f818c0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 -- 172.21.15.57:0/621982564 shutdown_connections 2024-04-21T08:45:26.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 -- 172.21.15.57:0/621982564 wait complete. 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 Processor -- start 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.388+0000 7f8191a9a640 1 -- start start 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8191a9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8191a9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 0x7f818c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8191a9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c101b40 0x7f818c1a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8191a9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f818c10fd00 con 0x7f818c1011a0 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8191a9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f818c10fb80 con 0x7f818c100840 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8191a9a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f818c10fe80 con 0x7f818c101b40 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8183fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 0x7f818c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c19a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:38928/0 (socket says 172.21.15.57:38928) 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c101b40 0x7f818c1a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8183fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 0x7f818c19a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46234/0 (socket says 172.21.15.57:46234) 2024-04-21T08:45:26.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 -- 172.21.15.57:0/2394142876 learned_addr learned my addr 172.21.15.57:0/2394142876 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 -- 172.21.15.57:0/2394142876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c101b40 msgr2=0x7f818c1a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c101b40 0x7f818c1a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 -- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 msgr2=0x7f818c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 0x7f818c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 -- 172.21.15.57:0/2394142876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c1a1ff0 con 0x7f818c100840 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f8183fff640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 0x7f818c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818bfff640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c101b40 0x7f818c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.389+0000 7f818b7fe640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c19a3d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b4a0 tx=0x7f817c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.390+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c014070 con 0x7f818c100840 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.390+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817c010ac0 con 0x7f818c100840 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.390+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c017c30 con 0x7f818c100840 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.390+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818c1a22e0 con 0x7f818c100840 2024-04-21T08:45:26.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.390+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818c1a2830 con 0x7f818c100840 2024-04-21T08:45:26.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.391+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f818c10b790 con 0x7f818c100840 2024-04-21T08:45:26.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.392+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f817c017070 con 0x7f818c100840 2024-04-21T08:45:26.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.393+0000 7f81897fa640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f81780a39b0 0x7f81780a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.393+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f817c0cfab0 con 0x7f818c100840 2024-04-21T08:45:26.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.393+0000 7f8183fff640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f81780a39b0 0x7f81780a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.394+0000 7f8183fff640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f81780a39b0 0x7f81780a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f818c19b8e0 tx=0x7f8174005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:26.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.398+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f817c09bcb0 con 0x7f818c100840 2024-04-21T08:45:26.520 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:26 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1844843399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.520 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:26 smithi057 ceph-mon[27968]: mgrmap e22: x(active, since 1.01845s), standbys: y, z 2024-04-21T08:45:26.520 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:26 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1543644643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.520 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:26 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2124313086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:26 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1844843399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:26 smithi129 ceph-mon[31272]: mgrmap e22: x(active, since 1.01845s), standbys: y, z 2024-04-21T08:45:26.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:26 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1543644643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:26 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2124313086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.625+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f818c062b50 con 0x7f818c100840 2024-04-21T08:45:26.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.626+0000 7f81897fa640 1 -- 172.21.15.57:0/2394142876 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f817c09ffb0 con 0x7f818c100840 2024-04-21T08:45:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:26.626 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:26.626 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f81780a39b0 msgr2=0x7f81780a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f81780a39b0 0x7f81780a5e70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f818c19b8e0 tx=0x7f8174005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 msgr2=0x7f818c19a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c19a3d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b4a0 tx=0x7f817c00b960 comp rx=0 tx=0).stop 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 shutdown_connections 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f81780a39b0 0x7f81780a5e70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f818c101b40 0x7f818c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f818c1011a0 0x7f818c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 --2- 172.21.15.57:0/2394142876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f818c100840 0x7f818c19a3d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 >> 172.21.15.57:0/2394142876 conn(0x7f818c0fc3e0 msgr2=0x7f818c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:26.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 shutdown_connections 2024-04-21T08:45:26.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.628+0000 7f8191a9a640 1 -- 172.21.15.57:0/2394142876 wait complete. 2024-04-21T08:45:26.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:26 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1844843399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:26 smithi136 ceph-mon[31791]: mgrmap e22: x(active, since 1.01845s), standbys: y, z 2024-04-21T08:45:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:26 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1543644643' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:26 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2124313086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:26.639 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-21T08:45:26.639 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-21T08:45:26.639 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.741+0000 7f25ac19f640 1 -- 172.21.15.57:0/2569048397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4100e70 msgr2=0x7f25a4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- 172.21.15.57:0/2569048397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4100e70 0x7f25a4101290 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2594009900 tx=0x7f259402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- 172.21.15.57:0/2569048397 shutdown_connections 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- 172.21.15.57:0/2569048397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25a4102980 0x7f25a41092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- 172.21.15.57:0/2569048397 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a4101fc0 0x7f25a4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- 172.21.15.57:0/2569048397 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4100e70 0x7f25a4101290 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- 172.21.15.57:0/2569048397 >> 172.21.15.57:0/2569048397 conn(0x7f25a40fc3e0 msgr2=0x7f25a40fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- 172.21.15.57:0/2569048397 shutdown_connections 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- 172.21.15.57:0/2569048397 wait complete. 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 Processor -- start 2024-04-21T08:45:26.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- start start 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25a4100e70 0x7f25a4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a4101fc0 0x7f25a41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 0x7f25a419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410c9a0 con 0x7f25a4102980 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410c820 con 0x7f25a4101fc0 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.742+0000 7f25ac19f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410cb20 con 0x7f25a4100e70 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 0x7f25a419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 0x7f25a419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46246/0 (socket says 172.21.15.57:46246) 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 -- 172.21.15.57:0/3565109508 learned_addr learned my addr 172.21.15.57:0/3565109508 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25a9f3b640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25a4100e70 0x7f25a4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25a973a640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a4101fc0 0x7f25a41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 -- 172.21.15.57:0/3565109508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25a4100e70 msgr2=0x7f25a4198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25a4100e70 0x7f25a4198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 -- 172.21.15.57:0/3565109508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a4101fc0 msgr2=0x7f25a41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a4101fc0 0x7f25a41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 -- 172.21.15.57:0/3565109508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a419fda0 con 0x7f25a4102980 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25aa73c640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 0x7f25a419f730 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f259c00eb10 tx=0x7f259c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c004b20 con 0x7f25a4102980 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f259c004cc0 con 0x7f25a4102980 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a41a0090 con 0x7f25a4102980 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c010670 con 0x7f25a4102980 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.743+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a05e0 con 0x7f25a4102980 2024-04-21T08:45:26.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.744+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a41a08a0 con 0x7f25a4102980 2024-04-21T08:45:26.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.746+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f259c005000 con 0x7f25a4102980 2024-04-21T08:45:26.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.747+0000 7f259affd640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f25900a3a30 0x7f25900a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:26.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.747+0000 7f25a9f3b640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f25900a3a30 0x7f25900a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:26.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.747+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f259c0d0bb0 con 0x7f25a4102980 2024-04-21T08:45:26.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.748+0000 7f25a9f3b640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f25900a3a30 0x7f25900a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2594009230 tx=0x7f2594005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:26.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.750+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f259c053ff0 con 0x7f25a4102980 2024-04-21T08:45:26.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.956+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f25a41a0a70 con 0x7f25900a3a30 2024-04-21T08:45:26.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.967+0000 7f259affd640 1 -- 172.21.15.57:0/3565109508 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339620 (secure 0 0 0) 0x7f25a41a0a70 con 0x7f25900a3a30 2024-04-21T08:45:26.971 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:26.972 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:45:26.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f25900a3a30 msgr2=0x7f25900a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f25900a3a30 0x7f25900a5ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2594009230 tx=0x7f2594005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 msgr2=0x7f25a419f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:26.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 0x7f25a419f730 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f259c00eb10 tx=0x7f259c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 shutdown_connections 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f25900a3a30 0x7f25900a5ef0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25a4102980 0x7f25a419f730 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a4101fc0 0x7f25a41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 --2- 172.21.15.57:0/3565109508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25a4100e70 0x7f25a4198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.970+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 >> 172.21.15.57:0/3565109508 conn(0x7f25a40fc3e0 msgr2=0x7f25a40fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.971+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 shutdown_connections 2024-04-21T08:45:26.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:26.971+0000 7f25ac19f640 1 -- 172.21.15.57:0/3565109508 wait complete. 2024-04-21T08:45:26.982 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T08:45:26.917452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":490,"num_read_kb":400,"num_write":383,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365900,"kb_used_data":12536,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575924,"statfs":{"total":1151940427776,"available":1151565746176,"internally_reserved":0,"allocated":12836864,"data_stored":9729285,"data_compressed":534364,"data_compressed_allocated":720896,"data_compressed_original":4186112,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.987182"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993678+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:45:24.993678+0000","last_peered":"2024-04-21T08:45:24.993678+0000","last_clean":"2024-04-21T08:45:24.993678+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:45:24.993678+0000","last_undegraded":"2024-04-21T08:45:24.993678+0000","last_fullsized":"2024-04-21T08:45:24.993678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337688+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:45:25.337688+0000","last_peered":"2024-04-21T08:45:25.337688+0000","last_clean":"2024-04-21T08:45:25.337688+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:45:25.337688+0000","last_undegraded":"2024-04-21T08:45:25.337688+0000","last_fullsized":"2024-04-21T08:45:25.337688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338255+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:45:25.338255+0000","last_peered":"2024-04-21T08:45:25.338255+0000","last_clean":"2024-04-21T08:45:25.338255+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:45:25.338255+0000","last_undegraded":"2024-04-21T08:45:25.338255+0000","last_fullsized":"2024-04-21T08:45:25.338255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904383+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:45:24.904383+0000","last_peered":"2024-04-21T08:45:24.904383+0000","last_clean":"2024-04-21T08:45:24.904383+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:45:24.904383+0000","last_undegraded":"2024-04-21T08:45:24.904383+0000","last_fullsized":"2024-04-21T08:45:24.904383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991927+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:45:24.991927+0000","last_peered":"2024-04-21T08:45:24.991927+0000","last_clean":"2024-04-21T08:45:24.991927+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:45:24.991927+0000","last_undegraded":"2024-04-21T08:45:24.991927+0000","last_fullsized":"2024-04-21T08:45:24.991927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992137+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:45:24.992137+0000","last_peered":"2024-04-21T08:45:24.992137+0000","last_clean":"2024-04-21T08:45:24.992137+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:45:24.992137+0000","last_undegraded":"2024-04-21T08:45:24.992137+0000","last_fullsized":"2024-04-21T08:45:24.992137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992575+0000","last_change":"2024-04-21T08:44:47.312984+0000","last_active":"2024-04-21T08:45:24.992575+0000","last_peered":"2024-04-21T08:45:24.992575+0000","last_clean":"2024-04-21T08:45:24.992575+0000","last_became_active":"2024-04-21T08:44:47.312465+0000","last_became_peered":"2024-04-21T08:44:47.312465+0000","last_unstale":"2024-04-21T08:45:24.992575+0000","last_undegraded":"2024-04-21T08:45:24.992575+0000","last_fullsized":"2024-04-21T08:45:24.992575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:59:51.214620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993071+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:45:24.993071+0000","last_peered":"2024-04-21T08:45:24.993071+0000","last_clean":"2024-04-21T08:45:24.993071+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:45:24.993071+0000","last_undegraded":"2024-04-21T08:45:24.993071+0000","last_fullsized":"2024-04-21T08:45:24.993071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993708+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:45:24.993708+0000","last_peered":"2024-04-21T08:45:24.993708+0000","last_clean":"2024-04-21T08:45:24.993708+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:45:24.993708+0000","last_undegraded":"2024-04-21T08:45:24.993708+0000","last_fullsized":"2024-04-21T08:45:24.993708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993237+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:45:24.993237+0000","last_peered":"2024-04-21T08:45:24.993237+0000","last_clean":"2024-04-21T08:45:24.993237+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:45:24.993237+0000","last_undegraded":"2024-04-21T08:45:24.993237+0000","last_fullsized":"2024-04-21T08:45:24.993237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904442+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:45:24.904442+0000","last_peered":"2024-04-21T08:45:24.904442+0000","last_clean":"2024-04-21T08:45:24.904442+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:45:24.904442+0000","last_undegraded":"2024-04-21T08:45:24.904442+0000","last_fullsized":"2024-04-21T08:45:24.904442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992519+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:45:24.992519+0000","last_peered":"2024-04-21T08:45:24.992519+0000","last_clean":"2024-04-21T08:45:24.992519+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:45:24.992519+0000","last_undegraded":"2024-04-21T08:45:24.992519+0000","last_fullsized":"2024-04-21T08:45:24.992519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337366+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:45:25.337366+0000","last_peered":"2024-04-21T08:45:25.337366+0000","last_clean":"2024-04-21T08:45:25.337366+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:45:25.337366+0000","last_undegraded":"2024-04-21T08:45:25.337366+0000","last_fullsized":"2024-04-21T08:45:25.337366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337428+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:45:25.337428+0000","last_peered":"2024-04-21T08:45:25.337428+0000","last_clean":"2024-04-21T08:45:25.337428+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:45:25.337428+0000","last_undegraded":"2024-04-21T08:45:25.337428+0000","last_fullsized":"2024-04-21T08:45:25.337428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991935+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:45:24.991935+0000","last_peered":"2024-04-21T08:45:24.991935+0000","last_clean":"2024-04-21T08:45:24.991935+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:45:24.991935+0000","last_undegraded":"2024-04-21T08:45:24.991935+0000","last_fullsized":"2024-04-21T08:45:24.991935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993546+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:45:24.993546+0000","last_peered":"2024-04-21T08:45:24.993546+0000","last_clean":"2024-04-21T08:45:24.993546+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:45:24.993546+0000","last_undegraded":"2024-04-21T08:45:24.993546+0000","last_fullsized":"2024-04-21T08:45:24.993546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993979+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:45:24.993979+0000","last_peered":"2024-04-21T08:45:24.993979+0000","last_clean":"2024-04-21T08:45:24.993979+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:45:24.993979+0000","last_undegraded":"2024-04-21T08:45:24.993979+0000","last_fullsized":"2024-04-21T08:45:24.993979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904038+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:45:24.904038+0000","last_peered":"2024-04-21T08:45:24.904038+0000","last_clean":"2024-04-21T08:45:24.904038+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:45:24.904038+0000","last_undegraded":"2024-04-21T08:45:24.904038+0000","last_fullsized":"2024-04-21T08:45:24.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903605+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:45:24.903605+0000","last_peered":"2024-04-21T08:45:24.903605+0000","last_clean":"2024-04-21T08:45:24.903605+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:45:24.903605+0000","last_undegraded":"2024-04-21T08:45:24.903605+0000","last_fullsized":"2024-04-21T08:45:24.903605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.994253+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:45:24.994253+0000","last_peered":"2024-04-21T08:45:24.994253+0000","last_clean":"2024-04-21T08:45:24.994253+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:45:24.994253+0000","last_undegraded":"2024-04-21T08:45:24.994253+0000","last_fullsized":"2024-04-21T08:45:24.994253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904457+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:45:24.904457+0000","last_peered":"2024-04-21T08:45:24.904457+0000","last_clean":"2024-04-21T08:45:24.904457+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:45:24.904457+0000","last_undegraded":"2024-04-21T08:45:24.904457+0000","last_fullsized":"2024-04-21T08:45:24.904457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904078+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:45:24.904078+0000","last_peered":"2024-04-21T08:45:24.904078+0000","last_clean":"2024-04-21T08:45:24.904078+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:45:24.904078+0000","last_undegraded":"2024-04-21T08:45:24.904078+0000","last_fullsized":"2024-04-21T08:45:24.904078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904639+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:45:24.904639+0000","last_peered":"2024-04-21T08:45:24.904639+0000","last_clean":"2024-04-21T08:45:24.904639+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:45:24.904639+0000","last_undegraded":"2024-04-21T08:45:24.904639+0000","last_fullsized":"2024-04-21T08:45:24.904639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338007+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:45:25.338007+0000","last_peered":"2024-04-21T08:45:25.338007+0000","last_clean":"2024-04-21T08:45:25.338007+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:45:25.338007+0000","last_undegraded":"2024-04-21T08:45:25.338007+0000","last_fullsized":"2024-04-21T08:45:25.338007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904338+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:45:24.904338+0000","last_peered":"2024-04-21T08:45:24.904338+0000","last_clean":"2024-04-21T08:45:24.904338+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:45:24.904338+0000","last_undegraded":"2024-04-21T08:45:24.904338+0000","last_fullsized":"2024-04-21T08:45:24.904338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992394+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:45:24.992394+0000","last_peered":"2024-04-21T08:45:24.992394+0000","last_clean":"2024-04-21T08:45:24.992394+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:45:24.992394+0000","last_undegraded":"2024-04-21T08:45:24.992394+0000","last_fullsized":"2024-04-21T08:45:24.992394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992458+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:45:24.992458+0000","last_peered":"2024-04-21T08:45:24.992458+0000","last_clean":"2024-04-21T08:45:24.992458+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:45:24.992458+0000","last_undegraded":"2024-04-21T08:45:24.992458+0000","last_fullsized":"2024-04-21T08:45:24.992458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904082+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:45:24.904082+0000","last_peered":"2024-04-21T08:45:24.904082+0000","last_clean":"2024-04-21T08:45:24.904082+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:45:24.904082+0000","last_undegraded":"2024-04-21T08:45:24.904082+0000","last_fullsized":"2024-04-21T08:45:24.904082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992123+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:45:24.992123+0000","last_peered":"2024-04-21T08:45:24.992123+0000","last_clean":"2024-04-21T08:45:24.992123+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:45:24.992123+0000","last_undegraded":"2024-04-21T08:45:24.992123+0000","last_fullsized":"2024-04-21T08:45:24.992123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903921+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:45:24.903921+0000","last_peered":"2024-04-21T08:45:24.903921+0000","last_clean":"2024-04-21T08:45:24.903921+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:45:24.903921+0000","last_undegraded":"2024-04-21T08:45:24.903921+0000","last_fullsized":"2024-04-21T08:45:24.903921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992899+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:45:24.992899+0000","last_peered":"2024-04-21T08:45:24.992899+0000","last_clean":"2024-04-21T08:45:24.992899+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:45:24.992899+0000","last_undegraded":"2024-04-21T08:45:24.992899+0000","last_fullsized":"2024-04-21T08:45:24.992899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992410+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:45:24.992410+0000","last_peered":"2024-04-21T08:45:24.992410+0000","last_clean":"2024-04-21T08:45:24.992410+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:45:24.992410+0000","last_undegraded":"2024-04-21T08:45:24.992410+0000","last_fullsized":"2024-04-21T08:45:24.992410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903852+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:45:24.903852+0000","last_peered":"2024-04-21T08:45:24.903852+0000","last_clean":"2024-04-21T08:45:24.903852+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:45:24.903852+0000","last_undegraded":"2024-04-21T08:45:24.903852+0000","last_fullsized":"2024-04-21T08:45:24.903852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992469+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:45:24.992469+0000","last_peered":"2024-04-21T08:45:24.992469+0000","last_clean":"2024-04-21T08:45:24.992469+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:45:24.992469+0000","last_undegraded":"2024-04-21T08:45:24.992469+0000","last_fullsized":"2024-04-21T08:45:24.992469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993305+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:45:24.993305+0000","last_peered":"2024-04-21T08:45:24.993305+0000","last_clean":"2024-04-21T08:45:24.993305+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:45:24.993305+0000","last_undegraded":"2024-04-21T08:45:24.993305+0000","last_fullsized":"2024-04-21T08:45:24.993305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993857+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:45:24.993857+0000","last_peered":"2024-04-21T08:45:24.993857+0000","last_clean":"2024-04-21T08:45:24.993857+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:45:24.993857+0000","last_undegraded":"2024-04-21T08:45:24.993857+0000","last_fullsized":"2024-04-21T08:45:24.993857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992300+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:45:24.992300+0000","last_peered":"2024-04-21T08:45:24.992300+0000","last_clean":"2024-04-21T08:45:24.992300+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:45:24.992300+0000","last_undegraded":"2024-04-21T08:45:24.992300+0000","last_fullsized":"2024-04-21T08:45:24.992300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903970+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:45:24.903970+0000","last_peered":"2024-04-21T08:45:24.903970+0000","last_clean":"2024-04-21T08:45:24.903970+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:45:24.903970+0000","last_undegraded":"2024-04-21T08:45:24.903970+0000","last_fullsized":"2024-04-21T08:45:24.903970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337570+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:45:25.337570+0000","last_peered":"2024-04-21T08:45:25.337570+0000","last_clean":"2024-04-21T08:45:25.337570+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:45:25.337570+0000","last_undegraded":"2024-04-21T08:45:25.337570+0000","last_fullsized":"2024-04-21T08:45:25.337570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992789+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:45:24.992789+0000","last_peered":"2024-04-21T08:45:24.992789+0000","last_clean":"2024-04-21T08:45:24.992789+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:45:24.992789+0000","last_undegraded":"2024-04-21T08:45:24.992789+0000","last_fullsized":"2024-04-21T08:45:24.992789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992907+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:45:24.992907+0000","last_peered":"2024-04-21T08:45:24.992907+0000","last_clean":"2024-04-21T08:45:24.992907+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:45:24.992907+0000","last_undegraded":"2024-04-21T08:45:24.992907+0000","last_fullsized":"2024-04-21T08:45:24.992907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903912+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:45:24.903912+0000","last_peered":"2024-04-21T08:45:24.903912+0000","last_clean":"2024-04-21T08:45:24.903912+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:45:24.903912+0000","last_undegraded":"2024-04-21T08:45:24.903912+0000","last_fullsized":"2024-04-21T08:45:24.903912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993629+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:45:24.993629+0000","last_peered":"2024-04-21T08:45:24.993629+0000","last_clean":"2024-04-21T08:45:24.993629+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:45:24.993629+0000","last_undegraded":"2024-04-21T08:45:24.993629+0000","last_fullsized":"2024-04-21T08:45:24.993629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992855+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:45:24.992855+0000","last_peered":"2024-04-21T08:45:24.992855+0000","last_clean":"2024-04-21T08:45:24.992855+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:45:24.992855+0000","last_undegraded":"2024-04-21T08:45:24.992855+0000","last_fullsized":"2024-04-21T08:45:24.992855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993149+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:45:24.993149+0000","last_peered":"2024-04-21T08:45:24.993149+0000","last_clean":"2024-04-21T08:45:24.993149+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:45:24.993149+0000","last_undegraded":"2024-04-21T08:45:24.993149+0000","last_fullsized":"2024-04-21T08:45:24.993149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337962+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:45:25.337962+0000","last_peered":"2024-04-21T08:45:25.337962+0000","last_clean":"2024-04-21T08:45:25.337962+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:45:25.337962+0000","last_undegraded":"2024-04-21T08:45:25.337962+0000","last_fullsized":"2024-04-21T08:45:25.337962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904574+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:45:24.904574+0000","last_peered":"2024-04-21T08:45:24.904574+0000","last_clean":"2024-04-21T08:45:24.904574+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:45:24.904574+0000","last_undegraded":"2024-04-21T08:45:24.904574+0000","last_fullsized":"2024-04-21T08:45:24.904574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993021+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:45:24.993021+0000","last_peered":"2024-04-21T08:45:24.993021+0000","last_clean":"2024-04-21T08:45:24.993021+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:45:24.993021+0000","last_undegraded":"2024-04-21T08:45:24.993021+0000","last_fullsized":"2024-04-21T08:45:24.993021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992689+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:45:24.992689+0000","last_peered":"2024-04-21T08:45:24.992689+0000","last_clean":"2024-04-21T08:45:24.992689+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:45:24.992689+0000","last_undegraded":"2024-04-21T08:45:24.992689+0000","last_fullsized":"2024-04-21T08:45:24.992689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993192+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:45:24.993192+0000","last_peered":"2024-04-21T08:45:24.993192+0000","last_clean":"2024-04-21T08:45:24.993192+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:45:24.993192+0000","last_undegraded":"2024-04-21T08:45:24.993192+0000","last_fullsized":"2024-04-21T08:45:24.993192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'37","reported_seq":328,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.632907+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:45:25.632907+0000","last_peered":"2024-04-21T08:45:25.632907+0000","last_clean":"2024-04-21T08:45:25.632907+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:45:25.632907+0000","last_undegraded":"2024-04-21T08:45:25.632907+0000","last_fullsized":"2024-04-21T08:45:25.632907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":258,"num_read_kb":198,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992174+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:45:24.992174+0000","last_peered":"2024-04-21T08:45:24.992174+0000","last_clean":"2024-04-21T08:45:24.992174+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:45:24.992174+0000","last_undegraded":"2024-04-21T08:45:24.992174+0000","last_fullsized":"2024-04-21T08:45:24.992174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993263+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:45:24.993263+0000","last_peered":"2024-04-21T08:45:24.993263+0000","last_clean":"2024-04-21T08:45:24.993263+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:45:24.993263+0000","last_undegraded":"2024-04-21T08:45:24.993263+0000","last_fullsized":"2024-04-21T08:45:24.993263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993119+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:45:24.993119+0000","last_peered":"2024-04-21T08:45:24.993119+0000","last_clean":"2024-04-21T08:45:24.993119+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:45:24.993119+0000","last_undegraded":"2024-04-21T08:45:24.993119+0000","last_fullsized":"2024-04-21T08:45:24.993119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992385+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:45:24.992385+0000","last_peered":"2024-04-21T08:45:24.992385+0000","last_clean":"2024-04-21T08:45:24.992385+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:45:24.992385+0000","last_undegraded":"2024-04-21T08:45:24.992385+0000","last_fullsized":"2024-04-21T08:45:24.992385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993270+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:45:24.993270+0000","last_peered":"2024-04-21T08:45:24.993270+0000","last_clean":"2024-04-21T08:45:24.993270+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:45:24.993270+0000","last_undegraded":"2024-04-21T08:45:24.993270+0000","last_fullsized":"2024-04-21T08:45:24.993270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992679+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:45:24.992679+0000","last_peered":"2024-04-21T08:45:24.992679+0000","last_clean":"2024-04-21T08:45:24.992679+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:45:24.992679+0000","last_undegraded":"2024-04-21T08:45:24.992679+0000","last_fullsized":"2024-04-21T08:45:24.992679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992953+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:45:24.992953+0000","last_peered":"2024-04-21T08:45:24.992953+0000","last_clean":"2024-04-21T08:45:24.992953+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:45:24.992953+0000","last_undegraded":"2024-04-21T08:45:24.992953+0000","last_fullsized":"2024-04-21T08:45:24.992953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993936+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:45:24.993936+0000","last_peered":"2024-04-21T08:45:24.993936+0000","last_clean":"2024-04-21T08:45:24.993936+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:45:24.993936+0000","last_undegraded":"2024-04-21T08:45:24.993936+0000","last_fullsized":"2024-04-21T08:45:24.993936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904172+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:45:24.904172+0000","last_peered":"2024-04-21T08:45:24.904172+0000","last_clean":"2024-04-21T08:45:24.904172+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:45:24.904172+0000","last_undegraded":"2024-04-21T08:45:24.904172+0000","last_fullsized":"2024-04-21T08:45:24.904172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992489+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:45:24.992489+0000","last_peered":"2024-04-21T08:45:24.992489+0000","last_clean":"2024-04-21T08:45:24.992489+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:45:24.992489+0000","last_undegraded":"2024-04-21T08:45:24.992489+0000","last_fullsized":"2024-04-21T08:45:24.992489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992860+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:45:24.992860+0000","last_peered":"2024-04-21T08:45:24.992860+0000","last_clean":"2024-04-21T08:45:24.992860+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:45:24.992860+0000","last_undegraded":"2024-04-21T08:45:24.992860+0000","last_fullsized":"2024-04-21T08:45:24.992860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992036+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:45:24.992036+0000","last_peered":"2024-04-21T08:45:24.992036+0000","last_clean":"2024-04-21T08:45:24.992036+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:45:24.992036+0000","last_undegraded":"2024-04-21T08:45:24.992036+0000","last_fullsized":"2024-04-21T08:45:24.992036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992792+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:45:24.992792+0000","last_peered":"2024-04-21T08:45:24.992792+0000","last_clean":"2024-04-21T08:45:24.992792+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:45:24.992792+0000","last_undegraded":"2024-04-21T08:45:24.992792+0000","last_fullsized":"2024-04-21T08:45:24.992792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992835+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:45:24.992835+0000","last_peered":"2024-04-21T08:45:24.992835+0000","last_clean":"2024-04-21T08:45:24.992835+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:45:24.992835+0000","last_undegraded":"2024-04-21T08:45:24.992835+0000","last_fullsized":"2024-04-21T08:45:24.992835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992275+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:45:24.992275+0000","last_peered":"2024-04-21T08:45:24.992275+0000","last_clean":"2024-04-21T08:45:24.992275+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:45:24.992275+0000","last_undegraded":"2024-04-21T08:45:24.992275+0000","last_fullsized":"2024-04-21T08:45:24.992275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993653+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:45:24.993653+0000","last_peered":"2024-04-21T08:45:24.993653+0000","last_clean":"2024-04-21T08:45:24.993653+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:45:24.993653+0000","last_undegraded":"2024-04-21T08:45:24.993653+0000","last_fullsized":"2024-04-21T08:45:24.993653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992241+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:45:24.992241+0000","last_peered":"2024-04-21T08:45:24.992241+0000","last_clean":"2024-04-21T08:45:24.992241+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:45:24.992241+0000","last_undegraded":"2024-04-21T08:45:24.992241+0000","last_fullsized":"2024-04-21T08:45:24.992241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993525+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:45:24.993525+0000","last_peered":"2024-04-21T08:45:24.993525+0000","last_clean":"2024-04-21T08:45:24.993525+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:45:24.993525+0000","last_undegraded":"2024-04-21T08:45:24.993525+0000","last_fullsized":"2024-04-21T08:45:24.993525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993389+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:45:24.993389+0000","last_peered":"2024-04-21T08:45:24.993389+0000","last_clean":"2024-04-21T08:45:24.993389+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:45:24.993389+0000","last_undegraded":"2024-04-21T08:45:24.993389+0000","last_fullsized":"2024-04-21T08:45:24.993389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992746+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:45:24.992746+0000","last_peered":"2024-04-21T08:45:24.992746+0000","last_clean":"2024-04-21T08:45:24.992746+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:45:24.992746+0000","last_undegraded":"2024-04-21T08:45:24.992746+0000","last_fullsized":"2024-04-21T08:45:24.992746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992763+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:45:24.992763+0000","last_peered":"2024-04-21T08:45:24.992763+0000","last_clean":"2024-04-21T08:45:24.992763+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:45:24.992763+0000","last_undegraded":"2024-04-21T08:45:24.992763+0000","last_fullsized":"2024-04-21T08:45:24.992763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992403+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:45:24.992403+0000","last_peered":"2024-04-21T08:45:24.992403+0000","last_clean":"2024-04-21T08:45:24.992403+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:45:24.992403+0000","last_undegraded":"2024-04-21T08:45:24.992403+0000","last_fullsized":"2024-04-21T08:45:24.992403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992847+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:45:24.992847+0000","last_peered":"2024-04-21T08:45:24.992847+0000","last_clean":"2024-04-21T08:45:24.992847+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:45:24.992847+0000","last_undegraded":"2024-04-21T08:45:24.992847+0000","last_fullsized":"2024-04-21T08:45:24.992847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992175+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:45:24.992175+0000","last_peered":"2024-04-21T08:45:24.992175+0000","last_clean":"2024-04-21T08:45:24.992175+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:45:24.992175+0000","last_undegraded":"2024-04-21T08:45:24.992175+0000","last_fullsized":"2024-04-21T08:45:24.992175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993373+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:45:24.993373+0000","last_peered":"2024-04-21T08:45:24.993373+0000","last_clean":"2024-04-21T08:45:24.993373+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:45:24.993373+0000","last_undegraded":"2024-04-21T08:45:24.993373+0000","last_fullsized":"2024-04-21T08:45:24.993373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904114+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:45:24.904114+0000","last_peered":"2024-04-21T08:45:24.904114+0000","last_clean":"2024-04-21T08:45:24.904114+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:45:24.904114+0000","last_undegraded":"2024-04-21T08:45:24.904114+0000","last_fullsized":"2024-04-21T08:45:24.904114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904460+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:45:24.904460+0000","last_peered":"2024-04-21T08:45:24.904460+0000","last_clean":"2024-04-21T08:45:24.904460+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:45:24.904460+0000","last_undegraded":"2024-04-21T08:45:24.904460+0000","last_fullsized":"2024-04-21T08:45:24.904460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992255+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:45:24.992255+0000","last_peered":"2024-04-21T08:45:24.992255+0000","last_clean":"2024-04-21T08:45:24.992255+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:45:24.992255+0000","last_undegraded":"2024-04-21T08:45:24.992255+0000","last_fullsized":"2024-04-21T08:45:24.992255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992034+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:45:24.992034+0000","last_peered":"2024-04-21T08:45:24.992034+0000","last_clean":"2024-04-21T08:45:24.992034+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:45:24.992034+0000","last_undegraded":"2024-04-21T08:45:24.992034+0000","last_fullsized":"2024-04-21T08:45:24.992034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992367+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:45:24.992367+0000","last_peered":"2024-04-21T08:45:24.992367+0000","last_clean":"2024-04-21T08:45:24.992367+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:45:24.992367+0000","last_undegraded":"2024-04-21T08:45:24.992367+0000","last_fullsized":"2024-04-21T08:45:24.992367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993257+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:45:24.993257+0000","last_peered":"2024-04-21T08:45:24.993257+0000","last_clean":"2024-04-21T08:45:24.993257+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:45:24.993257+0000","last_undegraded":"2024-04-21T08:45:24.993257+0000","last_fullsized":"2024-04-21T08:45:24.993257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992262+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:45:24.992262+0000","last_peered":"2024-04-21T08:45:24.992262+0000","last_clean":"2024-04-21T08:45:24.992262+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:45:24.992262+0000","last_undegraded":"2024-04-21T08:45:24.992262+0000","last_fullsized":"2024-04-21T08:45:24.992262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992695+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:45:24.992695+0000","last_peered":"2024-04-21T08:45:24.992695+0000","last_clean":"2024-04-21T08:45:24.992695+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:45:24.992695+0000","last_undegraded":"2024-04-21T08:45:24.992695+0000","last_fullsized":"2024-04-21T08:45:24.992695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903778+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:45:24.903778+0000","last_peered":"2024-04-21T08:45:24.903778+0000","last_clean":"2024-04-21T08:45:24.903778+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:45:24.903778+0000","last_undegraded":"2024-04-21T08:45:24.903778+0000","last_fullsized":"2024-04-21T08:45:24.903778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'43","reported_seq":99,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.136133+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:45:25.136133+0000","last_peered":"2024-04-21T08:45:25.136133+0000","last_clean":"2024-04-21T08:45:25.136133+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:45:25.136133+0000","last_undegraded":"2024-04-21T08:45:25.136133+0000","last_fullsized":"2024-04-21T08:45:25.136133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904474+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:45:24.904474+0000","last_peered":"2024-04-21T08:45:24.904474+0000","last_clean":"2024-04-21T08:45:24.904474+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:45:24.904474+0000","last_undegraded":"2024-04-21T08:45:24.904474+0000","last_fullsized":"2024-04-21T08:45:24.904474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903473+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:45:24.903473+0000","last_peered":"2024-04-21T08:45:24.903473+0000","last_clean":"2024-04-21T08:45:24.903473+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:45:24.903473+0000","last_undegraded":"2024-04-21T08:45:24.903473+0000","last_fullsized":"2024-04-21T08:45:24.903473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904682+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:45:24.904682+0000","last_peered":"2024-04-21T08:45:24.904682+0000","last_clean":"2024-04-21T08:45:24.904682+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:45:24.904682+0000","last_undegraded":"2024-04-21T08:45:24.904682+0000","last_fullsized":"2024-04-21T08:45:24.904682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337838+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:45:25.337838+0000","last_peered":"2024-04-21T08:45:25.337838+0000","last_clean":"2024-04-21T08:45:25.337838+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:45:25.337838+0000","last_undegraded":"2024-04-21T08:45:25.337838+0000","last_fullsized":"2024-04-21T08:45:25.337838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904527+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:45:24.904527+0000","last_peered":"2024-04-21T08:45:24.904527+0000","last_clean":"2024-04-21T08:45:24.904527+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:45:24.904527+0000","last_undegraded":"2024-04-21T08:45:24.904527+0000","last_fullsized":"2024-04-21T08:45:24.904527+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991872+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:45:24.991872+0000","last_peered":"2024-04-21T08:45:24.991872+0000","last_clean":"2024-04-21T08:45:24.991872+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:45:24.991872+0000","last_undegraded":"2024-04-21T08:45:24.991872+0000","last_fullsized":"2024-04-21T08:45:24.991872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991968+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:45:24.991968+0000","last_peered":"2024-04-21T08:45:24.991968+0000","last_clean":"2024-04-21T08:45:24.991968+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:45:24.991968+0000","last_undegraded":"2024-04-21T08:45:24.991968+0000","last_fullsized":"2024-04-21T08:45:24.991968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992202+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:45:24.992202+0000","last_peered":"2024-04-21T08:45:24.992202+0000","last_clean":"2024-04-21T08:45:24.992202+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:45:24.992202+0000","last_undegraded":"2024-04-21T08:45:24.992202+0000","last_fullsized":"2024-04-21T08:45:24.992202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903893+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:45:24.903893+0000","last_peered":"2024-04-21T08:45:24.903893+0000","last_clean":"2024-04-21T08:45:24.903893+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:45:24.903893+0000","last_undegraded":"2024-04-21T08:45:24.903893+0000","last_fullsized":"2024-04-21T08:45:24.903893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992008+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:45:24.992008+0000","last_peered":"2024-04-21T08:45:24.992008+0000","last_clean":"2024-04-21T08:45:24.992008+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:45:24.992008+0000","last_undegraded":"2024-04-21T08:45:24.992008+0000","last_fullsized":"2024-04-21T08:45:24.992008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904288+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:45:24.904288+0000","last_peered":"2024-04-21T08:45:24.904288+0000","last_clean":"2024-04-21T08:45:24.904288+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:45:24.904288+0000","last_undegraded":"2024-04-21T08:45:24.904288+0000","last_fullsized":"2024-04-21T08:45:24.904288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903726+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:45:24.903726+0000","last_peered":"2024-04-21T08:45:24.903726+0000","last_clean":"2024-04-21T08:45:24.903726+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:45:24.903726+0000","last_undegraded":"2024-04-21T08:45:24.903726+0000","last_fullsized":"2024-04-21T08:45:24.903726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903977+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:45:24.903977+0000","last_peered":"2024-04-21T08:45:24.903977+0000","last_clean":"2024-04-21T08:45:24.903977+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:45:24.903977+0000","last_undegraded":"2024-04-21T08:45:24.903977+0000","last_fullsized":"2024-04-21T08:45:24.903977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904319+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:45:24.904319+0000","last_peered":"2024-04-21T08:45:24.904319+0000","last_clean":"2024-04-21T08:45:24.904319+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:45:24.904319+0000","last_undegraded":"2024-04-21T08:45:24.904319+0000","last_fullsized":"2024-04-21T08:45:24.904319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993372+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:45:24.993372+0000","last_peered":"2024-04-21T08:45:24.993372+0000","last_clean":"2024-04-21T08:45:24.993372+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:45:24.993372+0000","last_undegraded":"2024-04-21T08:45:24.993372+0000","last_fullsized":"2024-04-21T08:45:24.993372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992918+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:45:24.992918+0000","last_peered":"2024-04-21T08:45:24.992918+0000","last_clean":"2024-04-21T08:45:24.992918+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:45:24.992918+0000","last_undegraded":"2024-04-21T08:45:24.992918+0000","last_fullsized":"2024-04-21T08:45:24.992918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992146+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:45:24.992146+0000","last_peered":"2024-04-21T08:45:24.992146+0000","last_clean":"2024-04-21T08:45:24.992146+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:45:24.992146+0000","last_undegraded":"2024-04-21T08:45:24.992146+0000","last_fullsized":"2024-04-21T08:45:24.992146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904004+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:45:24.904004+0000","last_peered":"2024-04-21T08:45:24.904004+0000","last_clean":"2024-04-21T08:45:24.904004+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:45:24.904004+0000","last_undegraded":"2024-04-21T08:45:24.904004+0000","last_fullsized":"2024-04-21T08:45:24.904004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337411+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:45:25.337411+0000","last_peered":"2024-04-21T08:45:25.337411+0000","last_clean":"2024-04-21T08:45:25.337411+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:45:25.337411+0000","last_undegraded":"2024-04-21T08:45:25.337411+0000","last_fullsized":"2024-04-21T08:45:25.337411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.128236+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:45:25.128236+0000","last_peered":"2024-04-21T08:45:25.128236+0000","last_clean":"2024-04-21T08:45:25.128236+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:45:25.128236+0000","last_undegraded":"2024-04-21T08:45:25.128236+0000","last_fullsized":"2024-04-21T08:45:25.128236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992998+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:45:24.992998+0000","last_peered":"2024-04-21T08:45:24.992998+0000","last_clean":"2024-04-21T08:45:24.992998+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:45:24.992998+0000","last_undegraded":"2024-04-21T08:45:24.992998+0000","last_fullsized":"2024-04-21T08:45:24.992998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992640+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:45:24.992640+0000","last_peered":"2024-04-21T08:45:24.992640+0000","last_clean":"2024-04-21T08:45:24.992640+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:45:24.992640+0000","last_undegraded":"2024-04-21T08:45:24.992640+0000","last_fullsized":"2024-04-21T08:45:24.992640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992686+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:45:24.992686+0000","last_peered":"2024-04-21T08:45:24.992686+0000","last_clean":"2024-04-21T08:45:24.992686+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:45:24.992686+0000","last_undegraded":"2024-04-21T08:45:24.992686+0000","last_fullsized":"2024-04-21T08:45:24.992686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993176+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:45:24.993176+0000","last_peered":"2024-04-21T08:45:24.993176+0000","last_clean":"2024-04-21T08:45:24.993176+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:45:24.993176+0000","last_undegraded":"2024-04-21T08:45:24.993176+0000","last_fullsized":"2024-04-21T08:45:24.993176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992191+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:45:24.992191+0000","last_peered":"2024-04-21T08:45:24.992191+0000","last_clean":"2024-04-21T08:45:24.992191+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:45:24.992191+0000","last_undegraded":"2024-04-21T08:45:24.992191+0000","last_fullsized":"2024-04-21T08:45:24.992191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903722+0000","last_change":"2024-04-21T08:44:47.315087+0000","last_active":"2024-04-21T08:45:24.903722+0000","last_peered":"2024-04-21T08:45:24.903722+0000","last_clean":"2024-04-21T08:45:24.903722+0000","last_became_active":"2024-04-21T08:44:47.314647+0000","last_became_peered":"2024-04-21T08:44:47.314647+0000","last_unstale":"2024-04-21T08:45:24.903722+0000","last_undegraded":"2024-04-21T08:45:24.903722+0000","last_fullsized":"2024-04-21T08:45:24.903722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:20.425332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903546+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:45:24.903546+0000","last_peered":"2024-04-21T08:45:24.903546+0000","last_clean":"2024-04-21T08:45:24.903546+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:45:24.903546+0000","last_undegraded":"2024-04-21T08:45:24.903546+0000","last_fullsized":"2024-04-21T08:45:24.903546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337673+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:45:25.337673+0000","last_peered":"2024-04-21T08:45:25.337673+0000","last_clean":"2024-04-21T08:45:25.337673+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:45:25.337673+0000","last_undegraded":"2024-04-21T08:45:25.337673+0000","last_fullsized":"2024-04-21T08:45:25.337673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992553+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:45:24.992553+0000","last_peered":"2024-04-21T08:45:24.992553+0000","last_clean":"2024-04-21T08:45:24.992553+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:45:24.992553+0000","last_undegraded":"2024-04-21T08:45:24.992553+0000","last_fullsized":"2024-04-21T08:45:24.992553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904194+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:45:24.904194+0000","last_peered":"2024-04-21T08:45:24.904194+0000","last_clean":"2024-04-21T08:45:24.904194+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:45:24.904194+0000","last_undegraded":"2024-04-21T08:45:24.904194+0000","last_fullsized":"2024-04-21T08:45:24.904194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904716+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:45:24.904716+0000","last_peered":"2024-04-21T08:45:24.904716+0000","last_clean":"2024-04-21T08:45:24.904716+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:45:24.904716+0000","last_undegraded":"2024-04-21T08:45:24.904716+0000","last_fullsized":"2024-04-21T08:45:24.904716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992795+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:45:24.992795+0000","last_peered":"2024-04-21T08:45:24.992795+0000","last_clean":"2024-04-21T08:45:24.992795+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:45:24.992795+0000","last_undegraded":"2024-04-21T08:45:24.992795+0000","last_fullsized":"2024-04-21T08:45:24.992795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903992+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:45:24.903992+0000","last_peered":"2024-04-21T08:45:24.903992+0000","last_clean":"2024-04-21T08:45:24.903992+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:45:24.903992+0000","last_undegraded":"2024-04-21T08:45:24.903992+0000","last_fullsized":"2024-04-21T08:45:24.903992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992360+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:45:24.992360+0000","last_peered":"2024-04-21T08:45:24.992360+0000","last_clean":"2024-04-21T08:45:24.992360+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:45:24.992360+0000","last_undegraded":"2024-04-21T08:45:24.992360+0000","last_fullsized":"2024-04-21T08:45:24.992360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992585+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:45:24.992585+0000","last_peered":"2024-04-21T08:45:24.992585+0000","last_clean":"2024-04-21T08:45:24.992585+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:45:24.992585+0000","last_undegraded":"2024-04-21T08:45:24.992585+0000","last_fullsized":"2024-04-21T08:45:24.992585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993390+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:45:24.993390+0000","last_peered":"2024-04-21T08:45:24.993390+0000","last_clean":"2024-04-21T08:45:24.993390+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:45:24.993390+0000","last_undegraded":"2024-04-21T08:45:24.993390+0000","last_fullsized":"2024-04-21T08:45:24.993390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":110016,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":354,"num_read_kb":281,"num_write":200,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1787046,"data_compressed":80790,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528781,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1169281,"data_compressed":64942,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":926641,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":922567,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1123793,"data_compressed":66315,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":524651,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514896,"data_compressed":26642,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1049571,"data_compressed":54945,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":528769,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1110016,"data_stored":1250897,"data_compressed":72471,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":201862,"data_compressed":7784,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:45:26.988 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 -- 172.21.15.57:0/1546655285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc102a40 msgr2=0x7f36dc104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/1546655285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc102a40 0x7f36dc104e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009860 tx=0x7f36d002bb70 comp rx=0 tx=0).stop 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 -- 172.21.15.57:0/1546655285 shutdown_connections 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/1546655285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc107ca0 0x7f36dc10a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/1546655285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc105370 0x7f36dc107760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/1546655285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc102a40 0x7f36dc104e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 -- 172.21.15.57:0/1546655285 >> 172.21.15.57:0/1546655285 conn(0x7f36dc0fc410 msgr2=0x7f36dc0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 -- 172.21.15.57:0/1546655285 shutdown_connections 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.147+0000 7f36e2b9f640 1 -- 172.21.15.57:0/1546655285 wait complete. 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 Processor -- start 2024-04-21T08:45:27.149 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 -- start start 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 0x7f36dc198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 0x7f36dc198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc107ca0 0x7f36dc19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc10da80 con 0x7f36dc105370 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc10d900 con 0x7f36dc102a40 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e2b9f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc10dc00 con 0x7f36dc107ca0 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 0x7f36dc198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e093b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 0x7f36dc198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 0x7f36dc198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46282/0 (socket says 172.21.15.57:46282) 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36dbfff640 1 -- 172.21.15.57:0/3608330826 learned_addr learned my addr 172.21.15.57:0/3608330826 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e093b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 0x7f36dc198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:38978/0 (socket says 172.21.15.57:38978) 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.148+0000 7f36e113c640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc107ca0 0x7f36dc19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e093b640 1 -- 172.21.15.57:0/3608330826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc107ca0 msgr2=0x7f36dc19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e093b640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc107ca0 0x7f36dc19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e093b640 1 -- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 msgr2=0x7f36dc198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e093b640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 0x7f36dc198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e093b640 1 -- 172.21.15.57:0/3608330826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc19fe50 con 0x7f36dc102a40 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36dbfff640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 0x7f36dc198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e093b640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 0x7f36dc198230 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36d002c0b0 tx=0x7f36d00042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0041020 con 0x7f36dc102a40 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36d00035f0 con 0x7f36dc102a40 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36dc1a00e0 con 0x7f36dc102a40 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0039040 con 0x7f36dc102a40 2024-04-21T08:45:27.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.149+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36dc1a05d0 con 0x7f36dc102a40 2024-04-21T08:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.150+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36dc0fdf00 con 0x7f36dc102a40 2024-04-21T08:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.152+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f36d0044020 con 0x7f36dc102a40 2024-04-21T08:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.152+0000 7f36d9ffb640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f36cc0a39b0 0x7f36cc0a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.153+0000 7f36dbfff640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f36cc0a39b0 0x7f36cc0a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.153+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f36d00f1e90 con 0x7f36dc102a40 2024-04-21T08:45:27.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.153+0000 7f36dbfff640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f36cc0a39b0 0x7f36cc0a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f36dc199740 tx=0x7f36c8005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.156 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.156+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36d004b020 con 0x7f36dc102a40 2024-04-21T08:45:27.353 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.354+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f36d00ba610 con 0x7f36dc102a40 2024-04-21T08:45:27.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.363+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36dc062390 con 0x7f36cc0a39b0 2024-04-21T08:45:27.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.374+0000 7f36d9ffb640 1 -- 172.21.15.57:0/3608330826 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339620 (secure 0 0 0) 0x7f36dc062390 con 0x7f36cc0a39b0 2024-04-21T08:45:27.377 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:27.379 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f36cc0a39b0 msgr2=0x7f36cc0a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f36cc0a39b0 0x7f36cc0a5e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f36dc199740 tx=0x7f36c8005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 msgr2=0x7f36dc198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 0x7f36dc198230 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f36d002c0b0 tx=0x7f36d00042f0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 shutdown_connections 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f36cc0a39b0 0x7f36cc0a5e70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36dc107ca0 0x7f36dc19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f36dc105370 0x7f36dc198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 --2- 172.21.15.57:0/3608330826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36dc102a40 0x7f36dc198230 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 >> 172.21.15.57:0/3608330826 conn(0x7f36dc0fc410 msgr2=0x7f36dc103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 shutdown_connections 2024-04-21T08:45:27.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.378+0000 7f36e2b9f640 1 -- 172.21.15.57:0/3608330826 wait complete. 2024-04-21T08:45:27.389 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T08:45:26.917452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":490,"num_read_kb":400,"num_write":383,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365900,"kb_used_data":12536,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575924,"statfs":{"total":1151940427776,"available":1151565746176,"internally_reserved":0,"allocated":12836864,"data_stored":9729285,"data_compressed":534364,"data_compressed_allocated":720896,"data_compressed_original":4186112,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.987182"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993678+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:45:24.993678+0000","last_peered":"2024-04-21T08:45:24.993678+0000","last_clean":"2024-04-21T08:45:24.993678+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:45:24.993678+0000","last_undegraded":"2024-04-21T08:45:24.993678+0000","last_fullsized":"2024-04-21T08:45:24.993678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337688+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:45:25.337688+0000","last_peered":"2024-04-21T08:45:25.337688+0000","last_clean":"2024-04-21T08:45:25.337688+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:45:25.337688+0000","last_undegraded":"2024-04-21T08:45:25.337688+0000","last_fullsized":"2024-04-21T08:45:25.337688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338255+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:45:25.338255+0000","last_peered":"2024-04-21T08:45:25.338255+0000","last_clean":"2024-04-21T08:45:25.338255+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:45:25.338255+0000","last_undegraded":"2024-04-21T08:45:25.338255+0000","last_fullsized":"2024-04-21T08:45:25.338255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904383+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:45:24.904383+0000","last_peered":"2024-04-21T08:45:24.904383+0000","last_clean":"2024-04-21T08:45:24.904383+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:45:24.904383+0000","last_undegraded":"2024-04-21T08:45:24.904383+0000","last_fullsized":"2024-04-21T08:45:24.904383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991927+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:45:24.991927+0000","last_peered":"2024-04-21T08:45:24.991927+0000","last_clean":"2024-04-21T08:45:24.991927+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:45:24.991927+0000","last_undegraded":"2024-04-21T08:45:24.991927+0000","last_fullsized":"2024-04-21T08:45:24.991927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992137+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:45:24.992137+0000","last_peered":"2024-04-21T08:45:24.992137+0000","last_clean":"2024-04-21T08:45:24.992137+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:45:24.992137+0000","last_undegraded":"2024-04-21T08:45:24.992137+0000","last_fullsized":"2024-04-21T08:45:24.992137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992575+0000","last_change":"2024-04-21T08:44:47.312984+0000","last_active":"2024-04-21T08:45:24.992575+0000","last_peered":"2024-04-21T08:45:24.992575+0000","last_clean":"2024-04-21T08:45:24.992575+0000","last_became_active":"2024-04-21T08:44:47.312465+0000","last_became_peered":"2024-04-21T08:44:47.312465+0000","last_unstale":"2024-04-21T08:45:24.992575+0000","last_undegraded":"2024-04-21T08:45:24.992575+0000","last_fullsized":"2024-04-21T08:45:24.992575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:59:51.214620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993071+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:45:24.993071+0000","last_peered":"2024-04-21T08:45:24.993071+0000","last_clean":"2024-04-21T08:45:24.993071+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:45:24.993071+0000","last_undegraded":"2024-04-21T08:45:24.993071+0000","last_fullsized":"2024-04-21T08:45:24.993071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993708+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:45:24.993708+0000","last_peered":"2024-04-21T08:45:24.993708+0000","last_clean":"2024-04-21T08:45:24.993708+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:45:24.993708+0000","last_undegraded":"2024-04-21T08:45:24.993708+0000","last_fullsized":"2024-04-21T08:45:24.993708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993237+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:45:24.993237+0000","last_peered":"2024-04-21T08:45:24.993237+0000","last_clean":"2024-04-21T08:45:24.993237+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:45:24.993237+0000","last_undegraded":"2024-04-21T08:45:24.993237+0000","last_fullsized":"2024-04-21T08:45:24.993237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904442+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:45:24.904442+0000","last_peered":"2024-04-21T08:45:24.904442+0000","last_clean":"2024-04-21T08:45:24.904442+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:45:24.904442+0000","last_undegraded":"2024-04-21T08:45:24.904442+0000","last_fullsized":"2024-04-21T08:45:24.904442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992519+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:45:24.992519+0000","last_peered":"2024-04-21T08:45:24.992519+0000","last_clean":"2024-04-21T08:45:24.992519+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:45:24.992519+0000","last_undegraded":"2024-04-21T08:45:24.992519+0000","last_fullsized":"2024-04-21T08:45:24.992519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337366+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:45:25.337366+0000","last_peered":"2024-04-21T08:45:25.337366+0000","last_clean":"2024-04-21T08:45:25.337366+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:45:25.337366+0000","last_undegraded":"2024-04-21T08:45:25.337366+0000","last_fullsized":"2024-04-21T08:45:25.337366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337428+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:45:25.337428+0000","last_peered":"2024-04-21T08:45:25.337428+0000","last_clean":"2024-04-21T08:45:25.337428+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:45:25.337428+0000","last_undegraded":"2024-04-21T08:45:25.337428+0000","last_fullsized":"2024-04-21T08:45:25.337428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991935+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:45:24.991935+0000","last_peered":"2024-04-21T08:45:24.991935+0000","last_clean":"2024-04-21T08:45:24.991935+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:45:24.991935+0000","last_undegraded":"2024-04-21T08:45:24.991935+0000","last_fullsized":"2024-04-21T08:45:24.991935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993546+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:45:24.993546+0000","last_peered":"2024-04-21T08:45:24.993546+0000","last_clean":"2024-04-21T08:45:24.993546+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:45:24.993546+0000","last_undegraded":"2024-04-21T08:45:24.993546+0000","last_fullsized":"2024-04-21T08:45:24.993546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993979+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:45:24.993979+0000","last_peered":"2024-04-21T08:45:24.993979+0000","last_clean":"2024-04-21T08:45:24.993979+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:45:24.993979+0000","last_undegraded":"2024-04-21T08:45:24.993979+0000","last_fullsized":"2024-04-21T08:45:24.993979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904038+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:45:24.904038+0000","last_peered":"2024-04-21T08:45:24.904038+0000","last_clean":"2024-04-21T08:45:24.904038+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:45:24.904038+0000","last_undegraded":"2024-04-21T08:45:24.904038+0000","last_fullsized":"2024-04-21T08:45:24.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903605+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:45:24.903605+0000","last_peered":"2024-04-21T08:45:24.903605+0000","last_clean":"2024-04-21T08:45:24.903605+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:45:24.903605+0000","last_undegraded":"2024-04-21T08:45:24.903605+0000","last_fullsized":"2024-04-21T08:45:24.903605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.994253+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:45:24.994253+0000","last_peered":"2024-04-21T08:45:24.994253+0000","last_clean":"2024-04-21T08:45:24.994253+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:45:24.994253+0000","last_undegraded":"2024-04-21T08:45:24.994253+0000","last_fullsized":"2024-04-21T08:45:24.994253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904457+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:45:24.904457+0000","last_peered":"2024-04-21T08:45:24.904457+0000","last_clean":"2024-04-21T08:45:24.904457+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:45:24.904457+0000","last_undegraded":"2024-04-21T08:45:24.904457+0000","last_fullsized":"2024-04-21T08:45:24.904457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904078+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:45:24.904078+0000","last_peered":"2024-04-21T08:45:24.904078+0000","last_clean":"2024-04-21T08:45:24.904078+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:45:24.904078+0000","last_undegraded":"2024-04-21T08:45:24.904078+0000","last_fullsized":"2024-04-21T08:45:24.904078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904639+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:45:24.904639+0000","last_peered":"2024-04-21T08:45:24.904639+0000","last_clean":"2024-04-21T08:45:24.904639+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:45:24.904639+0000","last_undegraded":"2024-04-21T08:45:24.904639+0000","last_fullsized":"2024-04-21T08:45:24.904639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338007+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:45:25.338007+0000","last_peered":"2024-04-21T08:45:25.338007+0000","last_clean":"2024-04-21T08:45:25.338007+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:45:25.338007+0000","last_undegraded":"2024-04-21T08:45:25.338007+0000","last_fullsized":"2024-04-21T08:45:25.338007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904338+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:45:24.904338+0000","last_peered":"2024-04-21T08:45:24.904338+0000","last_clean":"2024-04-21T08:45:24.904338+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:45:24.904338+0000","last_undegraded":"2024-04-21T08:45:24.904338+0000","last_fullsized":"2024-04-21T08:45:24.904338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992394+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:45:24.992394+0000","last_peered":"2024-04-21T08:45:24.992394+0000","last_clean":"2024-04-21T08:45:24.992394+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:45:24.992394+0000","last_undegraded":"2024-04-21T08:45:24.992394+0000","last_fullsized":"2024-04-21T08:45:24.992394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992458+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:45:24.992458+0000","last_peered":"2024-04-21T08:45:24.992458+0000","last_clean":"2024-04-21T08:45:24.992458+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:45:24.992458+0000","last_undegraded":"2024-04-21T08:45:24.992458+0000","last_fullsized":"2024-04-21T08:45:24.992458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904082+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:45:24.904082+0000","last_peered":"2024-04-21T08:45:24.904082+0000","last_clean":"2024-04-21T08:45:24.904082+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:45:24.904082+0000","last_undegraded":"2024-04-21T08:45:24.904082+0000","last_fullsized":"2024-04-21T08:45:24.904082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992123+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:45:24.992123+0000","last_peered":"2024-04-21T08:45:24.992123+0000","last_clean":"2024-04-21T08:45:24.992123+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:45:24.992123+0000","last_undegraded":"2024-04-21T08:45:24.992123+0000","last_fullsized":"2024-04-21T08:45:24.992123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903921+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:45:24.903921+0000","last_peered":"2024-04-21T08:45:24.903921+0000","last_clean":"2024-04-21T08:45:24.903921+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:45:24.903921+0000","last_undegraded":"2024-04-21T08:45:24.903921+0000","last_fullsized":"2024-04-21T08:45:24.903921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992899+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:45:24.992899+0000","last_peered":"2024-04-21T08:45:24.992899+0000","last_clean":"2024-04-21T08:45:24.992899+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:45:24.992899+0000","last_undegraded":"2024-04-21T08:45:24.992899+0000","last_fullsized":"2024-04-21T08:45:24.992899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992410+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:45:24.992410+0000","last_peered":"2024-04-21T08:45:24.992410+0000","last_clean":"2024-04-21T08:45:24.992410+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:45:24.992410+0000","last_undegraded":"2024-04-21T08:45:24.992410+0000","last_fullsized":"2024-04-21T08:45:24.992410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903852+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:45:24.903852+0000","last_peered":"2024-04-21T08:45:24.903852+0000","last_clean":"2024-04-21T08:45:24.903852+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:45:24.903852+0000","last_undegraded":"2024-04-21T08:45:24.903852+0000","last_fullsized":"2024-04-21T08:45:24.903852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992469+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:45:24.992469+0000","last_peered":"2024-04-21T08:45:24.992469+0000","last_clean":"2024-04-21T08:45:24.992469+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:45:24.992469+0000","last_undegraded":"2024-04-21T08:45:24.992469+0000","last_fullsized":"2024-04-21T08:45:24.992469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993305+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:45:24.993305+0000","last_peered":"2024-04-21T08:45:24.993305+0000","last_clean":"2024-04-21T08:45:24.993305+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:45:24.993305+0000","last_undegraded":"2024-04-21T08:45:24.993305+0000","last_fullsized":"2024-04-21T08:45:24.993305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993857+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:45:24.993857+0000","last_peered":"2024-04-21T08:45:24.993857+0000","last_clean":"2024-04-21T08:45:24.993857+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:45:24.993857+0000","last_undegraded":"2024-04-21T08:45:24.993857+0000","last_fullsized":"2024-04-21T08:45:24.993857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992300+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:45:24.992300+0000","last_peered":"2024-04-21T08:45:24.992300+0000","last_clean":"2024-04-21T08:45:24.992300+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:45:24.992300+0000","last_undegraded":"2024-04-21T08:45:24.992300+0000","last_fullsized":"2024-04-21T08:45:24.992300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903970+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:45:24.903970+0000","last_peered":"2024-04-21T08:45:24.903970+0000","last_clean":"2024-04-21T08:45:24.903970+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:45:24.903970+0000","last_undegraded":"2024-04-21T08:45:24.903970+0000","last_fullsized":"2024-04-21T08:45:24.903970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337570+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:45:25.337570+0000","last_peered":"2024-04-21T08:45:25.337570+0000","last_clean":"2024-04-21T08:45:25.337570+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:45:25.337570+0000","last_undegraded":"2024-04-21T08:45:25.337570+0000","last_fullsized":"2024-04-21T08:45:25.337570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992789+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:45:24.992789+0000","last_peered":"2024-04-21T08:45:24.992789+0000","last_clean":"2024-04-21T08:45:24.992789+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:45:24.992789+0000","last_undegraded":"2024-04-21T08:45:24.992789+0000","last_fullsized":"2024-04-21T08:45:24.992789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992907+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:45:24.992907+0000","last_peered":"2024-04-21T08:45:24.992907+0000","last_clean":"2024-04-21T08:45:24.992907+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:45:24.992907+0000","last_undegraded":"2024-04-21T08:45:24.992907+0000","last_fullsized":"2024-04-21T08:45:24.992907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903912+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:45:24.903912+0000","last_peered":"2024-04-21T08:45:24.903912+0000","last_clean":"2024-04-21T08:45:24.903912+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:45:24.903912+0000","last_undegraded":"2024-04-21T08:45:24.903912+0000","last_fullsized":"2024-04-21T08:45:24.903912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993629+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:45:24.993629+0000","last_peered":"2024-04-21T08:45:24.993629+0000","last_clean":"2024-04-21T08:45:24.993629+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:45:24.993629+0000","last_undegraded":"2024-04-21T08:45:24.993629+0000","last_fullsized":"2024-04-21T08:45:24.993629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992855+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:45:24.992855+0000","last_peered":"2024-04-21T08:45:24.992855+0000","last_clean":"2024-04-21T08:45:24.992855+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:45:24.992855+0000","last_undegraded":"2024-04-21T08:45:24.992855+0000","last_fullsized":"2024-04-21T08:45:24.992855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993149+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:45:24.993149+0000","last_peered":"2024-04-21T08:45:24.993149+0000","last_clean":"2024-04-21T08:45:24.993149+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:45:24.993149+0000","last_undegraded":"2024-04-21T08:45:24.993149+0000","last_fullsized":"2024-04-21T08:45:24.993149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337962+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:45:25.337962+0000","last_peered":"2024-04-21T08:45:25.337962+0000","last_clean":"2024-04-21T08:45:25.337962+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:45:25.337962+0000","last_undegraded":"2024-04-21T08:45:25.337962+0000","last_fullsized":"2024-04-21T08:45:25.337962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904574+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:45:24.904574+0000","last_peered":"2024-04-21T08:45:24.904574+0000","last_clean":"2024-04-21T08:45:24.904574+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:45:24.904574+0000","last_undegraded":"2024-04-21T08:45:24.904574+0000","last_fullsized":"2024-04-21T08:45:24.904574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993021+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:45:24.993021+0000","last_peered":"2024-04-21T08:45:24.993021+0000","last_clean":"2024-04-21T08:45:24.993021+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:45:24.993021+0000","last_undegraded":"2024-04-21T08:45:24.993021+0000","last_fullsized":"2024-04-21T08:45:24.993021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992689+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:45:24.992689+0000","last_peered":"2024-04-21T08:45:24.992689+0000","last_clean":"2024-04-21T08:45:24.992689+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:45:24.992689+0000","last_undegraded":"2024-04-21T08:45:24.992689+0000","last_fullsized":"2024-04-21T08:45:24.992689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993192+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:45:24.993192+0000","last_peered":"2024-04-21T08:45:24.993192+0000","last_clean":"2024-04-21T08:45:24.993192+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:45:24.993192+0000","last_undegraded":"2024-04-21T08:45:24.993192+0000","last_fullsized":"2024-04-21T08:45:24.993192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'37","reported_seq":328,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.632907+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:45:25.632907+0000","last_peered":"2024-04-21T08:45:25.632907+0000","last_clean":"2024-04-21T08:45:25.632907+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:45:25.632907+0000","last_undegraded":"2024-04-21T08:45:25.632907+0000","last_fullsized":"2024-04-21T08:45:25.632907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":258,"num_read_kb":198,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992174+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:45:24.992174+0000","last_peered":"2024-04-21T08:45:24.992174+0000","last_clean":"2024-04-21T08:45:24.992174+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:45:24.992174+0000","last_undegraded":"2024-04-21T08:45:24.992174+0000","last_fullsized":"2024-04-21T08:45:24.992174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993263+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:45:24.993263+0000","last_peered":"2024-04-21T08:45:24.993263+0000","last_clean":"2024-04-21T08:45:24.993263+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:45:24.993263+0000","last_undegraded":"2024-04-21T08:45:24.993263+0000","last_fullsized":"2024-04-21T08:45:24.993263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993119+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:45:24.993119+0000","last_peered":"2024-04-21T08:45:24.993119+0000","last_clean":"2024-04-21T08:45:24.993119+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:45:24.993119+0000","last_undegraded":"2024-04-21T08:45:24.993119+0000","last_fullsized":"2024-04-21T08:45:24.993119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992385+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:45:24.992385+0000","last_peered":"2024-04-21T08:45:24.992385+0000","last_clean":"2024-04-21T08:45:24.992385+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:45:24.992385+0000","last_undegraded":"2024-04-21T08:45:24.992385+0000","last_fullsized":"2024-04-21T08:45:24.992385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993270+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:45:24.993270+0000","last_peered":"2024-04-21T08:45:24.993270+0000","last_clean":"2024-04-21T08:45:24.993270+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:45:24.993270+0000","last_undegraded":"2024-04-21T08:45:24.993270+0000","last_fullsized":"2024-04-21T08:45:24.993270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992679+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:45:24.992679+0000","last_peered":"2024-04-21T08:45:24.992679+0000","last_clean":"2024-04-21T08:45:24.992679+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:45:24.992679+0000","last_undegraded":"2024-04-21T08:45:24.992679+0000","last_fullsized":"2024-04-21T08:45:24.992679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992953+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:45:24.992953+0000","last_peered":"2024-04-21T08:45:24.992953+0000","last_clean":"2024-04-21T08:45:24.992953+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:45:24.992953+0000","last_undegraded":"2024-04-21T08:45:24.992953+0000","last_fullsized":"2024-04-21T08:45:24.992953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993936+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:45:24.993936+0000","last_peered":"2024-04-21T08:45:24.993936+0000","last_clean":"2024-04-21T08:45:24.993936+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:45:24.993936+0000","last_undegraded":"2024-04-21T08:45:24.993936+0000","last_fullsized":"2024-04-21T08:45:24.993936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904172+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:45:24.904172+0000","last_peered":"2024-04-21T08:45:24.904172+0000","last_clean":"2024-04-21T08:45:24.904172+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:45:24.904172+0000","last_undegraded":"2024-04-21T08:45:24.904172+0000","last_fullsized":"2024-04-21T08:45:24.904172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992489+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:45:24.992489+0000","last_peered":"2024-04-21T08:45:24.992489+0000","last_clean":"2024-04-21T08:45:24.992489+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:45:24.992489+0000","last_undegraded":"2024-04-21T08:45:24.992489+0000","last_fullsized":"2024-04-21T08:45:24.992489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992860+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:45:24.992860+0000","last_peered":"2024-04-21T08:45:24.992860+0000","last_clean":"2024-04-21T08:45:24.992860+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:45:24.992860+0000","last_undegraded":"2024-04-21T08:45:24.992860+0000","last_fullsized":"2024-04-21T08:45:24.992860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992036+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:45:24.992036+0000","last_peered":"2024-04-21T08:45:24.992036+0000","last_clean":"2024-04-21T08:45:24.992036+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:45:24.992036+0000","last_undegraded":"2024-04-21T08:45:24.992036+0000","last_fullsized":"2024-04-21T08:45:24.992036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992792+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:45:24.992792+0000","last_peered":"2024-04-21T08:45:24.992792+0000","last_clean":"2024-04-21T08:45:24.992792+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:45:24.992792+0000","last_undegraded":"2024-04-21T08:45:24.992792+0000","last_fullsized":"2024-04-21T08:45:24.992792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992835+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:45:24.992835+0000","last_peered":"2024-04-21T08:45:24.992835+0000","last_clean":"2024-04-21T08:45:24.992835+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:45:24.992835+0000","last_undegraded":"2024-04-21T08:45:24.992835+0000","last_fullsized":"2024-04-21T08:45:24.992835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992275+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:45:24.992275+0000","last_peered":"2024-04-21T08:45:24.992275+0000","last_clean":"2024-04-21T08:45:24.992275+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:45:24.992275+0000","last_undegraded":"2024-04-21T08:45:24.992275+0000","last_fullsized":"2024-04-21T08:45:24.992275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993653+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:45:24.993653+0000","last_peered":"2024-04-21T08:45:24.993653+0000","last_clean":"2024-04-21T08:45:24.993653+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:45:24.993653+0000","last_undegraded":"2024-04-21T08:45:24.993653+0000","last_fullsized":"2024-04-21T08:45:24.993653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992241+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:45:24.992241+0000","last_peered":"2024-04-21T08:45:24.992241+0000","last_clean":"2024-04-21T08:45:24.992241+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:45:24.992241+0000","last_undegraded":"2024-04-21T08:45:24.992241+0000","last_fullsized":"2024-04-21T08:45:24.992241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993525+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:45:24.993525+0000","last_peered":"2024-04-21T08:45:24.993525+0000","last_clean":"2024-04-21T08:45:24.993525+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:45:24.993525+0000","last_undegraded":"2024-04-21T08:45:24.993525+0000","last_fullsized":"2024-04-21T08:45:24.993525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993389+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:45:24.993389+0000","last_peered":"2024-04-21T08:45:24.993389+0000","last_clean":"2024-04-21T08:45:24.993389+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:45:24.993389+0000","last_undegraded":"2024-04-21T08:45:24.993389+0000","last_fullsized":"2024-04-21T08:45:24.993389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992746+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:45:24.992746+0000","last_peered":"2024-04-21T08:45:24.992746+0000","last_clean":"2024-04-21T08:45:24.992746+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:45:24.992746+0000","last_undegraded":"2024-04-21T08:45:24.992746+0000","last_fullsized":"2024-04-21T08:45:24.992746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992763+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:45:24.992763+0000","last_peered":"2024-04-21T08:45:24.992763+0000","last_clean":"2024-04-21T08:45:24.992763+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:45:24.992763+0000","last_undegraded":"2024-04-21T08:45:24.992763+0000","last_fullsized":"2024-04-21T08:45:24.992763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992403+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:45:24.992403+0000","last_peered":"2024-04-21T08:45:24.992403+0000","last_clean":"2024-04-21T08:45:24.992403+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:45:24.992403+0000","last_undegraded":"2024-04-21T08:45:24.992403+0000","last_fullsized":"2024-04-21T08:45:24.992403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992847+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:45:24.992847+0000","last_peered":"2024-04-21T08:45:24.992847+0000","last_clean":"2024-04-21T08:45:24.992847+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:45:24.992847+0000","last_undegraded":"2024-04-21T08:45:24.992847+0000","last_fullsized":"2024-04-21T08:45:24.992847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992175+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:45:24.992175+0000","last_peered":"2024-04-21T08:45:24.992175+0000","last_clean":"2024-04-21T08:45:24.992175+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:45:24.992175+0000","last_undegraded":"2024-04-21T08:45:24.992175+0000","last_fullsized":"2024-04-21T08:45:24.992175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993373+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:45:24.993373+0000","last_peered":"2024-04-21T08:45:24.993373+0000","last_clean":"2024-04-21T08:45:24.993373+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:45:24.993373+0000","last_undegraded":"2024-04-21T08:45:24.993373+0000","last_fullsized":"2024-04-21T08:45:24.993373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904114+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:45:24.904114+0000","last_peered":"2024-04-21T08:45:24.904114+0000","last_clean":"2024-04-21T08:45:24.904114+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:45:24.904114+0000","last_undegraded":"2024-04-21T08:45:24.904114+0000","last_fullsized":"2024-04-21T08:45:24.904114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904460+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:45:24.904460+0000","last_peered":"2024-04-21T08:45:24.904460+0000","last_clean":"2024-04-21T08:45:24.904460+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:45:24.904460+0000","last_undegraded":"2024-04-21T08:45:24.904460+0000","last_fullsized":"2024-04-21T08:45:24.904460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992255+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:45:24.992255+0000","last_peered":"2024-04-21T08:45:24.992255+0000","last_clean":"2024-04-21T08:45:24.992255+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:45:24.992255+0000","last_undegraded":"2024-04-21T08:45:24.992255+0000","last_fullsized":"2024-04-21T08:45:24.992255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992034+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:45:24.992034+0000","last_peered":"2024-04-21T08:45:24.992034+0000","last_clean":"2024-04-21T08:45:24.992034+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:45:24.992034+0000","last_undegraded":"2024-04-21T08:45:24.992034+0000","last_fullsized":"2024-04-21T08:45:24.992034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992367+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:45:24.992367+0000","last_peered":"2024-04-21T08:45:24.992367+0000","last_clean":"2024-04-21T08:45:24.992367+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:45:24.992367+0000","last_undegraded":"2024-04-21T08:45:24.992367+0000","last_fullsized":"2024-04-21T08:45:24.992367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993257+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:45:24.993257+0000","last_peered":"2024-04-21T08:45:24.993257+0000","last_clean":"2024-04-21T08:45:24.993257+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:45:24.993257+0000","last_undegraded":"2024-04-21T08:45:24.993257+0000","last_fullsized":"2024-04-21T08:45:24.993257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992262+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:45:24.992262+0000","last_peered":"2024-04-21T08:45:24.992262+0000","last_clean":"2024-04-21T08:45:24.992262+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:45:24.992262+0000","last_undegraded":"2024-04-21T08:45:24.992262+0000","last_fullsized":"2024-04-21T08:45:24.992262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992695+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:45:24.992695+0000","last_peered":"2024-04-21T08:45:24.992695+0000","last_clean":"2024-04-21T08:45:24.992695+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:45:24.992695+0000","last_undegraded":"2024-04-21T08:45:24.992695+0000","last_fullsized":"2024-04-21T08:45:24.992695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903778+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:45:24.903778+0000","last_peered":"2024-04-21T08:45:24.903778+0000","last_clean":"2024-04-21T08:45:24.903778+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:45:24.903778+0000","last_undegraded":"2024-04-21T08:45:24.903778+0000","last_fullsized":"2024-04-21T08:45:24.903778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'43","reported_seq":99,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.136133+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:45:25.136133+0000","last_peered":"2024-04-21T08:45:25.136133+0000","last_clean":"2024-04-21T08:45:25.136133+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:45:25.136133+0000","last_undegraded":"2024-04-21T08:45:25.136133+0000","last_fullsized":"2024-04-21T08:45:25.136133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904474+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:45:24.904474+0000","last_peered":"2024-04-21T08:45:24.904474+0000","last_clean":"2024-04-21T08:45:24.904474+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:45:24.904474+0000","last_undegraded":"2024-04-21T08:45:24.904474+0000","last_fullsized":"2024-04-21T08:45:24.904474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903473+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:45:24.903473+0000","last_peered":"2024-04-21T08:45:24.903473+0000","last_clean":"2024-04-21T08:45:24.903473+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:45:24.903473+0000","last_undegraded":"2024-04-21T08:45:24.903473+0000","last_fullsized":"2024-04-21T08:45:24.903473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904682+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:45:24.904682+0000","last_peered":"2024-04-21T08:45:24.904682+0000","last_clean":"2024-04-21T08:45:24.904682+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:45:24.904682+0000","last_undegraded":"2024-04-21T08:45:24.904682+0000","last_fullsized":"2024-04-21T08:45:24.904682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337838+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:45:25.337838+0000","last_peered":"2024-04-21T08:45:25.337838+0000","last_clean":"2024-04-21T08:45:25.337838+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:45:25.337838+0000","last_undegraded":"2024-04-21T08:45:25.337838+0000","last_fullsized":"2024-04-21T08:45:25.337838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904527+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:45:24.904527+0000","last_peered":"2024-04-21T08:45:24.904527+0000","last_clean":"2024-04-21T08:45:24.904527+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:45:24.904527+0000","last_undegraded":"2024-04-21T08:45:24.904527+0000","last_fullsized":"2024-04-21T08:45:24.904527+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991872+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:45:24.991872+0000","last_peered":"2024-04-21T08:45:24.991872+0000","last_clean":"2024-04-21T08:45:24.991872+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:45:24.991872+0000","last_undegraded":"2024-04-21T08:45:24.991872+0000","last_fullsized":"2024-04-21T08:45:24.991872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991968+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:45:24.991968+0000","last_peered":"2024-04-21T08:45:24.991968+0000","last_clean":"2024-04-21T08:45:24.991968+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:45:24.991968+0000","last_undegraded":"2024-04-21T08:45:24.991968+0000","last_fullsized":"2024-04-21T08:45:24.991968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992202+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:45:24.992202+0000","last_peered":"2024-04-21T08:45:24.992202+0000","last_clean":"2024-04-21T08:45:24.992202+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:45:24.992202+0000","last_undegraded":"2024-04-21T08:45:24.992202+0000","last_fullsized":"2024-04-21T08:45:24.992202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903893+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:45:24.903893+0000","last_peered":"2024-04-21T08:45:24.903893+0000","last_clean":"2024-04-21T08:45:24.903893+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:45:24.903893+0000","last_undegraded":"2024-04-21T08:45:24.903893+0000","last_fullsized":"2024-04-21T08:45:24.903893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992008+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:45:24.992008+0000","last_peered":"2024-04-21T08:45:24.992008+0000","last_clean":"2024-04-21T08:45:24.992008+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:45:24.992008+0000","last_undegraded":"2024-04-21T08:45:24.992008+0000","last_fullsized":"2024-04-21T08:45:24.992008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904288+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:45:24.904288+0000","last_peered":"2024-04-21T08:45:24.904288+0000","last_clean":"2024-04-21T08:45:24.904288+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:45:24.904288+0000","last_undegraded":"2024-04-21T08:45:24.904288+0000","last_fullsized":"2024-04-21T08:45:24.904288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903726+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:45:24.903726+0000","last_peered":"2024-04-21T08:45:24.903726+0000","last_clean":"2024-04-21T08:45:24.903726+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:45:24.903726+0000","last_undegraded":"2024-04-21T08:45:24.903726+0000","last_fullsized":"2024-04-21T08:45:24.903726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903977+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:45:24.903977+0000","last_peered":"2024-04-21T08:45:24.903977+0000","last_clean":"2024-04-21T08:45:24.903977+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:45:24.903977+0000","last_undegraded":"2024-04-21T08:45:24.903977+0000","last_fullsized":"2024-04-21T08:45:24.903977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904319+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:45:24.904319+0000","last_peered":"2024-04-21T08:45:24.904319+0000","last_clean":"2024-04-21T08:45:24.904319+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:45:24.904319+0000","last_undegraded":"2024-04-21T08:45:24.904319+0000","last_fullsized":"2024-04-21T08:45:24.904319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993372+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:45:24.993372+0000","last_peered":"2024-04-21T08:45:24.993372+0000","last_clean":"2024-04-21T08:45:24.993372+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:45:24.993372+0000","last_undegraded":"2024-04-21T08:45:24.993372+0000","last_fullsized":"2024-04-21T08:45:24.993372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992918+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:45:24.992918+0000","last_peered":"2024-04-21T08:45:24.992918+0000","last_clean":"2024-04-21T08:45:24.992918+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:45:24.992918+0000","last_undegraded":"2024-04-21T08:45:24.992918+0000","last_fullsized":"2024-04-21T08:45:24.992918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992146+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:45:24.992146+0000","last_peered":"2024-04-21T08:45:24.992146+0000","last_clean":"2024-04-21T08:45:24.992146+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:45:24.992146+0000","last_undegraded":"2024-04-21T08:45:24.992146+0000","last_fullsized":"2024-04-21T08:45:24.992146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904004+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:45:24.904004+0000","last_peered":"2024-04-21T08:45:24.904004+0000","last_clean":"2024-04-21T08:45:24.904004+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:45:24.904004+0000","last_undegraded":"2024-04-21T08:45:24.904004+0000","last_fullsized":"2024-04-21T08:45:24.904004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337411+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:45:25.337411+0000","last_peered":"2024-04-21T08:45:25.337411+0000","last_clean":"2024-04-21T08:45:25.337411+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:45:25.337411+0000","last_undegraded":"2024-04-21T08:45:25.337411+0000","last_fullsized":"2024-04-21T08:45:25.337411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.128236+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:45:25.128236+0000","last_peered":"2024-04-21T08:45:25.128236+0000","last_clean":"2024-04-21T08:45:25.128236+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:45:25.128236+0000","last_undegraded":"2024-04-21T08:45:25.128236+0000","last_fullsized":"2024-04-21T08:45:25.128236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992998+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:45:24.992998+0000","last_peered":"2024-04-21T08:45:24.992998+0000","last_clean":"2024-04-21T08:45:24.992998+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:45:24.992998+0000","last_undegraded":"2024-04-21T08:45:24.992998+0000","last_fullsized":"2024-04-21T08:45:24.992998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992640+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:45:24.992640+0000","last_peered":"2024-04-21T08:45:24.992640+0000","last_clean":"2024-04-21T08:45:24.992640+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:45:24.992640+0000","last_undegraded":"2024-04-21T08:45:24.992640+0000","last_fullsized":"2024-04-21T08:45:24.992640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992686+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:45:24.992686+0000","last_peered":"2024-04-21T08:45:24.992686+0000","last_clean":"2024-04-21T08:45:24.992686+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:45:24.992686+0000","last_undegraded":"2024-04-21T08:45:24.992686+0000","last_fullsized":"2024-04-21T08:45:24.992686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993176+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:45:24.993176+0000","last_peered":"2024-04-21T08:45:24.993176+0000","last_clean":"2024-04-21T08:45:24.993176+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:45:24.993176+0000","last_undegraded":"2024-04-21T08:45:24.993176+0000","last_fullsized":"2024-04-21T08:45:24.993176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992191+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:45:24.992191+0000","last_peered":"2024-04-21T08:45:24.992191+0000","last_clean":"2024-04-21T08:45:24.992191+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:45:24.992191+0000","last_undegraded":"2024-04-21T08:45:24.992191+0000","last_fullsized":"2024-04-21T08:45:24.992191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903722+0000","last_change":"2024-04-21T08:44:47.315087+0000","last_active":"2024-04-21T08:45:24.903722+0000","last_peered":"2024-04-21T08:45:24.903722+0000","last_clean":"2024-04-21T08:45:24.903722+0000","last_became_active":"2024-04-21T08:44:47.314647+0000","last_became_peered":"2024-04-21T08:44:47.314647+0000","last_unstale":"2024-04-21T08:45:24.903722+0000","last_undegraded":"2024-04-21T08:45:24.903722+0000","last_fullsized":"2024-04-21T08:45:24.903722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:20.425332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903546+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:45:24.903546+0000","last_peered":"2024-04-21T08:45:24.903546+0000","last_clean":"2024-04-21T08:45:24.903546+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:45:24.903546+0000","last_undegraded":"2024-04-21T08:45:24.903546+0000","last_fullsized":"2024-04-21T08:45:24.903546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337673+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:45:25.337673+0000","last_peered":"2024-04-21T08:45:25.337673+0000","last_clean":"2024-04-21T08:45:25.337673+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:45:25.337673+0000","last_undegraded":"2024-04-21T08:45:25.337673+0000","last_fullsized":"2024-04-21T08:45:25.337673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992553+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:45:24.992553+0000","last_peered":"2024-04-21T08:45:24.992553+0000","last_clean":"2024-04-21T08:45:24.992553+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:45:24.992553+0000","last_undegraded":"2024-04-21T08:45:24.992553+0000","last_fullsized":"2024-04-21T08:45:24.992553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904194+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:45:24.904194+0000","last_peered":"2024-04-21T08:45:24.904194+0000","last_clean":"2024-04-21T08:45:24.904194+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:45:24.904194+0000","last_undegraded":"2024-04-21T08:45:24.904194+0000","last_fullsized":"2024-04-21T08:45:24.904194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904716+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:45:24.904716+0000","last_peered":"2024-04-21T08:45:24.904716+0000","last_clean":"2024-04-21T08:45:24.904716+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:45:24.904716+0000","last_undegraded":"2024-04-21T08:45:24.904716+0000","last_fullsized":"2024-04-21T08:45:24.904716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992795+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:45:24.992795+0000","last_peered":"2024-04-21T08:45:24.992795+0000","last_clean":"2024-04-21T08:45:24.992795+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:45:24.992795+0000","last_undegraded":"2024-04-21T08:45:24.992795+0000","last_fullsized":"2024-04-21T08:45:24.992795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903992+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:45:24.903992+0000","last_peered":"2024-04-21T08:45:24.903992+0000","last_clean":"2024-04-21T08:45:24.903992+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:45:24.903992+0000","last_undegraded":"2024-04-21T08:45:24.903992+0000","last_fullsized":"2024-04-21T08:45:24.903992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992360+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:45:24.992360+0000","last_peered":"2024-04-21T08:45:24.992360+0000","last_clean":"2024-04-21T08:45:24.992360+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:45:24.992360+0000","last_undegraded":"2024-04-21T08:45:24.992360+0000","last_fullsized":"2024-04-21T08:45:24.992360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992585+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:45:24.992585+0000","last_peered":"2024-04-21T08:45:24.992585+0000","last_clean":"2024-04-21T08:45:24.992585+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:45:24.992585+0000","last_undegraded":"2024-04-21T08:45:24.992585+0000","last_fullsized":"2024-04-21T08:45:24.992585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993390+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:45:24.993390+0000","last_peered":"2024-04-21T08:45:24.993390+0000","last_clean":"2024-04-21T08:45:24.993390+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:45:24.993390+0000","last_undegraded":"2024-04-21T08:45:24.993390+0000","last_fullsized":"2024-04-21T08:45:24.993390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":110016,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":354,"num_read_kb":281,"num_write":200,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1787046,"data_compressed":80790,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528781,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1169281,"data_compressed":64942,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":926641,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":922567,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1123793,"data_compressed":66315,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":524651,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514896,"data_compressed":26642,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1049571,"data_compressed":54945,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":528769,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1110016,"data_stored":1250897,"data_compressed":72471,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":201862,"data_compressed":7784,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:45:27.398 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-21T08:45:27.399 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:45:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.533+0000 7f41b78ac640 1 -- 172.21.15.57:0/3931241470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0107c70 msgr2=0x7f41b010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.533+0000 7f41b78ac640 1 --2- 172.21.15.57:0/3931241470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0107c70 0x7f41b010a0c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009ca0 tx=0x7f41a802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 -- 172.21.15.57:0/3931241470 shutdown_connections 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 --2- 172.21.15.57:0/3931241470 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0107c70 0x7f41b010a0c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 --2- 172.21.15.57:0/3931241470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41b0105340 0x7f41b0107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 --2- 172.21.15.57:0/3931241470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0102a10 0x7f41b0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 -- 172.21.15.57:0/3931241470 >> 172.21.15.57:0/3931241470 conn(0x7f41b00fc400 msgr2=0x7f41b00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 -- 172.21.15.57:0/3931241470 shutdown_connections 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 -- 172.21.15.57:0/3931241470 wait complete. 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 Processor -- start 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.534+0000 7f41b78ac640 1 -- start start 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b78ac640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 0x7f41b0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b78ac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41b0105340 0x7f41b0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b78ac640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0107c70 0x7f41b019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b78ac640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010dad0 con 0x7f41b0102a10 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b78ac640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010d950 con 0x7f41b0107c70 2024-04-21T08:45:27.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b78ac640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f41b010dc50 con 0x7f41b0105340 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 0x7f41b0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 0x7f41b0198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46308/0 (socket says 172.21.15.57:46308) 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 -- 172.21.15.57:0/593969925 learned_addr learned my addr 172.21.15.57:0/593969925 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b4e47640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41b0105340 0x7f41b0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5e49640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0107c70 0x7f41b019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 -- 172.21.15.57:0/593969925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41b0105340 msgr2=0x7f41b0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41b0105340 0x7f41b0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 -- 172.21.15.57:0/593969925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0107c70 msgr2=0x7f41b019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0107c70 0x7f41b019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 -- 172.21.15.57:0/593969925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b019fd60 con 0x7f41b0102a10 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41b5648640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 0x7f41b0198140 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00e9f0 tx=0x7f41ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac00cd80 con 0x7f41b0102a10 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.535+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41ac0108e0 con 0x7f41b0102a10 2024-04-21T08:45:27.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.536+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b019fff0 con 0x7f41b0102a10 2024-04-21T08:45:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.536+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac018980 con 0x7f41b0102a10 2024-04-21T08:45:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.536+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b01a04b0 con 0x7f41b0102a10 2024-04-21T08:45:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.537+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b00fdef0 con 0x7f41b0102a10 2024-04-21T08:45:27.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.538+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f41ac010430 con 0x7f41b0102a10 2024-04-21T08:45:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.539+0000 7f41a67fc640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f419c0a3d20 0x7f419c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.539+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f41ac0d12e0 con 0x7f41b0102a10 2024-04-21T08:45:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.539+0000 7f41b4e47640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f419c0a3d20 0x7f419c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.540+0000 7f41b4e47640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f419c0a3d20 0x7f419c0a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41b0199650 tx=0x7f41a0005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.542+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41ac09cfd0 con 0x7f41b0102a10 2024-04-21T08:45:27.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:27 smithi136 ceph-mon[31791]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:27.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:27 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2394142876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:27.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:27 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:27 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:27 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:27 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.728+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f41b0062390 con 0x7f419c0a3d20 2024-04-21T08:45:27.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.738+0000 7f41a67fc640 1 -- 172.21.15.57:0/593969925 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339620 (secure 0 0 0) 0x7f41b0062390 con 0x7f419c0a3d20 2024-04-21T08:45:27.741 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:27.743 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:45:27.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f419c0a3d20 msgr2=0x7f419c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f419c0a3d20 0x7f419c0a61e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f41b0199650 tx=0x7f41a0005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 msgr2=0x7f41b0198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 0x7f41b0198140 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f41ac00e9f0 tx=0x7f41ac00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 shutdown_connections 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f419c0a3d20 0x7f419c0a61e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41b0107c70 0x7f41b019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41b0105340 0x7f41b0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 --2- 172.21.15.57:0/593969925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41b0102a10 0x7f41b0198140 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 >> 172.21.15.57:0/593969925 conn(0x7f41b00fc400 msgr2=0x7f41b01032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 shutdown_connections 2024-04-21T08:45:27.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.742+0000 7f41b78ac640 1 -- 172.21.15.57:0/593969925 wait complete. 2024-04-21T08:45:27.751 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:27 smithi057 ceph-mon[27968]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:27.751 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:27 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2394142876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:27.752 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:27 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.752 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:27 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.752 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:27 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.752 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:27 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.753 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-04-21T08:45:26.917452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1186284,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":490,"num_read_kb":400,"num_write":383,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365900,"kb_used_data":12536,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575924,"statfs":{"total":1151940427776,"available":1151565746176,"internally_reserved":0,"allocated":12836864,"data_stored":9729285,"data_compressed":534364,"data_compressed_allocated":720896,"data_compressed_original":4186112,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.987182"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993678+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:45:24.993678+0000","last_peered":"2024-04-21T08:45:24.993678+0000","last_clean":"2024-04-21T08:45:24.993678+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:45:24.993678+0000","last_undegraded":"2024-04-21T08:45:24.993678+0000","last_fullsized":"2024-04-21T08:45:24.993678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337688+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:45:25.337688+0000","last_peered":"2024-04-21T08:45:25.337688+0000","last_clean":"2024-04-21T08:45:25.337688+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:45:25.337688+0000","last_undegraded":"2024-04-21T08:45:25.337688+0000","last_fullsized":"2024-04-21T08:45:25.337688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338255+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:45:25.338255+0000","last_peered":"2024-04-21T08:45:25.338255+0000","last_clean":"2024-04-21T08:45:25.338255+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:45:25.338255+0000","last_undegraded":"2024-04-21T08:45:25.338255+0000","last_fullsized":"2024-04-21T08:45:25.338255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904383+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:45:24.904383+0000","last_peered":"2024-04-21T08:45:24.904383+0000","last_clean":"2024-04-21T08:45:24.904383+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:45:24.904383+0000","last_undegraded":"2024-04-21T08:45:24.904383+0000","last_fullsized":"2024-04-21T08:45:24.904383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991927+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:45:24.991927+0000","last_peered":"2024-04-21T08:45:24.991927+0000","last_clean":"2024-04-21T08:45:24.991927+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:45:24.991927+0000","last_undegraded":"2024-04-21T08:45:24.991927+0000","last_fullsized":"2024-04-21T08:45:24.991927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992137+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:45:24.992137+0000","last_peered":"2024-04-21T08:45:24.992137+0000","last_clean":"2024-04-21T08:45:24.992137+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:45:24.992137+0000","last_undegraded":"2024-04-21T08:45:24.992137+0000","last_fullsized":"2024-04-21T08:45:24.992137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992575+0000","last_change":"2024-04-21T08:44:47.312984+0000","last_active":"2024-04-21T08:45:24.992575+0000","last_peered":"2024-04-21T08:45:24.992575+0000","last_clean":"2024-04-21T08:45:24.992575+0000","last_became_active":"2024-04-21T08:44:47.312465+0000","last_became_peered":"2024-04-21T08:44:47.312465+0000","last_unstale":"2024-04-21T08:45:24.992575+0000","last_undegraded":"2024-04-21T08:45:24.992575+0000","last_fullsized":"2024-04-21T08:45:24.992575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:59:51.214620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993071+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:45:24.993071+0000","last_peered":"2024-04-21T08:45:24.993071+0000","last_clean":"2024-04-21T08:45:24.993071+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:45:24.993071+0000","last_undegraded":"2024-04-21T08:45:24.993071+0000","last_fullsized":"2024-04-21T08:45:24.993071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993708+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:45:24.993708+0000","last_peered":"2024-04-21T08:45:24.993708+0000","last_clean":"2024-04-21T08:45:24.993708+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:45:24.993708+0000","last_undegraded":"2024-04-21T08:45:24.993708+0000","last_fullsized":"2024-04-21T08:45:24.993708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993237+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:45:24.993237+0000","last_peered":"2024-04-21T08:45:24.993237+0000","last_clean":"2024-04-21T08:45:24.993237+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:45:24.993237+0000","last_undegraded":"2024-04-21T08:45:24.993237+0000","last_fullsized":"2024-04-21T08:45:24.993237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904442+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:45:24.904442+0000","last_peered":"2024-04-21T08:45:24.904442+0000","last_clean":"2024-04-21T08:45:24.904442+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:45:24.904442+0000","last_undegraded":"2024-04-21T08:45:24.904442+0000","last_fullsized":"2024-04-21T08:45:24.904442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992519+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:45:24.992519+0000","last_peered":"2024-04-21T08:45:24.992519+0000","last_clean":"2024-04-21T08:45:24.992519+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:45:24.992519+0000","last_undegraded":"2024-04-21T08:45:24.992519+0000","last_fullsized":"2024-04-21T08:45:24.992519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337366+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:45:25.337366+0000","last_peered":"2024-04-21T08:45:25.337366+0000","last_clean":"2024-04-21T08:45:25.337366+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:45:25.337366+0000","last_undegraded":"2024-04-21T08:45:25.337366+0000","last_fullsized":"2024-04-21T08:45:25.337366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337428+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:45:25.337428+0000","last_peered":"2024-04-21T08:45:25.337428+0000","last_clean":"2024-04-21T08:45:25.337428+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:45:25.337428+0000","last_undegraded":"2024-04-21T08:45:25.337428+0000","last_fullsized":"2024-04-21T08:45:25.337428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991935+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:45:24.991935+0000","last_peered":"2024-04-21T08:45:24.991935+0000","last_clean":"2024-04-21T08:45:24.991935+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:45:24.991935+0000","last_undegraded":"2024-04-21T08:45:24.991935+0000","last_fullsized":"2024-04-21T08:45:24.991935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993546+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:45:24.993546+0000","last_peered":"2024-04-21T08:45:24.993546+0000","last_clean":"2024-04-21T08:45:24.993546+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:45:24.993546+0000","last_undegraded":"2024-04-21T08:45:24.993546+0000","last_fullsized":"2024-04-21T08:45:24.993546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993979+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:45:24.993979+0000","last_peered":"2024-04-21T08:45:24.993979+0000","last_clean":"2024-04-21T08:45:24.993979+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:45:24.993979+0000","last_undegraded":"2024-04-21T08:45:24.993979+0000","last_fullsized":"2024-04-21T08:45:24.993979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904038+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:45:24.904038+0000","last_peered":"2024-04-21T08:45:24.904038+0000","last_clean":"2024-04-21T08:45:24.904038+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:45:24.904038+0000","last_undegraded":"2024-04-21T08:45:24.904038+0000","last_fullsized":"2024-04-21T08:45:24.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903605+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:45:24.903605+0000","last_peered":"2024-04-21T08:45:24.903605+0000","last_clean":"2024-04-21T08:45:24.903605+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:45:24.903605+0000","last_undegraded":"2024-04-21T08:45:24.903605+0000","last_fullsized":"2024-04-21T08:45:24.903605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.994253+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:45:24.994253+0000","last_peered":"2024-04-21T08:45:24.994253+0000","last_clean":"2024-04-21T08:45:24.994253+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:45:24.994253+0000","last_undegraded":"2024-04-21T08:45:24.994253+0000","last_fullsized":"2024-04-21T08:45:24.994253+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904457+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:45:24.904457+0000","last_peered":"2024-04-21T08:45:24.904457+0000","last_clean":"2024-04-21T08:45:24.904457+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:45:24.904457+0000","last_undegraded":"2024-04-21T08:45:24.904457+0000","last_fullsized":"2024-04-21T08:45:24.904457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904078+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:45:24.904078+0000","last_peered":"2024-04-21T08:45:24.904078+0000","last_clean":"2024-04-21T08:45:24.904078+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:45:24.904078+0000","last_undegraded":"2024-04-21T08:45:24.904078+0000","last_fullsized":"2024-04-21T08:45:24.904078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904639+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:45:24.904639+0000","last_peered":"2024-04-21T08:45:24.904639+0000","last_clean":"2024-04-21T08:45:24.904639+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:45:24.904639+0000","last_undegraded":"2024-04-21T08:45:24.904639+0000","last_fullsized":"2024-04-21T08:45:24.904639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338007+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:45:25.338007+0000","last_peered":"2024-04-21T08:45:25.338007+0000","last_clean":"2024-04-21T08:45:25.338007+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:45:25.338007+0000","last_undegraded":"2024-04-21T08:45:25.338007+0000","last_fullsized":"2024-04-21T08:45:25.338007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904338+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:45:24.904338+0000","last_peered":"2024-04-21T08:45:24.904338+0000","last_clean":"2024-04-21T08:45:24.904338+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:45:24.904338+0000","last_undegraded":"2024-04-21T08:45:24.904338+0000","last_fullsized":"2024-04-21T08:45:24.904338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992394+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:45:24.992394+0000","last_peered":"2024-04-21T08:45:24.992394+0000","last_clean":"2024-04-21T08:45:24.992394+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:45:24.992394+0000","last_undegraded":"2024-04-21T08:45:24.992394+0000","last_fullsized":"2024-04-21T08:45:24.992394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992458+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:45:24.992458+0000","last_peered":"2024-04-21T08:45:24.992458+0000","last_clean":"2024-04-21T08:45:24.992458+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:45:24.992458+0000","last_undegraded":"2024-04-21T08:45:24.992458+0000","last_fullsized":"2024-04-21T08:45:24.992458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904082+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:45:24.904082+0000","last_peered":"2024-04-21T08:45:24.904082+0000","last_clean":"2024-04-21T08:45:24.904082+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:45:24.904082+0000","last_undegraded":"2024-04-21T08:45:24.904082+0000","last_fullsized":"2024-04-21T08:45:24.904082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992123+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:45:24.992123+0000","last_peered":"2024-04-21T08:45:24.992123+0000","last_clean":"2024-04-21T08:45:24.992123+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:45:24.992123+0000","last_undegraded":"2024-04-21T08:45:24.992123+0000","last_fullsized":"2024-04-21T08:45:24.992123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903921+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:45:24.903921+0000","last_peered":"2024-04-21T08:45:24.903921+0000","last_clean":"2024-04-21T08:45:24.903921+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:45:24.903921+0000","last_undegraded":"2024-04-21T08:45:24.903921+0000","last_fullsized":"2024-04-21T08:45:24.903921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992899+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:45:24.992899+0000","last_peered":"2024-04-21T08:45:24.992899+0000","last_clean":"2024-04-21T08:45:24.992899+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:45:24.992899+0000","last_undegraded":"2024-04-21T08:45:24.992899+0000","last_fullsized":"2024-04-21T08:45:24.992899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992410+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:45:24.992410+0000","last_peered":"2024-04-21T08:45:24.992410+0000","last_clean":"2024-04-21T08:45:24.992410+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:45:24.992410+0000","last_undegraded":"2024-04-21T08:45:24.992410+0000","last_fullsized":"2024-04-21T08:45:24.992410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903852+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:45:24.903852+0000","last_peered":"2024-04-21T08:45:24.903852+0000","last_clean":"2024-04-21T08:45:24.903852+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:45:24.903852+0000","last_undegraded":"2024-04-21T08:45:24.903852+0000","last_fullsized":"2024-04-21T08:45:24.903852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992469+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:45:24.992469+0000","last_peered":"2024-04-21T08:45:24.992469+0000","last_clean":"2024-04-21T08:45:24.992469+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:45:24.992469+0000","last_undegraded":"2024-04-21T08:45:24.992469+0000","last_fullsized":"2024-04-21T08:45:24.992469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993305+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:45:24.993305+0000","last_peered":"2024-04-21T08:45:24.993305+0000","last_clean":"2024-04-21T08:45:24.993305+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:45:24.993305+0000","last_undegraded":"2024-04-21T08:45:24.993305+0000","last_fullsized":"2024-04-21T08:45:24.993305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993857+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:45:24.993857+0000","last_peered":"2024-04-21T08:45:24.993857+0000","last_clean":"2024-04-21T08:45:24.993857+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:45:24.993857+0000","last_undegraded":"2024-04-21T08:45:24.993857+0000","last_fullsized":"2024-04-21T08:45:24.993857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992300+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:45:24.992300+0000","last_peered":"2024-04-21T08:45:24.992300+0000","last_clean":"2024-04-21T08:45:24.992300+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:45:24.992300+0000","last_undegraded":"2024-04-21T08:45:24.992300+0000","last_fullsized":"2024-04-21T08:45:24.992300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903970+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:45:24.903970+0000","last_peered":"2024-04-21T08:45:24.903970+0000","last_clean":"2024-04-21T08:45:24.903970+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:45:24.903970+0000","last_undegraded":"2024-04-21T08:45:24.903970+0000","last_fullsized":"2024-04-21T08:45:24.903970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337570+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:45:25.337570+0000","last_peered":"2024-04-21T08:45:25.337570+0000","last_clean":"2024-04-21T08:45:25.337570+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:45:25.337570+0000","last_undegraded":"2024-04-21T08:45:25.337570+0000","last_fullsized":"2024-04-21T08:45:25.337570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992789+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:45:24.992789+0000","last_peered":"2024-04-21T08:45:24.992789+0000","last_clean":"2024-04-21T08:45:24.992789+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:45:24.992789+0000","last_undegraded":"2024-04-21T08:45:24.992789+0000","last_fullsized":"2024-04-21T08:45:24.992789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992907+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:45:24.992907+0000","last_peered":"2024-04-21T08:45:24.992907+0000","last_clean":"2024-04-21T08:45:24.992907+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:45:24.992907+0000","last_undegraded":"2024-04-21T08:45:24.992907+0000","last_fullsized":"2024-04-21T08:45:24.992907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903912+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:45:24.903912+0000","last_peered":"2024-04-21T08:45:24.903912+0000","last_clean":"2024-04-21T08:45:24.903912+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:45:24.903912+0000","last_undegraded":"2024-04-21T08:45:24.903912+0000","last_fullsized":"2024-04-21T08:45:24.903912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993629+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:45:24.993629+0000","last_peered":"2024-04-21T08:45:24.993629+0000","last_clean":"2024-04-21T08:45:24.993629+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:45:24.993629+0000","last_undegraded":"2024-04-21T08:45:24.993629+0000","last_fullsized":"2024-04-21T08:45:24.993629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992855+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:45:24.992855+0000","last_peered":"2024-04-21T08:45:24.992855+0000","last_clean":"2024-04-21T08:45:24.992855+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:45:24.992855+0000","last_undegraded":"2024-04-21T08:45:24.992855+0000","last_fullsized":"2024-04-21T08:45:24.992855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993149+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:45:24.993149+0000","last_peered":"2024-04-21T08:45:24.993149+0000","last_clean":"2024-04-21T08:45:24.993149+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:45:24.993149+0000","last_undegraded":"2024-04-21T08:45:24.993149+0000","last_fullsized":"2024-04-21T08:45:24.993149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337962+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:45:25.337962+0000","last_peered":"2024-04-21T08:45:25.337962+0000","last_clean":"2024-04-21T08:45:25.337962+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:45:25.337962+0000","last_undegraded":"2024-04-21T08:45:25.337962+0000","last_fullsized":"2024-04-21T08:45:25.337962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904574+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:45:24.904574+0000","last_peered":"2024-04-21T08:45:24.904574+0000","last_clean":"2024-04-21T08:45:24.904574+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:45:24.904574+0000","last_undegraded":"2024-04-21T08:45:24.904574+0000","last_fullsized":"2024-04-21T08:45:24.904574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993021+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:45:24.993021+0000","last_peered":"2024-04-21T08:45:24.993021+0000","last_clean":"2024-04-21T08:45:24.993021+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:45:24.993021+0000","last_undegraded":"2024-04-21T08:45:24.993021+0000","last_fullsized":"2024-04-21T08:45:24.993021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992689+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:45:24.992689+0000","last_peered":"2024-04-21T08:45:24.992689+0000","last_clean":"2024-04-21T08:45:24.992689+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:45:24.992689+0000","last_undegraded":"2024-04-21T08:45:24.992689+0000","last_fullsized":"2024-04-21T08:45:24.992689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993192+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:45:24.993192+0000","last_peered":"2024-04-21T08:45:24.993192+0000","last_clean":"2024-04-21T08:45:24.993192+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:45:24.993192+0000","last_undegraded":"2024-04-21T08:45:24.993192+0000","last_fullsized":"2024-04-21T08:45:24.993192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'37","reported_seq":328,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.632907+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:45:25.632907+0000","last_peered":"2024-04-21T08:45:25.632907+0000","last_clean":"2024-04-21T08:45:25.632907+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:45:25.632907+0000","last_undegraded":"2024-04-21T08:45:25.632907+0000","last_fullsized":"2024-04-21T08:45:25.632907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":258,"num_read_kb":198,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992174+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:45:24.992174+0000","last_peered":"2024-04-21T08:45:24.992174+0000","last_clean":"2024-04-21T08:45:24.992174+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:45:24.992174+0000","last_undegraded":"2024-04-21T08:45:24.992174+0000","last_fullsized":"2024-04-21T08:45:24.992174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993263+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:45:24.993263+0000","last_peered":"2024-04-21T08:45:24.993263+0000","last_clean":"2024-04-21T08:45:24.993263+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:45:24.993263+0000","last_undegraded":"2024-04-21T08:45:24.993263+0000","last_fullsized":"2024-04-21T08:45:24.993263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993119+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:45:24.993119+0000","last_peered":"2024-04-21T08:45:24.993119+0000","last_clean":"2024-04-21T08:45:24.993119+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:45:24.993119+0000","last_undegraded":"2024-04-21T08:45:24.993119+0000","last_fullsized":"2024-04-21T08:45:24.993119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992385+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:45:24.992385+0000","last_peered":"2024-04-21T08:45:24.992385+0000","last_clean":"2024-04-21T08:45:24.992385+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:45:24.992385+0000","last_undegraded":"2024-04-21T08:45:24.992385+0000","last_fullsized":"2024-04-21T08:45:24.992385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993270+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:45:24.993270+0000","last_peered":"2024-04-21T08:45:24.993270+0000","last_clean":"2024-04-21T08:45:24.993270+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:45:24.993270+0000","last_undegraded":"2024-04-21T08:45:24.993270+0000","last_fullsized":"2024-04-21T08:45:24.993270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992679+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:45:24.992679+0000","last_peered":"2024-04-21T08:45:24.992679+0000","last_clean":"2024-04-21T08:45:24.992679+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:45:24.992679+0000","last_undegraded":"2024-04-21T08:45:24.992679+0000","last_fullsized":"2024-04-21T08:45:24.992679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992953+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:45:24.992953+0000","last_peered":"2024-04-21T08:45:24.992953+0000","last_clean":"2024-04-21T08:45:24.992953+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:45:24.992953+0000","last_undegraded":"2024-04-21T08:45:24.992953+0000","last_fullsized":"2024-04-21T08:45:24.992953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993936+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:45:24.993936+0000","last_peered":"2024-04-21T08:45:24.993936+0000","last_clean":"2024-04-21T08:45:24.993936+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:45:24.993936+0000","last_undegraded":"2024-04-21T08:45:24.993936+0000","last_fullsized":"2024-04-21T08:45:24.993936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904172+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:45:24.904172+0000","last_peered":"2024-04-21T08:45:24.904172+0000","last_clean":"2024-04-21T08:45:24.904172+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:45:24.904172+0000","last_undegraded":"2024-04-21T08:45:24.904172+0000","last_fullsized":"2024-04-21T08:45:24.904172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992489+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:45:24.992489+0000","last_peered":"2024-04-21T08:45:24.992489+0000","last_clean":"2024-04-21T08:45:24.992489+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:45:24.992489+0000","last_undegraded":"2024-04-21T08:45:24.992489+0000","last_fullsized":"2024-04-21T08:45:24.992489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992860+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:45:24.992860+0000","last_peered":"2024-04-21T08:45:24.992860+0000","last_clean":"2024-04-21T08:45:24.992860+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:45:24.992860+0000","last_undegraded":"2024-04-21T08:45:24.992860+0000","last_fullsized":"2024-04-21T08:45:24.992860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992036+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:45:24.992036+0000","last_peered":"2024-04-21T08:45:24.992036+0000","last_clean":"2024-04-21T08:45:24.992036+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:45:24.992036+0000","last_undegraded":"2024-04-21T08:45:24.992036+0000","last_fullsized":"2024-04-21T08:45:24.992036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992792+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:45:24.992792+0000","last_peered":"2024-04-21T08:45:24.992792+0000","last_clean":"2024-04-21T08:45:24.992792+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:45:24.992792+0000","last_undegraded":"2024-04-21T08:45:24.992792+0000","last_fullsized":"2024-04-21T08:45:24.992792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992835+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:45:24.992835+0000","last_peered":"2024-04-21T08:45:24.992835+0000","last_clean":"2024-04-21T08:45:24.992835+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:45:24.992835+0000","last_undegraded":"2024-04-21T08:45:24.992835+0000","last_fullsized":"2024-04-21T08:45:24.992835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992275+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:45:24.992275+0000","last_peered":"2024-04-21T08:45:24.992275+0000","last_clean":"2024-04-21T08:45:24.992275+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:45:24.992275+0000","last_undegraded":"2024-04-21T08:45:24.992275+0000","last_fullsized":"2024-04-21T08:45:24.992275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993653+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:45:24.993653+0000","last_peered":"2024-04-21T08:45:24.993653+0000","last_clean":"2024-04-21T08:45:24.993653+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:45:24.993653+0000","last_undegraded":"2024-04-21T08:45:24.993653+0000","last_fullsized":"2024-04-21T08:45:24.993653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992241+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:45:24.992241+0000","last_peered":"2024-04-21T08:45:24.992241+0000","last_clean":"2024-04-21T08:45:24.992241+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:45:24.992241+0000","last_undegraded":"2024-04-21T08:45:24.992241+0000","last_fullsized":"2024-04-21T08:45:24.992241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993525+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:45:24.993525+0000","last_peered":"2024-04-21T08:45:24.993525+0000","last_clean":"2024-04-21T08:45:24.993525+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:45:24.993525+0000","last_undegraded":"2024-04-21T08:45:24.993525+0000","last_fullsized":"2024-04-21T08:45:24.993525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993389+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:45:24.993389+0000","last_peered":"2024-04-21T08:45:24.993389+0000","last_clean":"2024-04-21T08:45:24.993389+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:45:24.993389+0000","last_undegraded":"2024-04-21T08:45:24.993389+0000","last_fullsized":"2024-04-21T08:45:24.993389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992746+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:45:24.992746+0000","last_peered":"2024-04-21T08:45:24.992746+0000","last_clean":"2024-04-21T08:45:24.992746+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:45:24.992746+0000","last_undegraded":"2024-04-21T08:45:24.992746+0000","last_fullsized":"2024-04-21T08:45:24.992746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992763+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:45:24.992763+0000","last_peered":"2024-04-21T08:45:24.992763+0000","last_clean":"2024-04-21T08:45:24.992763+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:45:24.992763+0000","last_undegraded":"2024-04-21T08:45:24.992763+0000","last_fullsized":"2024-04-21T08:45:24.992763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992403+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:45:24.992403+0000","last_peered":"2024-04-21T08:45:24.992403+0000","last_clean":"2024-04-21T08:45:24.992403+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:45:24.992403+0000","last_undegraded":"2024-04-21T08:45:24.992403+0000","last_fullsized":"2024-04-21T08:45:24.992403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992847+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:45:24.992847+0000","last_peered":"2024-04-21T08:45:24.992847+0000","last_clean":"2024-04-21T08:45:24.992847+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:45:24.992847+0000","last_undegraded":"2024-04-21T08:45:24.992847+0000","last_fullsized":"2024-04-21T08:45:24.992847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992175+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:45:24.992175+0000","last_peered":"2024-04-21T08:45:24.992175+0000","last_clean":"2024-04-21T08:45:24.992175+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:45:24.992175+0000","last_undegraded":"2024-04-21T08:45:24.992175+0000","last_fullsized":"2024-04-21T08:45:24.992175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993373+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:45:24.993373+0000","last_peered":"2024-04-21T08:45:24.993373+0000","last_clean":"2024-04-21T08:45:24.993373+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:45:24.993373+0000","last_undegraded":"2024-04-21T08:45:24.993373+0000","last_fullsized":"2024-04-21T08:45:24.993373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904114+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:45:24.904114+0000","last_peered":"2024-04-21T08:45:24.904114+0000","last_clean":"2024-04-21T08:45:24.904114+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:45:24.904114+0000","last_undegraded":"2024-04-21T08:45:24.904114+0000","last_fullsized":"2024-04-21T08:45:24.904114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904460+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:45:24.904460+0000","last_peered":"2024-04-21T08:45:24.904460+0000","last_clean":"2024-04-21T08:45:24.904460+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:45:24.904460+0000","last_undegraded":"2024-04-21T08:45:24.904460+0000","last_fullsized":"2024-04-21T08:45:24.904460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992255+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:45:24.992255+0000","last_peered":"2024-04-21T08:45:24.992255+0000","last_clean":"2024-04-21T08:45:24.992255+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:45:24.992255+0000","last_undegraded":"2024-04-21T08:45:24.992255+0000","last_fullsized":"2024-04-21T08:45:24.992255+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992034+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:45:24.992034+0000","last_peered":"2024-04-21T08:45:24.992034+0000","last_clean":"2024-04-21T08:45:24.992034+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:45:24.992034+0000","last_undegraded":"2024-04-21T08:45:24.992034+0000","last_fullsized":"2024-04-21T08:45:24.992034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992367+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:45:24.992367+0000","last_peered":"2024-04-21T08:45:24.992367+0000","last_clean":"2024-04-21T08:45:24.992367+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:45:24.992367+0000","last_undegraded":"2024-04-21T08:45:24.992367+0000","last_fullsized":"2024-04-21T08:45:24.992367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993257+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:45:24.993257+0000","last_peered":"2024-04-21T08:45:24.993257+0000","last_clean":"2024-04-21T08:45:24.993257+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:45:24.993257+0000","last_undegraded":"2024-04-21T08:45:24.993257+0000","last_fullsized":"2024-04-21T08:45:24.993257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992262+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:45:24.992262+0000","last_peered":"2024-04-21T08:45:24.992262+0000","last_clean":"2024-04-21T08:45:24.992262+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:45:24.992262+0000","last_undegraded":"2024-04-21T08:45:24.992262+0000","last_fullsized":"2024-04-21T08:45:24.992262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992695+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:45:24.992695+0000","last_peered":"2024-04-21T08:45:24.992695+0000","last_clean":"2024-04-21T08:45:24.992695+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:45:24.992695+0000","last_undegraded":"2024-04-21T08:45:24.992695+0000","last_fullsized":"2024-04-21T08:45:24.992695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903778+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:45:24.903778+0000","last_peered":"2024-04-21T08:45:24.903778+0000","last_clean":"2024-04-21T08:45:24.903778+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:45:24.903778+0000","last_undegraded":"2024-04-21T08:45:24.903778+0000","last_fullsized":"2024-04-21T08:45:24.903778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'43","reported_seq":99,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.136133+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:45:25.136133+0000","last_peered":"2024-04-21T08:45:25.136133+0000","last_clean":"2024-04-21T08:45:25.136133+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:45:25.136133+0000","last_undegraded":"2024-04-21T08:45:25.136133+0000","last_fullsized":"2024-04-21T08:45:25.136133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":77,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904474+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:45:24.904474+0000","last_peered":"2024-04-21T08:45:24.904474+0000","last_clean":"2024-04-21T08:45:24.904474+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:45:24.904474+0000","last_undegraded":"2024-04-21T08:45:24.904474+0000","last_fullsized":"2024-04-21T08:45:24.904474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903473+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:45:24.903473+0000","last_peered":"2024-04-21T08:45:24.903473+0000","last_clean":"2024-04-21T08:45:24.903473+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:45:24.903473+0000","last_undegraded":"2024-04-21T08:45:24.903473+0000","last_fullsized":"2024-04-21T08:45:24.903473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904682+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:45:24.904682+0000","last_peered":"2024-04-21T08:45:24.904682+0000","last_clean":"2024-04-21T08:45:24.904682+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:45:24.904682+0000","last_undegraded":"2024-04-21T08:45:24.904682+0000","last_fullsized":"2024-04-21T08:45:24.904682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337838+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:45:25.337838+0000","last_peered":"2024-04-21T08:45:25.337838+0000","last_clean":"2024-04-21T08:45:25.337838+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:45:25.337838+0000","last_undegraded":"2024-04-21T08:45:25.337838+0000","last_fullsized":"2024-04-21T08:45:25.337838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904527+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:45:24.904527+0000","last_peered":"2024-04-21T08:45:24.904527+0000","last_clean":"2024-04-21T08:45:24.904527+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:45:24.904527+0000","last_undegraded":"2024-04-21T08:45:24.904527+0000","last_fullsized":"2024-04-21T08:45:24.904527+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991872+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:45:24.991872+0000","last_peered":"2024-04-21T08:45:24.991872+0000","last_clean":"2024-04-21T08:45:24.991872+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:45:24.991872+0000","last_undegraded":"2024-04-21T08:45:24.991872+0000","last_fullsized":"2024-04-21T08:45:24.991872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991968+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:45:24.991968+0000","last_peered":"2024-04-21T08:45:24.991968+0000","last_clean":"2024-04-21T08:45:24.991968+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:45:24.991968+0000","last_undegraded":"2024-04-21T08:45:24.991968+0000","last_fullsized":"2024-04-21T08:45:24.991968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992202+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:45:24.992202+0000","last_peered":"2024-04-21T08:45:24.992202+0000","last_clean":"2024-04-21T08:45:24.992202+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:45:24.992202+0000","last_undegraded":"2024-04-21T08:45:24.992202+0000","last_fullsized":"2024-04-21T08:45:24.992202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903893+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:45:24.903893+0000","last_peered":"2024-04-21T08:45:24.903893+0000","last_clean":"2024-04-21T08:45:24.903893+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:45:24.903893+0000","last_undegraded":"2024-04-21T08:45:24.903893+0000","last_fullsized":"2024-04-21T08:45:24.903893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992008+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:45:24.992008+0000","last_peered":"2024-04-21T08:45:24.992008+0000","last_clean":"2024-04-21T08:45:24.992008+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:45:24.992008+0000","last_undegraded":"2024-04-21T08:45:24.992008+0000","last_fullsized":"2024-04-21T08:45:24.992008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904288+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:45:24.904288+0000","last_peered":"2024-04-21T08:45:24.904288+0000","last_clean":"2024-04-21T08:45:24.904288+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:45:24.904288+0000","last_undegraded":"2024-04-21T08:45:24.904288+0000","last_fullsized":"2024-04-21T08:45:24.904288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903726+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:45:24.903726+0000","last_peered":"2024-04-21T08:45:24.903726+0000","last_clean":"2024-04-21T08:45:24.903726+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:45:24.903726+0000","last_undegraded":"2024-04-21T08:45:24.903726+0000","last_fullsized":"2024-04-21T08:45:24.903726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903977+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:45:24.903977+0000","last_peered":"2024-04-21T08:45:24.903977+0000","last_clean":"2024-04-21T08:45:24.903977+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:45:24.903977+0000","last_undegraded":"2024-04-21T08:45:24.903977+0000","last_fullsized":"2024-04-21T08:45:24.903977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904319+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:45:24.904319+0000","last_peered":"2024-04-21T08:45:24.904319+0000","last_clean":"2024-04-21T08:45:24.904319+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:45:24.904319+0000","last_undegraded":"2024-04-21T08:45:24.904319+0000","last_fullsized":"2024-04-21T08:45:24.904319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993372+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:45:24.993372+0000","last_peered":"2024-04-21T08:45:24.993372+0000","last_clean":"2024-04-21T08:45:24.993372+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:45:24.993372+0000","last_undegraded":"2024-04-21T08:45:24.993372+0000","last_fullsized":"2024-04-21T08:45:24.993372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992918+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:45:24.992918+0000","last_peered":"2024-04-21T08:45:24.992918+0000","last_clean":"2024-04-21T08:45:24.992918+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:45:24.992918+0000","last_undegraded":"2024-04-21T08:45:24.992918+0000","last_fullsized":"2024-04-21T08:45:24.992918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992146+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:45:24.992146+0000","last_peered":"2024-04-21T08:45:24.992146+0000","last_clean":"2024-04-21T08:45:24.992146+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:45:24.992146+0000","last_undegraded":"2024-04-21T08:45:24.992146+0000","last_fullsized":"2024-04-21T08:45:24.992146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904004+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:45:24.904004+0000","last_peered":"2024-04-21T08:45:24.904004+0000","last_clean":"2024-04-21T08:45:24.904004+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:45:24.904004+0000","last_undegraded":"2024-04-21T08:45:24.904004+0000","last_fullsized":"2024-04-21T08:45:24.904004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337411+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:45:25.337411+0000","last_peered":"2024-04-21T08:45:25.337411+0000","last_clean":"2024-04-21T08:45:25.337411+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:45:25.337411+0000","last_undegraded":"2024-04-21T08:45:25.337411+0000","last_fullsized":"2024-04-21T08:45:25.337411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.128236+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:45:25.128236+0000","last_peered":"2024-04-21T08:45:25.128236+0000","last_clean":"2024-04-21T08:45:25.128236+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:45:25.128236+0000","last_undegraded":"2024-04-21T08:45:25.128236+0000","last_fullsized":"2024-04-21T08:45:25.128236+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992998+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:45:24.992998+0000","last_peered":"2024-04-21T08:45:24.992998+0000","last_clean":"2024-04-21T08:45:24.992998+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:45:24.992998+0000","last_undegraded":"2024-04-21T08:45:24.992998+0000","last_fullsized":"2024-04-21T08:45:24.992998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992640+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:45:24.992640+0000","last_peered":"2024-04-21T08:45:24.992640+0000","last_clean":"2024-04-21T08:45:24.992640+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:45:24.992640+0000","last_undegraded":"2024-04-21T08:45:24.992640+0000","last_fullsized":"2024-04-21T08:45:24.992640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992686+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:45:24.992686+0000","last_peered":"2024-04-21T08:45:24.992686+0000","last_clean":"2024-04-21T08:45:24.992686+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:45:24.992686+0000","last_undegraded":"2024-04-21T08:45:24.992686+0000","last_fullsized":"2024-04-21T08:45:24.992686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993176+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:45:24.993176+0000","last_peered":"2024-04-21T08:45:24.993176+0000","last_clean":"2024-04-21T08:45:24.993176+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:45:24.993176+0000","last_undegraded":"2024-04-21T08:45:24.993176+0000","last_fullsized":"2024-04-21T08:45:24.993176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992191+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:45:24.992191+0000","last_peered":"2024-04-21T08:45:24.992191+0000","last_clean":"2024-04-21T08:45:24.992191+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:45:24.992191+0000","last_undegraded":"2024-04-21T08:45:24.992191+0000","last_fullsized":"2024-04-21T08:45:24.992191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903722+0000","last_change":"2024-04-21T08:44:47.315087+0000","last_active":"2024-04-21T08:45:24.903722+0000","last_peered":"2024-04-21T08:45:24.903722+0000","last_clean":"2024-04-21T08:45:24.903722+0000","last_became_active":"2024-04-21T08:44:47.314647+0000","last_became_peered":"2024-04-21T08:44:47.314647+0000","last_unstale":"2024-04-21T08:45:24.903722+0000","last_undegraded":"2024-04-21T08:45:24.903722+0000","last_fullsized":"2024-04-21T08:45:24.903722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:20.425332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903546+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:45:24.903546+0000","last_peered":"2024-04-21T08:45:24.903546+0000","last_clean":"2024-04-21T08:45:24.903546+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:45:24.903546+0000","last_undegraded":"2024-04-21T08:45:24.903546+0000","last_fullsized":"2024-04-21T08:45:24.903546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337673+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:45:25.337673+0000","last_peered":"2024-04-21T08:45:25.337673+0000","last_clean":"2024-04-21T08:45:25.337673+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:45:25.337673+0000","last_undegraded":"2024-04-21T08:45:25.337673+0000","last_fullsized":"2024-04-21T08:45:25.337673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992553+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:45:24.992553+0000","last_peered":"2024-04-21T08:45:24.992553+0000","last_clean":"2024-04-21T08:45:24.992553+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:45:24.992553+0000","last_undegraded":"2024-04-21T08:45:24.992553+0000","last_fullsized":"2024-04-21T08:45:24.992553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904194+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:45:24.904194+0000","last_peered":"2024-04-21T08:45:24.904194+0000","last_clean":"2024-04-21T08:45:24.904194+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:45:24.904194+0000","last_undegraded":"2024-04-21T08:45:24.904194+0000","last_fullsized":"2024-04-21T08:45:24.904194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904716+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:45:24.904716+0000","last_peered":"2024-04-21T08:45:24.904716+0000","last_clean":"2024-04-21T08:45:24.904716+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:45:24.904716+0000","last_undegraded":"2024-04-21T08:45:24.904716+0000","last_fullsized":"2024-04-21T08:45:24.904716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992795+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:45:24.992795+0000","last_peered":"2024-04-21T08:45:24.992795+0000","last_clean":"2024-04-21T08:45:24.992795+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:45:24.992795+0000","last_undegraded":"2024-04-21T08:45:24.992795+0000","last_fullsized":"2024-04-21T08:45:24.992795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903992+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:45:24.903992+0000","last_peered":"2024-04-21T08:45:24.903992+0000","last_clean":"2024-04-21T08:45:24.903992+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:45:24.903992+0000","last_undegraded":"2024-04-21T08:45:24.903992+0000","last_fullsized":"2024-04-21T08:45:24.903992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992360+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:45:24.992360+0000","last_peered":"2024-04-21T08:45:24.992360+0000","last_clean":"2024-04-21T08:45:24.992360+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:45:24.992360+0000","last_undegraded":"2024-04-21T08:45:24.992360+0000","last_fullsized":"2024-04-21T08:45:24.992360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992585+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:45:24.992585+0000","last_peered":"2024-04-21T08:45:24.992585+0000","last_clean":"2024-04-21T08:45:24.992585+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:45:24.992585+0000","last_undegraded":"2024-04-21T08:45:24.992585+0000","last_fullsized":"2024-04-21T08:45:24.992585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993390+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:45:24.993390+0000","last_peered":"2024-04-21T08:45:24.993390+0000","last_clean":"2024-04-21T08:45:24.993390+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:45:24.993390+0000","last_undegraded":"2024-04-21T08:45:24.993390+0000","last_fullsized":"2024-04-21T08:45:24.993390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":110016,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":595682,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":354,"num_read_kb":281,"num_write":200,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":1787046,"data_compressed":80790,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528781,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645360,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1169281,"data_compressed":64942,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":926641,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":922567,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1123793,"data_compressed":66315,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495686,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":524651,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":514896,"data_compressed":26642,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":1049571,"data_compressed":54945,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":528769,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1110016,"data_stored":1250897,"data_compressed":72471,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":524920,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":131782,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":201862,"data_compressed":7784,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:45:27.758 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:27 smithi129 ceph-mon[31272]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:27 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2394142876' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:27 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:27 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:27 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:27 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:27.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 -- 172.21.15.57:0/2694877198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 msgr2=0x7fb82410e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 --2- 172.21.15.57:0/2694877198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb82410e010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb818009860 tx=0x7fb81802bb70 comp rx=0 tx=0).stop 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 -- 172.21.15.57:0/2694877198 shutdown_connections 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 --2- 172.21.15.57:0/2694877198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb82410e550 0x7fb824114de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 --2- 172.21.15.57:0/2694877198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb82410e010 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 --2- 172.21.15.57:0/2694877198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb82410cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 -- 172.21.15.57:0/2694877198 >> 172.21.15.57:0/2694877198 conn(0x7fb82406ca20 msgr2=0x7fb82406ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 -- 172.21.15.57:0/2694877198 shutdown_connections 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 -- 172.21.15.57:0/2694877198 wait complete. 2024-04-21T08:45:27.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.898+0000 7fb82a262640 1 Processor -- start 2024-04-21T08:45:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 -- start start 2024-04-21T08:45:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb8241a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb8241a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb82410e550 0x7fb8241aa150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb824118480 con 0x7fb82410c9a0 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb824118300 con 0x7fb82410db90 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb82a262640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb824118600 con 0x7fb82410e550 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb8237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb8241a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb822ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb8241a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb8237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb8241a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46334/0 (socket says 172.21.15.57:46334) 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb8237fe640 1 -- 172.21.15.57:0/2060784817 learned_addr learned my addr 172.21.15.57:0/2060784817 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb823fff640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb82410e550 0x7fb8241aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb822ffd640 1 -- 172.21.15.57:0/2060784817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb82410e550 msgr2=0x7fb8241aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb822ffd640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb82410e550 0x7fb8241aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb822ffd640 1 -- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 msgr2=0x7fb8241a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:27.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb822ffd640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb8241a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb822ffd640 1 -- 172.21.15.57:0/2060784817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8241aa7c0 con 0x7fb82410db90 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.899+0000 7fb8237fe640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb8241a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.900+0000 7fb822ffd640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb8241a30e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb81802c0b0 tx=0x7fb8180042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.900+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818041020 con 0x7fb82410db90 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.900+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8180035f0 con 0x7fb82410db90 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.900+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8241aaa50 con 0x7fb82410db90 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.900+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818039040 con 0x7fb82410db90 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.900+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8241aaf40 con 0x7fb82410db90 2024-04-21T08:45:27.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.901+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb824113d60 con 0x7fb82410db90 2024-04-21T08:45:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.902+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fb818044020 con 0x7fb82410db90 2024-04-21T08:45:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.902+0000 7fb820ff9640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb80c0a3d20 0x7fb80c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.903+0000 7fb8237fe640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb80c0a3d20 0x7fb80c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.903+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fb8180f26f0 con 0x7fb82410db90 2024-04-21T08:45:27.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.903+0000 7fb8237fe640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb80c0a3d20 0x7fb80c0a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8140096c0 tx=0x7fb814009260 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:27.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:27.906+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb81804b020 con 0x7fb82410db90 2024-04-21T08:45:28.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.097+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb82402d810 con 0x7fb82410db90 2024-04-21T08:45:28.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.099+0000 7fb820ff9640 1 -- 172.21.15.57:0/2060784817 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25074 (secure 0 0 0) 0x7fb8180beb40 con 0x7fb82410db90 2024-04-21T08:45:28.099 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:28.099 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:45:24.897611+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb80c0a3d20 msgr2=0x7fb80c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb80c0a3d20 0x7fb80c0a61e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb8140096c0 tx=0x7fb814009260 comp rx=0 tx=0).stop 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 msgr2=0x7fb8241a30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb8241a30e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb81802c0b0 tx=0x7fb8180042f0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 shutdown_connections 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb80c0a3d20 0x7fb80c0a61e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb82410e550 0x7fb8241aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb82410db90 0x7fb8241a30e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 --2- 172.21.15.57:0/2060784817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb82410c9a0 0x7fb8241a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 >> 172.21.15.57:0/2060784817 conn(0x7fb82406ca20 msgr2=0x7fb824110f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 shutdown_connections 2024-04-21T08:45:28.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.101+0000 7fb82a262640 1 -- 172.21.15.57:0/2060784817 wait complete. 2024-04-21T08:45:28.112 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:28.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 -- 172.21.15.57:0/2841651840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 msgr2=0x7f07941012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 --2- 172.21.15.57:0/2841651840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f07941012b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0784009900 tx=0x7f078402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 -- 172.21.15.57:0/2841651840 shutdown_connections 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 --2- 172.21.15.57:0/2841651840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07941029a0 0x7f07941092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 --2- 172.21.15.57:0/2841651840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0794101fe0 0x7f0794102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 --2- 172.21.15.57:0/2841651840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f07941012b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 -- 172.21.15.57:0/2841651840 >> 172.21.15.57:0/2841651840 conn(0x7f07940fc400 msgr2=0x7f07940fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 -- 172.21.15.57:0/2841651840 shutdown_connections 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.224+0000 7f0799a8c640 1 -- 172.21.15.57:0/2841651840 wait complete. 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 Processor -- start 2024-04-21T08:45:28.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 -- start start 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f0794069c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0794101fe0 0x7f079419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07941029a0 0x7f079419f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f079410c9e0 con 0x7f0794100e90 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f079410c860 con 0x7f0794101fe0 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0799a8c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f079410cb60 con 0x7f07941029a0 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f07937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f0794069c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f07937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f0794069c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46346/0 (socket says 172.21.15.57:46346) 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f07937fe640 1 -- 172.21.15.57:0/3426690436 learned_addr learned my addr 172.21.15.57:0/3426690436 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f0793fff640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07941029a0 0x7f079419f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.225+0000 7f078bfff640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0794101fe0 0x7f079419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07937fe640 1 -- 172.21.15.57:0/3426690436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07941029a0 msgr2=0x7f079419f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07937fe640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07941029a0 0x7f079419f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07937fe640 1 -- 172.21.15.57:0/3426690436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0794101fe0 msgr2=0x7f079419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07937fe640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0794101fe0 0x7f079419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07937fe640 1 -- 172.21.15.57:0/3426690436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079419fe40 con 0x7f0794100e90 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07937fe640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f0794069c70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0784006010 tx=0x7f0784005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784043020 con 0x7f0794100e90 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07840043d0 con 0x7f0794100e90 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784003b80 con 0x7f0794100e90 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07941a00d0 con 0x7f0794100e90 2024-04-21T08:45:28.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.226+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07941a0670 con 0x7f0794100e90 2024-04-21T08:45:28.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.227+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0794108210 con 0x7f0794100e90 2024-04-21T08:45:28.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.228+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f0784046020 con 0x7f0794100e90 2024-04-21T08:45:28.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.229+0000 7f07917fa640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f07800a3da0 0x7f07800a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.229+0000 7f078bfff640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f07800a3da0 0x7f07800a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.229+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f07840f2c50 con 0x7f0794100e90 2024-04-21T08:45:28.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.230+0000 7f078bfff640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f07800a3da0 0x7f07800a6260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f07940699e0 tx=0x7f077c005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.233 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.234+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07840f8020 con 0x7f0794100e90 2024-04-21T08:45:28.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.422+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f079402d810 con 0x7f0794100e90 2024-04-21T08:45:28.422 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.423+0000 7f07917fa640 1 -- 172.21.15.57:0/3426690436 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f078404d020 con 0x7f0794100e90 2024-04-21T08:45:28.423 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:28.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f07800a3da0 msgr2=0x7f07800a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f07800a3da0 0x7f07800a6260 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f07940699e0 tx=0x7f077c005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:28.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 msgr2=0x7f0794069c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f0794069c70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0784006010 tx=0x7f0784005e40 comp rx=0 tx=0).stop 2024-04-21T08:45:28.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 shutdown_connections 2024-04-21T08:45:28.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f07800a3da0 0x7f07800a6260 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07941029a0 0x7f079419f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0794101fe0 0x7f079419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 --2- 172.21.15.57:0/3426690436 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0794100e90 0x7f0794069c70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.425+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 >> 172.21.15.57:0/3426690436 conn(0x7f07940fc400 msgr2=0x7f07941053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.426+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 shutdown_connections 2024-04-21T08:45:28.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.426+0000 7f0799a8c640 1 -- 172.21.15.57:0/3426690436 wait complete. 2024-04-21T08:45:28.436 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='client.25082 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2060784817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.437 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.438 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.438 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.438 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.438 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.438 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:28 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2988812742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8100e90 msgr2=0x7f1cc81012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2988812742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8100e90 0x7f1cc81012b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009900 tx=0x7f1cb802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2988812742 shutdown_connections 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2988812742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc81029a0 0x7f1cc81092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2988812742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2988812742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8100e90 0x7f1cc81012b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2988812742 >> 172.21.15.57:0/2988812742 conn(0x7f1cc80fc400 msgr2=0x7f1cc80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2988812742 shutdown_connections 2024-04-21T08:45:28.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2988812742 wait complete. 2024-04-21T08:45:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.564+0000 7f1ccdf50640 1 Processor -- start 2024-04-21T08:45:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 -- start start 2024-04-21T08:45:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cc8100e90 0x7f1cc8197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc81029a0 0x7f1cc819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc810c9c0 con 0x7f1cc8101fe0 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc810c840 con 0x7f1cc8100e90 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1ccdf50640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc810cb40 con 0x7f1cc81029a0 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46356/0 (socket says 172.21.15.57:46356) 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 -- 172.21.15.57:0/2933245320 learned_addr learned my addr 172.21.15.57:0/2933245320 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cc77fe640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cc8100e90 0x7f1cc8197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cc7fff640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc81029a0 0x7f1cc819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 -- 172.21.15.57:0/2933245320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc81029a0 msgr2=0x7f1cc819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc81029a0 0x7f1cc819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 -- 172.21.15.57:0/2933245320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cc8100e90 msgr2=0x7f1cc8197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cc8100e90 0x7f1cc8197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.565+0000 7f1cbffff640 1 -- 172.21.15.57:0/2933245320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc819ed20 con 0x7f1cc8101fe0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.566+0000 7f1cbffff640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8197640 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009970 tx=0x7f1cb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.566+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000d030 con 0x7f1cc8101fe0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.566+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb0007d10 con 0x7f1cc8101fe0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.566+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000cbe0 con 0x7f1cc8101fe0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.566+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc8069140 con 0x7f1cc8101fe0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.566+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc80696e0 con 0x7f1cc8101fe0 2024-04-21T08:45:28.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.567+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc8107d80 con 0x7f1cc8101fe0 2024-04-21T08:45:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.568+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f1cb0007860 con 0x7f1cc8101fe0 2024-04-21T08:45:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.569+0000 7f1cc57fa640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1cb40a3da0 0x7f1cb40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.569+0000 7f1cc77fe640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1cb40a3da0 0x7f1cb40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.569+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1cb00d2300 con 0x7f1cc8101fe0 2024-04-21T08:45:28.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.570+0000 7f1cc77fe640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1cb40a3da0 0x7f1cb40a6260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009230 tx=0x7f1cb8005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.574+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cb009e5e0 con 0x7f1cc8101fe0 2024-04-21T08:45:28.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:28.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:28.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:45:28.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='client.25082 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:28.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2060784817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:28.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.640 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.641 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:28 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:28.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:28.724 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='client.25082 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2060784817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.725 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.726 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.726 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.726 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:28 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.807+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1cc81081a0 con 0x7f1cc8101fe0 2024-04-21T08:45:28.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.808+0000 7f1cc57fa640 1 -- 172.21.15.57:0/2933245320 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f1cb009dee0 con 0x7f1cc8101fe0 2024-04-21T08:45:28.807 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:28.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1cb40a3da0 msgr2=0x7f1cb40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1cb40a3da0 0x7f1cb40a6260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009230 tx=0x7f1cb8005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 msgr2=0x7f1cc8197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8197640 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009970 tx=0x7f1cb0009e30 comp rx=0 tx=0).stop 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 shutdown_connections 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1cb40a3da0 0x7f1cb40a6260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc81029a0 0x7f1cc819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cc8101fe0 0x7f1cc8197640 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 --2- 172.21.15.57:0/2933245320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1cc8100e90 0x7f1cc8197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 >> 172.21.15.57:0/2933245320 conn(0x7f1cc80fc400 msgr2=0x7f1cc81054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 shutdown_connections 2024-04-21T08:45:28.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.810+0000 7f1ccdf50640 1 -- 172.21.15.57:0/2933245320 wait complete. 2024-04-21T08:45:28.823 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 -- 172.21.15.57:0/1703176492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 msgr2=0x7fc618100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 --2- 172.21.15.57:0/1703176492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618100670 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc614009930 tx=0x7fc61402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 -- 172.21.15.57:0/1703176492 shutdown_connections 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 --2- 172.21.15.57:0/1703176492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618103be0 0x7fc61810a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 --2- 172.21.15.57:0/1703176492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc618103220 0x7fc6181036a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 --2- 172.21.15.57:0/1703176492 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618100670 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 -- 172.21.15.57:0/1703176492 >> 172.21.15.57:0/1703176492 conn(0x7fc6180fbb80 msgr2=0x7fc6180fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 -- 172.21.15.57:0/1703176492 shutdown_connections 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.932+0000 7fc61f74b640 1 -- 172.21.15.57:0/1703176492 wait complete. 2024-04-21T08:45:28.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 Processor -- start 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 -- start start 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc618103220 0x7fc6181987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618103be0 0x7fc61819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810db40 con 0x7fc618100250 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810d9c0 con 0x7fc618103be0 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61f74b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc61810dcc0 con 0x7fc618103220 2024-04-21T08:45:28.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46386/0 (socket says 172.21.15.57:46386) 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 -- 172.21.15.57:0/3706235654 learned_addr learned my addr 172.21.15.57:0/3706235654 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61dce8640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618103be0 0x7fc61819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61cce6640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc618103220 0x7fc6181987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 -- 172.21.15.57:0/3706235654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc618103220 msgr2=0x7fc6181987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc618103220 0x7fc6181987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 -- 172.21.15.57:0/3706235654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618103be0 msgr2=0x7fc61819f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.933+0000 7fc61d4e7640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618103be0 0x7fc61819f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc61d4e7640 1 -- 172.21.15.57:0/3706235654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61819fea0 con 0x7fc618100250 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc61d4e7640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618198280 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc614000c00 tx=0x7fc6140040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614043020 con 0x7fc618100250 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6140044c0 con 0x7fc618100250 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc614003c60 con 0x7fc618100250 2024-04-21T08:45:28.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6181a0130 con 0x7fc618100250 2024-04-21T08:45:28.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.934+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181a0680 con 0x7fc618100250 2024-04-21T08:45:28.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.935+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc618109490 con 0x7fc618100250 2024-04-21T08:45:28.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.936+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fc614046020 con 0x7fc618100250 2024-04-21T08:45:28.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.937+0000 7fc60e7fc640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc6040a3d20 0x7fc6040a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:28.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.937+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fc6140f2bd0 con 0x7fc618100250 2024-04-21T08:45:28.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.937+0000 7fc61cce6640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc6040a3d20 0x7fc6040a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:28.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.938+0000 7fc61cce6640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc6040a3d20 0x7fc6040a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc618199790 tx=0x7fc608005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:28.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:28.942+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc61404d020 con 0x7fc618100250 2024-04-21T08:45:29.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.148+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc618062b50 con 0x7fc618100250 2024-04-21T08:45:29.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.148+0000 7fc60e7fc640 1 -- 172.21.15.57:0/3706235654 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fc6140bf140 con 0x7fc618100250 2024-04-21T08:45:29.148 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:29.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.150+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc6040a3d20 msgr2=0x7fc6040a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.150+0000 7fc61f74b640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc6040a3d20 0x7fc6040a61e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc618199790 tx=0x7fc608005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:29.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.150+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 msgr2=0x7fc618198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.150+0000 7fc61f74b640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618198280 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc614000c00 tx=0x7fc6140040c0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 shutdown_connections 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc6040a3d20 0x7fc6040a61e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc618103be0 0x7fc61819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc618103220 0x7fc6181987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 --2- 172.21.15.57:0/3706235654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc618100250 0x7fc618198280 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 >> 172.21.15.57:0/3706235654 conn(0x7fc6180fbb80 msgr2=0x7fc618106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 shutdown_connections 2024-04-21T08:45:29.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.151+0000 7fc61f74b640 1 -- 172.21.15.57:0/3706235654 wait complete. 2024-04-21T08:45:29.161 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:29.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 -- 172.21.15.57:0/4131685053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 msgr2=0x7f02e41012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 --2- 172.21.15.57:0/4131685053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e41012b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f02e0009900 tx=0x7f02e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 -- 172.21.15.57:0/4131685053 shutdown_connections 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 --2- 172.21.15.57:0/4131685053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02e41029a0 0x7f02e41092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 --2- 172.21.15.57:0/4131685053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02e4101fe0 0x7f02e4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 --2- 172.21.15.57:0/4131685053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e41012b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 -- 172.21.15.57:0/4131685053 >> 172.21.15.57:0/4131685053 conn(0x7f02e40fc400 msgr2=0x7f02e40fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 -- 172.21.15.57:0/4131685053 shutdown_connections 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 -- 172.21.15.57:0/4131685053 wait complete. 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 Processor -- start 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.269+0000 7f02eb99d640 1 -- start start 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02eb99d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e4069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02eb99d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02e4101fe0 0x7f02e419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02eb99d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02e41029a0 0x7f02e419f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02eb99d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410c9e0 con 0x7f02e4100e90 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02eb99d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410c860 con 0x7f02e4101fe0 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02eb99d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f02e410cb60 con 0x7f02e41029a0 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e4069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e4069cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46396/0 (socket says 172.21.15.57:46396) 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 -- 172.21.15.57:0/3896805164 learned_addr learned my addr 172.21.15.57:0/3896805164 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9f3a640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02e41029a0 0x7f02e419f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e8f38640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02e4101fe0 0x7f02e419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 -- 172.21.15.57:0/3896805164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02e41029a0 msgr2=0x7f02e419f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02e41029a0 0x7f02e419f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 -- 172.21.15.57:0/3896805164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02e4101fe0 msgr2=0x7f02e419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02e4101fe0 0x7f02e419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 -- 172.21.15.57:0/3896805164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e419fe30 con 0x7f02e4100e90 2024-04-21T08:45:29.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02e9739640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e4069cb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f02e0006010 tx=0x7f02e0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.270+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0043020 con 0x7f02e4100e90 2024-04-21T08:45:29.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.271+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02e00043d0 con 0x7f02e4100e90 2024-04-21T08:45:29.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.271+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e41a00c0 con 0x7f02e4100e90 2024-04-21T08:45:29.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.271+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0003b80 con 0x7f02e4100e90 2024-04-21T08:45:29.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.271+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e41a0610 con 0x7f02e4100e90 2024-04-21T08:45:29.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.272+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e4108210 con 0x7f02e4100e90 2024-04-21T08:45:29.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.273+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f02e0046020 con 0x7f02e4100e90 2024-04-21T08:45:29.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.274+0000 7f02da7fc640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02d00a3d20 0x7f02d00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.274+0000 7f02e8f38640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02d00a3d20 0x7f02d00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.274+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f02e00f32d0 con 0x7f02e4100e90 2024-04-21T08:45:29.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.275+0000 7f02e8f38640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02d00a3d20 0x7f02d00a61e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f02e4069a20 tx=0x7f02d4005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.278+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02e004d020 con 0x7f02e4100e90 2024-04-21T08:45:29.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.524+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f02e4062b50 con 0x7f02e4100e90 2024-04-21T08:45:29.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.525+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f02e00f0940 con 0x7f02e4100e90 2024-04-21T08:45:29.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.526+0000 7f02da7fc640 1 -- 172.21.15.57:0/3896805164 <== mon.0 v2:172.21.15.57:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f02e00f81e0 con 0x7f02e4100e90 2024-04-21T08:45:29.525 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:29.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02d00a3d20 msgr2=0x7f02d00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02d00a3d20 0x7f02d00a61e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f02e4069a20 tx=0x7f02d4005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 msgr2=0x7f02e4069cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e4069cb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f02e0006010 tx=0x7f02e0005e40 comp rx=0 tx=0).stop 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 shutdown_connections 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02d00a3d20 0x7f02d00a61e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02e41029a0 0x7f02e419f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02e4101fe0 0x7f02e419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 --2- 172.21.15.57:0/3896805164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02e4100e90 0x7f02e4069cb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 >> 172.21.15.57:0/3896805164 conn(0x7f02e40fc400 msgr2=0x7f02e41053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 shutdown_connections 2024-04-21T08:45:29.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.528+0000 7f02eb99d640 1 -- 172.21.15.57:0/3896805164 wait complete. 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: [21/Apr/2024:08:45:27] ENGINE Bus STARTING 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: [21/Apr/2024:08:45:27] ENGINE Serving on https://172.21.15.57:7150 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: [21/Apr/2024:08:45:27] ENGINE Serving on http://172.21.15.57:8765 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: [21/Apr/2024:08:45:27] ENGINE Bus STARTED 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: Unable to set osd_memory_target on smithi136 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: Unable to set osd_memory_target on smithi129 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3426690436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:29.542 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2933245320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.543 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:29 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3706235654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: [21/Apr/2024:08:45:27] ENGINE Bus STARTING 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: [21/Apr/2024:08:45:27] ENGINE Serving on https://172.21.15.57:7150 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: [21/Apr/2024:08:45:27] ENGINE Serving on http://172.21.15.57:8765 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: [21/Apr/2024:08:45:27] ENGINE Bus STARTED 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: Unable to set osd_memory_target on smithi136 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: Unable to set osd_memory_target on smithi129 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-21T08:45:29.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3426690436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2933245320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:29 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3706235654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.649+0000 7f428825c640 1 -- 172.21.15.57:0/540491784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280107260 msgr2=0x7f428010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.649+0000 7f428825c640 1 --2- 172.21.15.57:0/540491784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280107260 0x7f428010ee30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4278009ca0 tx=0x7f427802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- 172.21.15.57:0/540491784 shutdown_connections 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 --2- 172.21.15.57:0/540491784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280107260 0x7f428010ee30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 --2- 172.21.15.57:0/540491784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4280102b40 0x7f4280106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 --2- 172.21.15.57:0/540491784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4280102190 0x7f4280102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- 172.21.15.57:0/540491784 >> 172.21.15.57:0/540491784 conn(0x7f42800fd9f0 msgr2=0x7f42800ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- 172.21.15.57:0/540491784 shutdown_connections 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- 172.21.15.57:0/540491784 wait complete. 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 Processor -- start 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- start start 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4280102190 0x7f4280074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4280107260 0x7f4280074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 0x7f42801a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f42801121d0 con 0x7f4280074cd0 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.650+0000 7f428825c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4280112050 con 0x7f4280102190 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f428825c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4280112350 con 0x7f4280107260 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 0x7f42801a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 0x7f42801a68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46418/0 (socket says 172.21.15.57:46418) 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 -- 172.21.15.57:0/2178866533 learned_addr learned my addr 172.21.15.57:0/2178866533 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f4285ff8640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4280102190 0x7f4280074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42857f7640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4280107260 0x7f4280074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 -- 172.21.15.57:0/2178866533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4280107260 msgr2=0x7f4280074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4280107260 0x7f4280074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 -- 172.21.15.57:0/2178866533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4280102190 msgr2=0x7f4280074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4280102190 0x7f4280074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 -- 172.21.15.57:0/2178866533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42801a6fc0 con 0x7f4280074cd0 2024-04-21T08:45:29.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.651+0000 7f42867f9640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 0x7f42801a68c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4278005e30 tx=0x7f42780346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.655+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278041020 con 0x7f4280074cd0 2024-04-21T08:45:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.655+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42801a7250 con 0x7f4280074cd0 2024-04-21T08:45:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.655+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42801a7790 con 0x7f4280074cd0 2024-04-21T08:45:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.655+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4278032970 con 0x7f4280074cd0 2024-04-21T08:45:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.655+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278039040 con 0x7f4280074cd0 2024-04-21T08:45:29.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.656+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4264005290 con 0x7f4280074cd0 2024-04-21T08:45:29.656 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.657+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f4278044020 con 0x7f4280074cd0 2024-04-21T08:45:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.658+0000 7f4276ffd640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f426c0a3de0 0x7f426c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.658+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f42780f2180 con 0x7f4280074cd0 2024-04-21T08:45:29.657 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.658+0000 7f4285ff8640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f426c0a3de0 0x7f426c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:29.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.658+0000 7f4285ff8640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f426c0a3de0 0x7f426c0a62a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4270009bd0 tx=0x7f4270005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:29.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.662+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f427804b030 con 0x7f4280074cd0 2024-04-21T08:45:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: [21/Apr/2024:08:45:27] ENGINE Bus STARTING 2024-04-21T08:45:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: [21/Apr/2024:08:45:27] ENGINE Serving on https://172.21.15.57:7150 2024-04-21T08:45:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: [21/Apr/2024:08:45:27] ENGINE Serving on http://172.21.15.57:8765 2024-04-21T08:45:29.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: [21/Apr/2024:08:45:27] ENGINE Bus STARTED 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: Unable to set osd_memory_target on smithi136 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: Unable to set osd_memory_target on smithi129 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3426690436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2933245320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:29 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3706235654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:29.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.911+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4264005850 con 0x7f4280074cd0 2024-04-21T08:45:29.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.911+0000 7f4276ffd640 1 -- 172.21.15.57:0/2178866533 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f42780be460 con 0x7f4280074cd0 2024-04-21T08:45:29.911 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:29.911 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:29.911 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:29.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f426c0a3de0 msgr2=0x7f426c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f426c0a3de0 0x7f426c0a62a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4270009bd0 tx=0x7f4270005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 msgr2=0x7f42801a68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 0x7f42801a68c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4278005e30 tx=0x7f42780346d0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 shutdown_connections 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f426c0a3de0 0x7f426c0a62a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4280074cd0 0x7f42801a68c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4280107260 0x7f4280074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 --2- 172.21.15.57:0/2178866533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4280102190 0x7f4280074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 >> 172.21.15.57:0/2178866533 conn(0x7f42800fd9f0 msgr2=0x7f42800ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 shutdown_connections 2024-04-21T08:45:29.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:29.914+0000 7f428825c640 1 -- 172.21.15.57:0/2178866533 wait complete. 2024-04-21T08:45:29.925 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:30.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 -- 172.21.15.57:0/3379244941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c069620 msgr2=0x7f826c069aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 --2- 172.21.15.57:0/3379244941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c069620 0x7f826c069aa0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f825c009900 tx=0x7f825c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 -- 172.21.15.57:0/3379244941 shutdown_connections 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 --2- 172.21.15.57:0/3379244941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c10e8a0 0x7f826c110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 --2- 172.21.15.57:0/3379244941 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c069620 0x7f826c069aa0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 --2- 172.21.15.57:0/3379244941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f826c102ef0 0x7f826c0690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 -- 172.21.15.57:0/3379244941 >> 172.21.15.57:0/3379244941 conn(0x7f826c077e30 msgr2=0x7f826c078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 -- 172.21.15.57:0/3379244941 shutdown_connections 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.035+0000 7f82739b6640 1 -- 172.21.15.57:0/3379244941 wait complete. 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 Processor -- start 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 -- start start 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f826c069620 0x7f826c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 0x7f826c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c10e8a0 0x7f826c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f826c1143a0 con 0x7f826c102ef0 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f826c114220 con 0x7f826c10e8a0 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f82739b6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f826c114520 con 0x7f826c069620 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f8270f51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 0x7f826c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f8270f51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 0x7f826c1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46434/0 (socket says 172.21.15.57:46434) 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f8270f51640 1 -- 172.21.15.57:0/337835812 learned_addr learned my addr 172.21.15.57:0/337835812 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f8271f53640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c10e8a0 0x7f826c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.036+0000 7f8271752640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f826c069620 0x7f826c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f8270f51640 1 -- 172.21.15.57:0/337835812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f826c069620 msgr2=0x7f826c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f8270f51640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f826c069620 0x7f826c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f8270f51640 1 -- 172.21.15.57:0/337835812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c10e8a0 msgr2=0x7f826c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f8270f51640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c10e8a0 0x7f826c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f8270f51640 1 -- 172.21.15.57:0/337835812 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826c19afe0 con 0x7f826c102ef0 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f8270f51640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 0x7f826c1a0ba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f825c005940 tx=0x7f825c02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c043020 con 0x7f826c102ef0 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826c19b270 con 0x7f826c102ef0 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f825c02ca70 con 0x7f826c102ef0 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c0039f0 con 0x7f826c102ef0 2024-04-21T08:45:30.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.037+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826c1a75e0 con 0x7f826c102ef0 2024-04-21T08:45:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.038+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f826c101440 con 0x7f826c102ef0 2024-04-21T08:45:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.039+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f825c046020 con 0x7f826c102ef0 2024-04-21T08:45:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.040+0000 7f82627fc640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82580a3db0 0x7f82580a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.040+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f825c0f2c10 con 0x7f826c102ef0 2024-04-21T08:45:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.040+0000 7f8271752640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82580a3db0 0x7f82580a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.040+0000 7f8271752640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82580a3db0 0x7f82580a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82680095f0 tx=0x7f8268005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.044+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f825c04d020 con 0x7f826c102ef0 2024-04-21T08:45:30.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.232+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f826c19bc80 con 0x7f826c102ef0 2024-04-21T08:45:30.233 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.233+0000 7f82627fc640 1 -- 172.21.15.57:0/337835812 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25074 (secure 0 0 0) 0x7f826c19bc80 con 0x7f826c102ef0 2024-04-21T08:45:30.233 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:30.233 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:45:24.897611+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82580a3db0 msgr2=0x7f82580a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82580a3db0 0x7f82580a6270 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82680095f0 tx=0x7f8268005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 msgr2=0x7f826c1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 0x7f826c1a0ba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f825c005940 tx=0x7f825c02c5f0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 shutdown_connections 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82580a3db0 0x7f82580a6270 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f826c10e8a0 0x7f826c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f826c102ef0 0x7f826c1a0ba0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 --2- 172.21.15.57:0/337835812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f826c069620 0x7f826c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 >> 172.21.15.57:0/337835812 conn(0x7f826c077e30 msgr2=0x7f826c100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.235+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 shutdown_connections 2024-04-21T08:45:30.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.236+0000 7f82739b6640 1 -- 172.21.15.57:0/337835812 wait complete. 2024-04-21T08:45:30.246 DEBUG:tasks.fwd_scrub:Yielding 2024-04-21T08:45:30.247 INFO:teuthology.run_tasks:Running task exec... 2024-04-21T08:45:30.258 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:45:30.258 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:45:30.258 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-21T08:45:30.258 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-21T08:45:30.259 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:45:30.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:45:30.259 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:45:30.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 -- 172.21.15.57:0/122233987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 msgr2=0x7f1460102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 --2- 172.21.15.57:0/122233987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f1460102570 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f145c009860 tx=0x7f145c02bb70 comp rx=0 tx=0).stop 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 -- 172.21.15.57:0/122233987 shutdown_connections 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 --2- 172.21.15.57:0/122233987 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 --2- 172.21.15.57:0/122233987 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1460102b40 0x7f1460106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 --2- 172.21.15.57:0/122233987 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f1460102570 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 -- 172.21.15.57:0/122233987 >> 172.21.15.57:0/122233987 conn(0x7f14600793a0 msgr2=0x7f14600797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 -- 172.21.15.57:0/122233987 shutdown_connections 2024-04-21T08:45:30.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f1467a26640 1 -- 172.21.15.57:0/122233987 wait complete. 2024-04-21T08:45:30.368 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 -- 172.21.15.57:0/1972100769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc81021a0 msgr2=0x7f5fc8102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.368 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 --2- 172.21.15.57:0/1972100769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc81021a0 0x7f5fc8102580 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009900 tx=0x7f5fb802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:30.368 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 -- 172.21.15.57:0/1972100769 shutdown_connections 2024-04-21T08:45:30.368 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 --2- 172.21.15.57:0/1972100769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc8107270 0x7f5fc810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.368 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 --2- 172.21.15.57:0/1972100769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fc8102b50 0x7f5fc8106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.368 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 --2- 172.21.15.57:0/1972100769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc81021a0 0x7f5fc8102580 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.366+0000 7f5fcd759640 1 -- 172.21.15.57:0/1972100769 >> 172.21.15.57:0/1972100769 conn(0x7f5fc80fda00 msgr2=0x7f5fc80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f5fcd759640 1 -- 172.21.15.57:0/1972100769 shutdown_connections 2024-04-21T08:45:30.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f5fcd759640 1 -- 172.21.15.57:0/1972100769 wait complete. 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 Processor -- start 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 -- start start 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f146019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14601985d0 0x7f1460198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1460112150 con 0x7f1460107260 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1460111fd0 con 0x7f1460102190 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1467a26640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f14601122d0 con 0x7f14601985d0 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146019e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46458/0 (socket says 172.21.15.57:46458) 2024-04-21T08:45:30.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 -- 172.21.15.57:0/2904024727 learned_addr learned my addr 172.21.15.57:0/2904024727 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f5fcd759640 1 Processor -- start 2024-04-21T08:45:30.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1465fc3640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14601985d0 0x7f1460198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f14657c2640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f146019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 -- 172.21.15.57:0/2904024727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14601985d0 msgr2=0x7f1460198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14601985d0 0x7f1460198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 -- 172.21.15.57:0/2904024727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 msgr2=0x7f146019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f146019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 -- 172.21.15.57:0/2904024727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14601990f0 con 0x7f1460107260 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1465fc3640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14601985d0 0x7f1460198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f14657c2640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f146019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:30.372 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f5fcd759640 1 -- start start 2024-04-21T08:45:30.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.367+0000 7f1464fc1640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146019e960 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f145000e9c0 tx=0x7f145000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f14567fc640 1 -- 172.21.15.57:0/2904024727 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145000cd80 con 0x7f1460107260 2024-04-21T08:45:30.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fcd759640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 0x7f5fc81a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fcd759640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fc8107270 0x7f5fc81a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.373 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14601993e0 con 0x7f1460107260 2024-04-21T08:45:30.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fcd759640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc819a760 0x7f5fc819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f14567fc640 1 -- 172.21.15.57:0/2904024727 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14500054d0 con 0x7f1460107260 2024-04-21T08:45:30.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14601a5680 con 0x7f1460107260 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fcd759640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8112200 con 0x7f5fc819a760 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fcd759640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8112080 con 0x7f5fc8107270 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fcd759640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc8112380 con 0x7f5fc81021a0 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 0x7f5fc81a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc819a760 0x7f5fc819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 0x7f5fc81a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47432/0 (socket says 172.21.15.57:47432) 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 -- 172.21.15.57:0/3114081405 learned_addr learned my addr 172.21.15.57:0/3114081405 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f14567fc640 1 -- 172.21.15.57:0/2904024727 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1450010430 con 0x7f1460107260 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc67fc640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fc8107270 0x7f5fc81a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 -- 172.21.15.57:0/3114081405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fc8107270 msgr2=0x7f5fc81a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fc8107270 0x7f5fc81a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 -- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc819a760 msgr2=0x7f5fc819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc819a760 0x7f5fc819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 -- 172.21.15.57:0/3114081405 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc819b360 con 0x7f5fc81021a0 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.368+0000 7f5fc6ffd640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 0x7f5fc81a06a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8005940 tx=0x7f5fb80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.369+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8043020 con 0x7f5fc81021a0 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.369+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc819b5f0 con 0x7f5fc81021a0 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.369+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fb802caf0 con 0x7f5fc81021a0 2024-04-21T08:45:30.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.369+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc81a77e0 con 0x7f5fc81021a0 2024-04-21T08:45:30.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.369+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8003af0 con 0x7f5fc81021a0 2024-04-21T08:45:30.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.370+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb4005290 con 0x7f5fc81021a0 2024-04-21T08:45:30.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f5fb8046020 con 0x7f5fc81021a0 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 Processor -- start 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f14567fc640 1 -- 172.21.15.57:0/2904024727 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f14500105d0 con 0x7f1460107260 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 -- start start 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb1708e0 0x5571fb170cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb179840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb179e10 0x5571fb17c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5571fb0d4f00 con 0x5571fb171290 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5571fb072590 con 0x5571fb179e10 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f146a60f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5571fb0d0ec0 con 0x5571fb1708e0 2024-04-21T08:45:30.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f14657c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb179840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f1465fc3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb1708e0 0x5571fb170cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f14657c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb179840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46466/0 (socket says 172.21.15.57:46466) 2024-04-21T08:45:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f14657c2640 1 -- 172.21.15.57:0/2439430628 learned_addr learned my addr 172.21.15.57:0/2439430628 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f5faffff640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5fb00a3e60 0x7f5fb00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.371+0000 7f1464fc1640 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb179e10 0x5571fb17c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1465fc3640 1 -- 172.21.15.57:0/2439430628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb179e10 msgr2=0x5571fb17c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f5fc67fc640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5fb00a3e60 0x7f5fb00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1465fc3640 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb179e10 0x5571fb17c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1465fc3640 1 -- 172.21.15.57:0/2439430628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 msgr2=0x5571fb179840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1465fc3640 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb179840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1465fc3640 1 -- 172.21.15.57:0/2439430628 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571faeaf7e0 con 0x5571fb1708e0 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f14657c2640 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb179840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1464fc1640 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb179e10 0x5571fb17c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f14567fc640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f144c0a3e30 0x7f144c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5fb80f2ca0 con 0x7f5fc81021a0 2024-04-21T08:45:30.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f1465fc3640 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb1708e0 0x5571fb170cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1458009e50 tx=0x7f145800f7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e160ca89ada94c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f14567fc640 1 -- 172.21.15.57:0/2904024727 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f14500d1090 con 0x7f1460107260 2024-04-21T08:45:30.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f14657c2640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f144c0a3e30 0x7f144c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f5fc67fc640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5fb00a3e60 0x7f5fb00a6320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fc819bfb0 tx=0x7f5fbc00a250 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.372+0000 7f14657c2640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f144c0a3e30 0x7f144c0a62f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x5571fb171690 tx=0x7f145c009490 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.376+0000 7f144b7fe640 1 -- 172.21.15.57:0/2439430628 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458004b20 con 0x5571fb1708e0 2024-04-21T08:45:30.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.376+0000 7f144b7fe640 1 -- 172.21.15.57:0/2439430628 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1458004cc0 con 0x5571fb1708e0 2024-04-21T08:45:30.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.376+0000 7f144b7fe640 1 -- 172.21.15.57:0/2439430628 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458005410 con 0x5571fb1708e0 2024-04-21T08:45:30.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.376+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fb804d020 con 0x7f5fc81021a0 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 -- 172.21.15.57:0/2439430628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb1708e0 msgr2=0x5571fb170cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb1708e0 0x5571fb170cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1458009e50 tx=0x7f145800f7d0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 -- 172.21.15.57:0/2439430628 shutdown_connections 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb179e10 0x5571fb17c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb179840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 --2- 172.21.15.57:0/2439430628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb1708e0 0x5571fb170cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 -- 172.21.15.57:0/2439430628 >> 172.21.15.57:0/2439430628 conn(0x5571fb16afb0 msgr2=0x5571fb16b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 -- 172.21.15.57:0/2439430628 shutdown_connections 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 -- 172.21.15.57:0/2439430628 wait complete. 2024-04-21T08:45:30.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 -- start start 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.377+0000 7f146a60f740 1 Processor -- start 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb176d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb1772a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb1777e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5571fb17cd80 con 0x5571fb179e10 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5571fb08e580 con 0x5571fb1708e0 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5571fb17cf00 con 0x5571fb171290 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f146a60f740 10 client.0 _ll_register_callbacks cb 0x5571fb094650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb1777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb1777e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46480/0 (socket says 172.21.15.57:46480) 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 -- 172.21.15.57:0/137219344 learned_addr learned my addr 172.21.15.57:0/137219344 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1464fc1640 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb1772a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 -- 172.21.15.57:0/137219344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 msgr2=0x5571fb1772a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb1772a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 -- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 msgr2=0x5571fb176d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb176d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 -- 172.21.15.57:0/137219344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571faf0ff70 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1465fc3640 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb1777e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f145800e000 tx=0x7f145800ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643b4d37d1a96799 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.378+0000 7f1448ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f1448ff9640 1 -- 172.21.15.57:0/137219344 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458026020 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f1448ff9640 1 -- 172.21.15.57:0/137219344 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1458005150 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1458004cc0 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f1448ff9640 1 -- 172.21.15.57:0/137219344 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1458031050 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 10 client.15579 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5571fada7170 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f1448ff9640 1 -- 172.21.15.57:0/137219344 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1458004cc0 con 0x5571fb179e10 2024-04-21T08:45:30.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f1448ff9640 1 -- 172.21.15.57:0/137219344 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f145800b040 con 0x5571fb179e10 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 10 client.15579 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 20 client.15579 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb18a260 0x5571fb18c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 4 client.15579 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.379+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571fb0e2920 con 0x5571fb18a260 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.380+0000 7f14657c2640 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb18a260 0x5571fb18c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.380+0000 7f14657c2640 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb18a260 0x5571fb18c650 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=383d373694f3963d server_cookie=7959cb7c07170a90 in_seq=0 out_seq=0 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.380+0000 7f1448ff9640 10 client.15579 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f1448ff9640 1 -- 172.21.15.57:0/137219344 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571fb0e2920 con 0x5571fb18a260 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f1448ff9640 10 client.15579 handle_command_reply: tid=0 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 client.15579 shutdown 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 msgr2=0x5571fb1777e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb1777e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f145800e000 tx=0x7f145800ddc0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 shutdown_connections 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb18a260 0x5571fb18c650 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb1777e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb1772a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 --2- 172.21.15.57:0/137219344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb176d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 >> 172.21.15.57:0/137219344 conn(0x5571fb16afb0 msgr2=0x5571fb16cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f1448ff9640 0 client.15579 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 shutdown_connections 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 1 -- 172.21.15.57:0/137219344 wait complete. 2024-04-21T08:45:30.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.381+0000 7f146a60f740 20 client.15579 trim_cache size 0 max 16384 2024-04-21T08:45:30.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 Processor -- start 2024-04-21T08:45:30.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 -- start start 2024-04-21T08:45:30.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb179490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb185ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb169e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f144400ddf0 con 0x5571fb179e10 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1444010650 con 0x5571fb1708e0 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f146a60f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f144400e440 con 0x5571fb171290 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb169e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb169e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46484/0 (socket says 172.21.15.57:46484) 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 -- 172.21.15.57:0/741469935 learned_addr learned my addr 172.21.15.57:0/741469935 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1464fc1640 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb185ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f14657c2640 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb179490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 -- 172.21.15.57:0/741469935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 msgr2=0x5571fb185ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb185ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 -- 172.21.15.57:0/741469935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 msgr2=0x5571fb179490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb179490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.399+0000 7f1465fc3640 1 -- 172.21.15.57:0/741469935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571fb1752b0 con 0x5571fb179e10 2024-04-21T08:45:30.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.400+0000 7f1465fc3640 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb169e90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1458009e30 tx=0x7f145800de30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb66104785d8ac4b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.400+0000 7f144affd640 1 -- 172.21.15.57:0/741469935 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458026070 con 0x5571fb179e10 2024-04-21T08:45:30.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.400+0000 7f144affd640 1 -- 172.21.15.57:0/741469935 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f145800fc10 con 0x5571fb179e10 2024-04-21T08:45:30.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.400+0000 7f144affd640 1 -- 172.21.15.57:0/741469935 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458022d00 con 0x5571fb179e10 2024-04-21T08:45:30.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- 172.21.15.57:0/741469935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 msgr2=0x5571fb169e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb169e90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f1458009e30 tx=0x7f145800de30 comp rx=0 tx=0).stop 2024-04-21T08:45:30.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- 172.21.15.57:0/741469935 shutdown_connections 2024-04-21T08:45:30.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb179e10 0x5571fb169e90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb171290 0x5571fb185ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- 172.21.15.57:0/741469935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb179490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- 172.21.15.57:0/741469935 >> 172.21.15.57:0/741469935 conn(0x5571fb16afb0 msgr2=0x5571fb169780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- 172.21.15.57:0/741469935 shutdown_connections 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- 172.21.15.57:0/741469935 wait complete. 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- start start 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 Processor -- start 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb173ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb173ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb179e10 0x5571fb174530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f144400eb10 con 0x5571fb171290 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f144400ff80 con 0x5571fb1708e0 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.401+0000 7f146a60f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f144400f1e0 con 0x5571fb179e10 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f146a60f740 10 client.0 _ll_register_callbacks cb 0x7f145c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:30.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb173ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb173ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46498/0 (socket says 172.21.15.57:46498) 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 -- 172.21.15.57:0/811748877 learned_addr learned my addr 172.21.15.57:0/811748877 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f14657c2640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb173ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1465fc3640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb179e10 0x5571fb174530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 -- 172.21.15.57:0/811748877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb179e10 msgr2=0x5571fb174530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb179e10 0x5571fb174530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 -- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 msgr2=0x5571fb173ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb173ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 -- 172.21.15.57:0/811748877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571fb1ef720 con 0x5571fb171290 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1464fc1640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb173ff0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1450099530 tx=0x7f145009c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5aa3ebbe153c200 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1448ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1448ff9640 1 -- 172.21.15.57:0/811748877 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145009cb60 con 0x5571fb171290 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1448ff9640 1 -- 172.21.15.57:0/811748877 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f14500046c0 con 0x5571fb171290 2024-04-21T08:45:30.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f145800fc10 con 0x5571fb171290 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f1448ff9640 1 -- 172.21.15.57:0/811748877 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f14500a0f10 con 0x5571fb171290 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f146a60f740 10 client.15585 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.402+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5571fb1f0a40 con 0x5571fb171290 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f1448ff9640 1 -- 172.21.15.57:0/811748877 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f145800fc10 con 0x5571fb171290 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f1448ff9640 1 -- 172.21.15.57:0/811748877 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f14500cebd0 con 0x5571fb171290 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f146a60f740 10 client.15585 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f146a60f740 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f146a60f740 20 client.15585 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f146a60f740 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb1f11f0 0x5571fb1f3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f146a60f740 4 client.15585 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5571fb0dc790 con 0x5571fb1f11f0 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f14657c2640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb1f11f0 0x5571fb1f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f14657c2640 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb1f11f0 0x5571fb1f3610 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8412fae3e413032 server_cookie=f4f57711eae66e9b in_seq=0 out_seq=0 2024-04-21T08:45:30.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.403+0000 7f1448ff9640 10 client.15585 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.404+0000 7f1448ff9640 1 -- 172.21.15.57:0/811748877 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5571fb0dc790 con 0x5571fb1f11f0 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.404+0000 7f1448ff9640 10 client.15585 handle_command_reply: tid=0 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.404+0000 7f146a60f740 1 client.15585 shutdown 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.404+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 msgr2=0x5571fb173ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.404+0000 7f146a60f740 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb173ff0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1450099530 tx=0x7f145009c720 comp rx=0 tx=0).stop 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 shutdown_connections 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571fb1f11f0 0x5571fb1f3610 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571fb179e10 0x5571fb174530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571fb171290 0x5571fb173ff0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 --2- 172.21.15.57:0/811748877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571fb1708e0 0x5571fb173ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 >> 172.21.15.57:0/811748877 conn(0x5571fb16afb0 msgr2=0x5571fb16a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1448ff9640 0 client.15585 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 shutdown_connections 2024-04-21T08:45:30.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 1 -- 172.21.15.57:0/811748877 wait complete. 2024-04-21T08:45:30.410 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f146a60f740 20 client.15585 trim_cache size 0 max 16384 2024-04-21T08:45:30.410 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:45:30.410 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:45:30.410 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "46517aef-40ab-4c5f-a4b7-6625836c57a1", 2024-04-21T08:45:30.410 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:45:30.410 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f144c0a3e30 msgr2=0x7f144c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f144c0a3e30 0x7f144c0a62f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x5571fb171690 tx=0x7f145c009490 comp rx=0 tx=0).stop 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 msgr2=0x7f146019e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146019e960 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f145000e9c0 tx=0x7f145000ee80 comp rx=0 tx=0).stop 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 shutdown_connections 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f144c0a3e30 0x7f144c0a62f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14601985d0 0x7f1460198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1460107260 0x7f146019e960 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 --2- 172.21.15.57:0/2904024727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1460102190 0x7f146019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 >> 172.21.15.57:0/2904024727 conn(0x7f14600793a0 msgr2=0x7f1460100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.405+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 shutdown_connections 2024-04-21T08:45:30.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.406+0000 7f1467a26640 1 -- 172.21.15.57:0/2904024727 wait complete. 2024-04-21T08:45:30.415 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:45:30.416 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:45:30.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 -- 172.21.15.57:0/3458357976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 msgr2=0x7f59a41035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 --2- 172.21.15.57:0/3458357976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a41035a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5998009900 tx=0x7f599802bd60 comp rx=0 tx=0).stop 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 -- 172.21.15.57:0/3458357976 shutdown_connections 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 --2- 172.21.15.57:0/3458357976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 0x7f59a410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 --2- 172.21.15.57:0/3458357976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a4103ae0 0x7f59a410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 --2- 172.21.15.57:0/3458357976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a41035a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 -- 172.21.15.57:0/3458357976 >> 172.21.15.57:0/3458357976 conn(0x7f59a4078000 msgr2=0x7f59a4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 -- 172.21.15.57:0/3458357976 shutdown_connections 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 -- 172.21.15.57:0/3458357976 wait complete. 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 Processor -- start 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.521+0000 7f59a8881640 1 -- start start 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a8881640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a8881640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a4103ae0 0x7f59a419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a8881640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 0x7f59a4198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a8881640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f59a4112190 con 0x7f59a41031c0 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a8881640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59a4112010 con 0x7f59a4103ae0 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a8881640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f59a4112310 con 0x7f59a410c500 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46526/0 (socket says 172.21.15.57:46526) 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 -- 172.21.15.57:0/879575899 learned_addr learned my addr 172.21.15.57:0/879575899 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a4103ae0 0x7f59a419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 0x7f59a4198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 -- 172.21.15.57:0/879575899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 msgr2=0x7f59a4198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 0x7f59a4198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 -- 172.21.15.57:0/879575899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a4103ae0 msgr2=0x7f59a419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a4103ae0 0x7f59a419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 -- 172.21.15.57:0/879575899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a4198d10 con 0x7f59a41031c0 2024-04-21T08:45:30.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 0x7f59a4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59a259c640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a419e3e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5998005940 tx=0x7f59980045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.522+0000 7f59937fe640 1 -- 172.21.15.57:0/879575899 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998043020 con 0x7f59a41031c0 2024-04-21T08:45:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.523+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a4198fa0 con 0x7f59a41031c0 2024-04-21T08:45:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.523+0000 7f59937fe640 1 -- 172.21.15.57:0/879575899 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f599802dc30 con 0x7f59a41031c0 2024-04-21T08:45:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.523+0000 7f59937fe640 1 -- 172.21.15.57:0/879575899 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998003a10 con 0x7f59a41031c0 2024-04-21T08:45:30.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.523+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a41a52a0 con 0x7f59a41031c0 2024-04-21T08:45:30.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59937fe640 1 -- 172.21.15.57:0/879575899 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f5998046020 con 0x7f59a41031c0 2024-04-21T08:45:30.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 Processor -- start 2024-04-21T08:45:30.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 -- start start 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda52b0 0x55d69eda5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 0x55d69edb2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edb5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55d69ed0a1e0 con 0x55d69eda5bd0 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d69eca7390 con 0x55d69edb2e90 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59ab47c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d69ed061a0 con 0x55d69eda52b0 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.525+0000 7f59937fe640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f598c0a3db0 0x7f598c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 0x55d69edb2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 0x55d69edb2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46532/0 (socket says 172.21.15.57:46532) 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 -- 172.21.15.57:0/3160334876 learned_addr learned my addr 172.21.15.57:0/3160334876 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda52b0 0x55d69eda5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edb5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f598c0a3db0 0x7f598c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59937fe640 1 -- 172.21.15.57:0/879575899 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f59980f2d00 con 0x7f59a41031c0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 -- 172.21.15.57:0/3160334876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda52b0 msgr2=0x55d69eda5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda52b0 0x55d69eda5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 -- 172.21.15.57:0/3160334876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 msgr2=0x55d69edb5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edb5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 -- 172.21.15.57:0/3160334876 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d69eae67d0 con 0x55d69eda5bd0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda52b0 0x55d69eda5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edb5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a259c640 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 0x55d69edb2950 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f59980bad40 tx=0x7f59980bacb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ae0515dce8209c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f598c0a3db0 0x7f598c0a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f599c00afd0 tx=0x7f599c005c10 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f5990ff9640 1 -- 172.21.15.57:0/3160334876 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980be240 con 0x55d69eda5bd0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f5990ff9640 1 -- 172.21.15.57:0/3160334876 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59980befe0 con 0x55d69eda5bd0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.526+0000 7f5990ff9640 1 -- 172.21.15.57:0/3160334876 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980f02e0 con 0x55d69eda5bd0 2024-04-21T08:45:30.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 -- 172.21.15.57:0/3160334876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 msgr2=0x55d69edb2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 0x55d69edb2950 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f59980bad40 tx=0x7f59980bacb0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 -- 172.21.15.57:0/3160334876 shutdown_connections 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edb5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda5bd0 0x55d69edb2950 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 --2- 172.21.15.57:0/3160334876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda52b0 0x55d69eda5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 -- 172.21.15.57:0/3160334876 >> 172.21.15.57:0/3160334876 conn(0x55d69ed9fcb0 msgr2=0x55d69eda0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 -- 172.21.15.57:0/3160334876 shutdown_connections 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.527+0000 7f59ab47c740 1 -- 172.21.15.57:0/3160334876 wait complete. 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 -- start start 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 Processor -- start 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edaff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edb04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edb0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55d69edb5a10 con 0x55d69eda52b0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d69ecc32f0 con 0x55d69eda5bd0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d69edb5b90 con 0x55d69edb2e90 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59ab47c740 10 client.0 _ll_register_callbacks cb 0x55d69ecd0580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edaff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edaff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46534/0 (socket says 172.21.15.57:46534) 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 -- 172.21.15.57:0/336975661 learned_addr learned my addr 172.21.15.57:0/336975661 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edb04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edb0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 -- 172.21.15.57:0/336975661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 msgr2=0x55d69edb0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edb0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 -- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 msgr2=0x55d69edb04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edb04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a259c640 1 -- 172.21.15.57:0/336975661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69ea1c3f0 con 0x55d69eda52b0 2024-04-21T08:45:30.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.528+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edb04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59a259c640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edaff80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f59980bcb90 tx=0x7f59980babf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca01cd3d509825b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f5989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f5989ffb640 1 -- 172.21.15.57:0/336975661 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980c7550 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f5989ffb640 1 -- 172.21.15.57:0/336975661 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f59980892f0 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59980befe0 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f5989ffb640 1 -- 172.21.15.57:0/336975661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f59980c2c90 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 10 client.15597 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d69e9dc0f0 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f5989ffb640 1 -- 172.21.15.57:0/336975661 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59980befe0 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f5989ffb640 1 -- 172.21.15.57:0/336975661 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f59980ba630 con 0x55d69eda52b0 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 10 client.15597 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 20 client.15597 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69edc2ae0 0x55d69edc4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 4 client.15597 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.529+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d69ed17c00 con 0x55d69edc2ae0 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.530+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69edc2ae0 0x55d69edc4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.530+0000 7f5989ffb640 10 client.15597 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.530+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69edc2ae0 0x55d69edc4ed0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=726abd5d31c3117d server_cookie=baf9d7f265bc5094 in_seq=0 out_seq=0 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f5989ffb640 1 -- 172.21.15.57:0/336975661 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d69ed17c00 con 0x55d69edc2ae0 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f5989ffb640 10 client.15597 handle_command_reply: tid=0 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 client.15597 shutdown 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 msgr2=0x55d69edaff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edaff80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f59980bcb90 tx=0x7f59980babf0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 shutdown_connections 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69edc2ae0 0x55d69edc4ed0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edb0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edb04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f5989ffb640 0 client.15597 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 --2- 172.21.15.57:0/336975661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edaff80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 >> 172.21.15.57:0/336975661 conn(0x55d69ed9fcb0 msgr2=0x55d69eda1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 shutdown_connections 2024-04-21T08:45:30.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 1 -- 172.21.15.57:0/336975661 wait complete. 2024-04-21T08:45:30.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.531+0000 7f59ab47c740 20 client.15597 trim_cache size 0 max 16384 2024-04-21T08:45:30.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 Processor -- start 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 -- start start 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edb26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edbe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edaeec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f59a41ab0a0 con 0x55d69eda52b0 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59a41ad900 con 0x55d69eda5bd0 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59ab47c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f59a41ab6f0 con 0x55d69edb2e90 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edb26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edb26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46548/0 (socket says 172.21.15.57:46548) 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a259c640 1 -- 172.21.15.57:0/2889946212 learned_addr learned my addr 172.21.15.57:0/2889946212 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edbe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a259c640 1 -- 172.21.15.57:0/2889946212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 msgr2=0x55d69edaeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edaeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.549+0000 7f59a259c640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edaeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59a259c640 1 -- 172.21.15.57:0/2889946212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 msgr2=0x55d69edbe8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59a259c640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edbe8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59a259c640 1 -- 172.21.15.57:0/2889946212 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d69edb55f0 con 0x55d69eda52b0 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edbe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edaeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59a259c640 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edb26b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f59980f06f0 tx=0x7f59980baf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb485a3c091ee8e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f598bfff640 1 -- 172.21.15.57:0/2889946212 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980bdb10 con 0x55d69eda52b0 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f598bfff640 1 -- 172.21.15.57:0/2889946212 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59980befe0 con 0x55d69eda52b0 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f598bfff640 1 -- 172.21.15.57:0/2889946212 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980ba630 con 0x55d69eda52b0 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59ab47c740 1 -- 172.21.15.57:0/2889946212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 msgr2=0x55d69edb26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edb26b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f59980f06f0 tx=0x7f59980baf50 comp rx=0 tx=0).stop 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59ab47c740 1 -- 172.21.15.57:0/2889946212 shutdown_connections 2024-04-21T08:45:30.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69edb2e90 0x55d69edaeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69eda5bd0 0x55d69edbe8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.550+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2889946212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edb26b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- 172.21.15.57:0/2889946212 >> 172.21.15.57:0/2889946212 conn(0x55d69ed9fcb0 msgr2=0x55d69ed9ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- 172.21.15.57:0/2889946212 shutdown_connections 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- 172.21.15.57:0/2889946212 wait complete. 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- start start 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 Processor -- start 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edacd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda5bd0 0x55d69edad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f59a41abdc0 con 0x55d69eda52b0 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59a41ad230 con 0x55d69edb2e90 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f59a41ac490 con 0x55d69eda5bd0 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59ab47c740 10 client.0 _ll_register_callbacks cb 0x7f59940097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:30.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edacd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59a259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edacd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46556/0 (socket says 172.21.15.57:46556) 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59a259c640 1 -- 172.21.15.57:0/2910567372 learned_addr learned my addr 172.21.15.57:0/2910567372 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59a2d9d640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.551+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda5bd0 0x55d69edad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59a259c640 1 -- 172.21.15.57:0/2910567372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda5bd0 msgr2=0x55d69edad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59a259c640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda5bd0 0x55d69edad2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59a259c640 1 -- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 msgr2=0x55d69edad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59a259c640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59a259c640 1 -- 172.21.15.57:0/2910567372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d69eda0ee0 con 0x55d69eda52b0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59a259c640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edacd90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f59980bf190 tx=0x7f59980ba550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ce1cdcf25a2752 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f5989ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f5989ffb640 1 -- 172.21.15.57:0/2910567372 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59980c7550 con 0x55d69eda52b0 2024-04-21T08:45:30.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f5989ffb640 1 -- 172.21.15.57:0/2910567372 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f59980c70c0 con 0x55d69eda52b0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59980befe0 con 0x55d69eda52b0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f5989ffb640 1 -- 172.21.15.57:0/2910567372 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f59980be8e0 con 0x55d69eda52b0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 10 client.15603 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d69eda0ee0 con 0x55d69eda52b0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f5989ffb640 1 -- 172.21.15.57:0/2910567372 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59980befe0 con 0x55d69eda52b0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f5989ffb640 1 -- 172.21.15.57:0/2910567372 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f59980ddaa0 con 0x55d69eda52b0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 10 client.15603 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 20 client.15603 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69ee29280 0x55d69ee2b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 4 client.15603 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.552+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d69ed11a70 con 0x55d69ee29280 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.553+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69ee29280 0x55d69ee2b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.553+0000 7f59a1d9b640 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69ee29280 0x55d69ee2b670 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d0147f00a32d39 server_cookie=f95edd9420d2dc5b in_seq=0 out_seq=0 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.553+0000 7f5989ffb640 10 client.15603 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.553+0000 7f5989ffb640 1 -- 172.21.15.57:0/2910567372 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d69ed11a70 con 0x55d69ee29280 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.553+0000 7f5989ffb640 10 client.15603 handle_command_reply: tid=0 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.553+0000 7f59ab47c740 1 client.15603 shutdown 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 msgr2=0x55d69edacd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edacd90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f59980bf190 tx=0x7f59980ba550 comp rx=0 tx=0).stop 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 shutdown_connections 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d69ee29280 0x55d69ee2b670 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d69edb2e90 0x55d69edad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f5989ffb640 0 client.15603 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d69eda5bd0 0x55d69edad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 --2- 172.21.15.57:0/2910567372 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d69eda52b0 0x55d69edacd90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 >> 172.21.15.57:0/2910567372 conn(0x55d69ed9fcb0 msgr2=0x55d69eda10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 shutdown_connections 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 1 -- 172.21.15.57:0/2910567372 wait complete. 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59ab47c740 20 client.15603 trim_cache size 0 max 16384 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: "46517aef-40ab-4c5f-a4b7-6625836c57a1": { 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "46517aef-40ab-4c5f-a4b7-6625836c57a1", 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:45:30.560 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:45:30.561 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:45:30.561 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:45:30.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f598c0a3db0 msgr2=0x7f598c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f598c0a3db0 0x7f598c0a6270 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f599c00afd0 tx=0x7f599c005c10 comp rx=0 tx=0).stop 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 msgr2=0x7f59a419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a419e3e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5998005940 tx=0x7f59980045e0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 shutdown_connections 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f598c0a3db0 0x7f598c0a6270 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59a410c500 0x7f59a4198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59a4103ae0 0x7f59a419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 --2- 172.21.15.57:0/879575899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f59a41031c0 0x7f59a419e3e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 >> 172.21.15.57:0/879575899 conn(0x7f59a4078000 msgr2=0x7f59a40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 shutdown_connections 2024-04-21T08:45:30.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.554+0000 7f59a8881640 1 -- 172.21.15.57:0/879575899 wait complete. 2024-04-21T08:45:30.563 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi057 to 177.2M 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Unable to set osd_memory_target on smithi057 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi057:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.564 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3896805164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2178866533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.565 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:30 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/337835812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.566 INFO:tasks.cephfs.filesystem:scrub status for tag:46517aef-40ab-4c5f-a4b7-6625836c57a1 - {'path': '/', 'tag': '46517aef-40ab-4c5f-a4b7-6625836c57a1', 'options': 'recursive,force'} 2024-04-21T08:45:30.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.595+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f5fb4000bc0 con 0x7f5fb00a3e60 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi057 to 177.2M 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Unable to set osd_memory_target on smithi057 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi057:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3896805164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2178866533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.637 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:30 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/337835812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.659 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.660+0000 7f5faffff640 1 -- 172.21.15.57:0/3114081405 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5fb4000bc0 con 0x7f5fb00a3e60 2024-04-21T08:45:30.660 INFO:teuthology.orchestra.run.smithi057.stdout:Schedule set for path / 2024-04-21T08:45:30.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5fb00a3e60 msgr2=0x7f5fb00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5fb00a3e60 0x7f5fb00a6320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fc819bfb0 tx=0x7f5fbc00a250 comp rx=0 tx=0).stop 2024-04-21T08:45:30.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 msgr2=0x7f5fc81a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 0x7f5fc81a06a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8005940 tx=0x7f5fb80045e0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.662 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 shutdown_connections 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5fb00a3e60 0x7f5fb00a6320 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5fc819a760 0x7f5fc819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5fc8107270 0x7f5fc81a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 --2- 172.21.15.57:0/3114081405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fc81021a0 0x7f5fc81a06a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.662+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 >> 172.21.15.57:0/3114081405 conn(0x7f5fc80fda00 msgr2=0x7f5fc80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.663+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 shutdown_connections 2024-04-21T08:45:30.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.663+0000 7f5fcd759640 1 -- 172.21.15.57:0/3114081405 wait complete. 2024-04-21T08:45:30.673 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-21T08:45:30.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 -- 172.21.15.57:0/2995805138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb128101a90 msgr2=0x7fb128105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 --2- 172.21.15.57:0/2995805138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb128101a90 0x7fb128105ad0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb118009900 tx=0x7fb11802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 -- 172.21.15.57:0/2995805138 shutdown_connections 2024-04-21T08:45:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 --2- 172.21.15.57:0/2995805138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1281061b0 0x7fb12810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 --2- 172.21.15.57:0/2995805138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb128101a90 0x7fb128105ad0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 --2- 172.21.15.57:0/2995805138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1281010e0 0x7fb1281014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 -- 172.21.15.57:0/2995805138 >> 172.21.15.57:0/2995805138 conn(0x7fb1280771d0 msgr2=0x7fb1280775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 -- 172.21.15.57:0/2995805138 shutdown_connections 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.819+0000 7fb12f76d640 1 -- 172.21.15.57:0/2995805138 wait complete. 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 Processor -- start 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 -- start start 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1281010e0 0x7fb1281986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1281061b0 0x7fb128198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 0x7fb12819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb128112230 con 0x7fb12819cf70 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281120b0 con 0x7fb1281010e0 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12f76d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb1281123b0 con 0x7fb1281061b0 2024-04-21T08:45:30.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 0x7fb12819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 0x7fb12819d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46568/0 (socket says 172.21.15.57:46568) 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 -- 172.21.15.57:0/568925127 learned_addr learned my addr 172.21.15.57:0/568925127 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12d509640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1281010e0 0x7fb1281986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 -- 172.21.15.57:0/568925127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1281061b0 msgr2=0x7fb128198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1281061b0 0x7fb128198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 -- 172.21.15.57:0/568925127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1281010e0 msgr2=0x7fb1281986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1281010e0 0x7fb1281986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.820+0000 7fb12dd0a640 1 -- 172.21.15.57:0/568925127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12819da60 con 0x7fb12819cf70 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.821+0000 7fb12dd0a640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 0x7fb12819d420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb12000eb10 tx=0x7fb12000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.821+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120004b20 con 0x7fb12819cf70 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.821+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb120004cc0 con 0x7fb12819cf70 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.821+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12819dd50 con 0x7fb12819cf70 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.821+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120010670 con 0x7fb12819cf70 2024-04-21T08:45:30.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.821+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1281a54f0 con 0x7fb12819cf70 2024-04-21T08:45:30.844 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi057 to 177.2M 2024-04-21T08:45:30.844 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Unable to set osd_memory_target on smithi057 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-21T08:45:30.844 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi057:/etc/ceph/ceph.conf 2024-04-21T08:45:30.844 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi129:/etc/ceph/ceph.conf 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.conf 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi057:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: Updating smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3896805164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:30.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2178866533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:30 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/337835812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.822+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12804fce0 con 0x7fb12819cf70 2024-04-21T08:45:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.823+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fb120005000 con 0x7fb12819cf70 2024-04-21T08:45:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.824+0000 7fb11e7fc640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb1140a3e60 0x7fb1140a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:30.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.824+0000 7fb12d509640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb1140a3e60 0x7fb1140a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.824+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fb1200d1530 con 0x7fb12819cf70 2024-04-21T08:45:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.824+0000 7fb12d509640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb1140a3e60 0x7fb1140a6320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb124009bd0 tx=0x7fb124005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:30.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:30.827+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb120053ff0 con 0x7fb12819cf70 2024-04-21T08:45:31.039 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.040+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fb128101a90 con 0x7fb1140a3e60 2024-04-21T08:45:31.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.101+0000 7fb11e7fc640 1 -- 172.21.15.57:0/568925127 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fb128101a90 con 0x7fb1140a3e60 2024-04-21T08:45:31.102 INFO:teuthology.orchestra.run.smithi057.stdout:Retention added to path / 2024-04-21T08:45:31.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb1140a3e60 msgr2=0x7fb1140a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb1140a3e60 0x7fb1140a6320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb124009bd0 tx=0x7fb124005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:31.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 msgr2=0x7fb12819d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 0x7fb12819d420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb12000eb10 tx=0x7fb12000c2d0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 shutdown_connections 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb1140a3e60 0x7fb1140a6320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb12819cf70 0x7fb12819d420 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1281061b0 0x7fb128198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 --2- 172.21.15.57:0/568925127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1281010e0 0x7fb1281986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 >> 172.21.15.57:0/568925127 conn(0x7fb1280771d0 msgr2=0x7fb1280fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 shutdown_connections 2024-04-21T08:45:31.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.103+0000 7fb12f76d640 1 -- 172.21.15.57:0/568925127 wait complete. 2024-04-21T08:45:31.113 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:45:31.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 -- 172.21.15.57:0/3687723201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1072a0 msgr2=0x7fa37c10ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 --2- 172.21.15.57:0/3687723201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1072a0 0x7fa37c10ee70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa374009ca0 tx=0x7fa37402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 -- 172.21.15.57:0/3687723201 shutdown_connections 2024-04-21T08:45:31.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 --2- 172.21.15.57:0/3687723201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1072a0 0x7fa37c10ee70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 --2- 172.21.15.57:0/3687723201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa37c102b80 0x7fa37c106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 --2- 172.21.15.57:0/3687723201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1021d0 0x7fa37c1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.216+0000 7fa38365c640 1 -- 172.21.15.57:0/3687723201 >> 172.21.15.57:0/3687723201 conn(0x7fa37c0793e0 msgr2=0x7fa37c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 -- 172.21.15.57:0/3687723201 shutdown_connections 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 -- 172.21.15.57:0/3687723201 wait complete. 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 Processor -- start 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 -- start start 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 0x7fa37c074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1072a0 0x7fa37c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa37c074cd0 0x7fa37c1a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c112210 con 0x7fa37c1021d0 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c112090 con 0x7fa37c1072a0 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa38365c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c112390 con 0x7fa37c074cd0 2024-04-21T08:45:31.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa3813f8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 0x7fa37c074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa3813f8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 0x7fa37c074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46580/0 (socket says 172.21.15.57:46580) 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.217+0000 7fa3813f8640 1 -- 172.21.15.57:0/361780187 learned_addr learned my addr 172.21.15.57:0/361780187 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa380bf7640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1072a0 0x7fa37c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa381bf9640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa37c074cd0 0x7fa37c1a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3813f8640 1 -- 172.21.15.57:0/361780187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa37c074cd0 msgr2=0x7fa37c1a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3813f8640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa37c074cd0 0x7fa37c1a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3813f8640 1 -- 172.21.15.57:0/361780187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1072a0 msgr2=0x7fa37c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3813f8640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1072a0 0x7fa37c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3813f8640 1 -- 172.21.15.57:0/361780187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37c1a6f40 con 0x7fa37c1021d0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa380bf7640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1072a0 0x7fa37c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3813f8640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 0x7fa37c074250 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa37800d950 tx=0x7fa37800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378017070 con 0x7fa37c1021d0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa378009c30 con 0x7fa37c1021d0 2024-04-21T08:45:31.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa37c1a7230 con 0x7fa37c1021d0 2024-04-21T08:45:31.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37800f620 con 0x7fa37c1021d0 2024-04-21T08:45:31.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.218+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa37c1a77d0 con 0x7fa37c1021d0 2024-04-21T08:45:31.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.219+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa37c04fce0 con 0x7fa37c1021d0 2024-04-21T08:45:31.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.220+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fa378010460 con 0x7fa37c1021d0 2024-04-21T08:45:31.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.221+0000 7fa3727fc640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa3680a3f10 0x7fa3680a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.221+0000 7fa380bf7640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa3680a3f10 0x7fa3680a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.221+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa3780d0950 con 0x7fa37c1021d0 2024-04-21T08:45:31.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.222+0000 7fa380bf7640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa3680a3f10 0x7fa3680a63d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa36c005fd0 tx=0x7fa36c00b5d0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:31.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.226+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa37809cb20 con 0x7fa37c1021d0 2024-04-21T08:45:31.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.436+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa37c102bf0 con 0x7fa3680a3f10 2024-04-21T08:45:31.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.444+0000 7fa3727fc640 1 -- 172.21.15.57:0/361780187 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fa37c102bf0 con 0x7fa3680a3f10 2024-04-21T08:45:31.443 INFO:teuthology.orchestra.run.smithi057.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:45:30", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-21T08:45:31.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa3680a3f10 msgr2=0x7fa3680a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa3680a3f10 0x7fa3680a63d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa36c005fd0 tx=0x7fa36c00b5d0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 msgr2=0x7fa37c074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 0x7fa37c074250 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa37800d950 tx=0x7fa37800de10 comp rx=0 tx=0).stop 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 shutdown_connections 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa3680a3f10 0x7fa3680a63d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa37c074cd0 0x7fa37c1a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa37c1072a0 0x7fa37c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 --2- 172.21.15.57:0/361780187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa37c1021d0 0x7fa37c074250 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 >> 172.21.15.57:0/361780187 conn(0x7fa37c0793e0 msgr2=0x7fa37c0ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 shutdown_connections 2024-04-21T08:45:31.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.446+0000 7fa38365c640 1 -- 172.21.15.57:0/361780187 wait complete. 2024-04-21T08:45:31.455 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:31 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:45:31.455 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:31 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.455 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:31 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:45:31.455 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:31 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.455 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:31 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:45:31.456 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:45:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 -- 172.21.15.57:0/3320261153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f0101a90 msgr2=0x7f41f0105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 --2- 172.21.15.57:0/3320261153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f0101a90 0x7f41f0105ad0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009900 tx=0x7f41e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 -- 172.21.15.57:0/3320261153 shutdown_connections 2024-04-21T08:45:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 --2- 172.21.15.57:0/3320261153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f01061b0 0x7f41f010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 --2- 172.21.15.57:0/3320261153 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f0101a90 0x7f41f0105ad0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 --2- 172.21.15.57:0/3320261153 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 0x7f41f01014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 -- 172.21.15.57:0/3320261153 >> 172.21.15.57:0/3320261153 conn(0x7f41f00771d0 msgr2=0x7f41f00775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 -- 172.21.15.57:0/3320261153 shutdown_connections 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.560+0000 7f41f874b640 1 -- 172.21.15.57:0/3320261153 wait complete. 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 Processor -- start 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 -- start start 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 0x7f41f0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f01061b0 0x7f41f0198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 0x7f41f019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f41f01110d0 con 0x7f41f019cf40 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41f0110f50 con 0x7f41f01061b0 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f874b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f41f0111250 con 0x7f41f01010e0 2024-04-21T08:45:31.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 0x7f41f019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 0x7f41f019d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46602/0 (socket says 172.21.15.57:46602) 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 -- 172.21.15.57:0/2613388073 learned_addr learned my addr 172.21.15.57:0/2613388073 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f5ce6640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f01061b0 0x7f41f0198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f64e7640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 0x7f41f0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 -- 172.21.15.57:0/2613388073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 msgr2=0x7f41f0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 0x7f41f0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 -- 172.21.15.57:0/2613388073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f01061b0 msgr2=0x7f41f0198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f01061b0 0x7f41f0198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f6ce8640 1 -- 172.21.15.57:0/2613388073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f019da30 con 0x7f41f019cf40 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.561+0000 7f41f64e7640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 0x7f41f0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.562+0000 7f41f6ce8640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 0x7f41f019d3f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f41e800eaa0 tx=0x7f41e800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.562+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e800cdb0 con 0x7f41f019cf40 2024-04-21T08:45:31.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.562+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41e8005020 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.562+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f019dd20 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.562+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e80116d0 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.562+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f01a5490 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.563+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f004fce0 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.564+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f41e80051d0 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.565+0000 7f41e77fe640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f41dc0a3e60 0x7f41dc0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.565+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f41e80d1270 con 0x7f41f019cf40 2024-04-21T08:45:31.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.565+0000 7f41f64e7640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f41dc0a3e60 0x7f41dc0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.565+0000 7f41f64e7640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f41dc0a3e60 0x7f41dc0a6320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41ec009c20 tx=0x7f41ec005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:31.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.568+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41e8016030 con 0x7f41f019cf40 2024-04-21T08:45:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:31 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:45:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:31 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:31 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:45:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:31 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:31 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:45:31.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.780+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f41f0101a90 con 0x7f41dc0a3e60 2024-04-21T08:45:31.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.787+0000 7f41e77fe640 1 -- 172.21.15.57:0/2613388073 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f41f0101a90 con 0x7f41dc0a3e60 2024-04-21T08:45:31.787 INFO:teuthology.orchestra.run.smithi057.stdout:/ 1m 6m3h 2024-04-21T08:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.789+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f41dc0a3e60 msgr2=0x7f41dc0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.789+0000 7f41f874b640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f41dc0a3e60 0x7f41dc0a6320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f41ec009c20 tx=0x7f41ec005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.789+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 msgr2=0x7f41f019d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.789+0000 7f41f874b640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 0x7f41f019d3f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f41e800eaa0 tx=0x7f41e800ef60 comp rx=0 tx=0).stop 2024-04-21T08:45:31.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 shutdown_connections 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f41dc0a3e60 0x7f41dc0a6320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f41f019cf40 0x7f41f019d3f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41f01061b0 0x7f41f0198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 --2- 172.21.15.57:0/2613388073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f01010e0 0x7f41f0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 >> 172.21.15.57:0/2613388073 conn(0x7f41f00771d0 msgr2=0x7f41f006aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 shutdown_connections 2024-04-21T08:45:31.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.790+0000 7f41f874b640 1 -- 172.21.15.57:0/2613388073 wait complete. 2024-04-21T08:45:31.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-21T08:45:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:31 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:45:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:31 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:31 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:45:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:31 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:45:31.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:31 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:45:31.825 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-21T08:45:31.835 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-21T08:45:31.851 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:45:31.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-21T08:45:31.867 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-21T08:45:31.879 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:31.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 -- 172.21.15.57:0/2272660766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd76810c5a0 msgr2=0x7fd76810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/2272660766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd76810c5a0 0x7fd76810e9d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd760009ca0 tx=0x7fd76002c180 comp rx=0 tx=0).stop 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 -- 172.21.15.57:0/2272660766 shutdown_connections 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/2272660766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd76810c5a0 0x7fd76810e9d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/2272660766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd768103af0 0x7fd76810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/2272660766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7681031d0 0x7fd7681035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 -- 172.21.15.57:0/2272660766 >> 172.21.15.57:0/2272660766 conn(0x7fd7680771d0 msgr2=0x7fd7680775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 -- 172.21.15.57:0/2272660766 shutdown_connections 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.981+0000 7fd76e5b1640 1 -- 172.21.15.57:0/2272660766 wait complete. 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 Processor -- start 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 -- start start 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7681031d0 0x7fd7681986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 0x7fd768198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd76810c5a0 0x7fd76819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd7681121a0 con 0x7fd768103af0 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd768112020 con 0x7fd7681031d0 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd76e5b1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd768112320 con 0x7fd76810c5a0 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd7677fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 0x7fd768198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd7677fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 0x7fd768198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46614/0 (socket says 172.21.15.57:46614) 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.982+0000 7fd7677fe640 1 -- 172.21.15.57:0/1765726282 learned_addr learned my addr 172.21.15.57:0/1765726282 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd767fff640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7681031d0 0x7fd7681986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd76cb4e640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd76810c5a0 0x7fd76819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7677fe640 1 -- 172.21.15.57:0/1765726282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd76810c5a0 msgr2=0x7fd76819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7677fe640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd76810c5a0 0x7fd76819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7677fe640 1 -- 172.21.15.57:0/1765726282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7681031d0 msgr2=0x7fd7681986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:31.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7677fe640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7681031d0 0x7fd7681986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7677fe640 1 -- 172.21.15.57:0/1765726282 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76819d6b0 con 0x7fd768103af0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd76cb4e640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd76810c5a0 0x7fd76819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd767fff640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7681031d0 0x7fd7681986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7677fe640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 0x7fd768198c00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd75800e9f0 tx=0x7fd75800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75800cd80 con 0x7fd768103af0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7580108e0 con 0x7fd768103af0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76819d9a0 con 0x7fd768103af0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758018980 con 0x7fd768103af0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.983+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7681a5230 con 0x7fd768103af0 2024-04-21T08:45:31.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.984+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd768069230 con 0x7fd768103af0 2024-04-21T08:45:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.985+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fd758010430 con 0x7fd768103af0 2024-04-21T08:45:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.986+0000 7fd7657fa640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd7500a3db0 0x7fd7500a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.986+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd7580d1120 con 0x7fd768103af0 2024-04-21T08:45:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.986+0000 7fd767fff640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd7500a3db0 0x7fd7500a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:31.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.986+0000 7fd767fff640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd7500a3db0 0x7fd7500a6270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd75c0058c0 tx=0x7fd75c005850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:31.990 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:31.990+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd75809d290 con 0x7fd768103af0 2024-04-21T08:45:32.178 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.179+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7681076f0 con 0x7fd768103af0 2024-04-21T08:45:32.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.181+0000 7fd7657fa640 1 -- 172.21.15.57:0/1765726282 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25074 (secure 0 0 0) 0x7fd75809cb90 con 0x7fd768103af0 2024-04-21T08:45:32.180 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:32.180 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:45:24.897611+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:32.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.182+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd7500a3db0 msgr2=0x7fd7500a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.182+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd7500a3db0 0x7fd7500a6270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd75c0058c0 tx=0x7fd75c005850 comp rx=0 tx=0).stop 2024-04-21T08:45:32.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.182+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 msgr2=0x7fd768198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.182+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 0x7fd768198c00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd75800e9f0 tx=0x7fd75800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 shutdown_connections 2024-04-21T08:45:32.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd7500a3db0 0x7fd7500a6270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd76810c5a0 0x7fd76819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd768103af0 0x7fd768198c00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 --2- 172.21.15.57:0/1765726282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7681031d0 0x7fd7681986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 >> 172.21.15.57:0/1765726282 conn(0x7fd7680771d0 msgr2=0x7fd7680fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:32.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 shutdown_connections 2024-04-21T08:45:32.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.183+0000 7fd76e5b1640 1 -- 172.21.15.57:0/1765726282 wait complete. 2024-04-21T08:45:32.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:32.194 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:32.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/138867718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 msgr2=0x7ffa4406aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/138867718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa4406aba0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffa40009900 tx=0x7ffa4002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/138867718 shutdown_connections 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/138867718 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa44107280 0x7ffa4410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/138867718 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa4406b170 0x7ffa44106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/138867718 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa4406aba0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/138867718 >> 172.21.15.57:0/138867718 conn(0x7ffa44076c50 msgr2=0x7ffa44077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/138867718 shutdown_connections 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.300+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/138867718 wait complete. 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 Processor -- start 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 -- start start 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa44198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa44107280 0x7ffa44198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa4419cfb0 0x7ffa4419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ffa44112240 con 0x7ffa4406a7c0 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441120c0 con 0x7ffa44107280 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4c1ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffa441123c0 con 0x7ffa4419cfb0 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa44198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa44198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46618/0 (socket says 172.21.15.57:46618) 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 -- 172.21.15.57:0/3073303 learned_addr learned my addr 172.21.15.57:0/3073303 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49769640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa44107280 0x7ffa44198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa4a76b640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa4419cfb0 0x7ffa4419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 -- 172.21.15.57:0/3073303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa4419cfb0 msgr2=0x7ffa4419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa4419cfb0 0x7ffa4419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 -- 172.21.15.57:0/3073303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa44107280 msgr2=0x7ffa44198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa44107280 0x7ffa44198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.301+0000 7ffa49f6a640 1 -- 172.21.15.57:0/3073303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4419daa0 con 0x7ffa4406a7c0 2024-04-21T08:45:32.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.302+0000 7ffa49f6a640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa44198740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffa40000c00 tx=0x7ffa4002c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:32.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.302+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa40043020 con 0x7ffa4406a7c0 2024-04-21T08:45:32.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.302+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa4002cb30 con 0x7ffa4406a7c0 2024-04-21T08:45:32.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.302+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa4419dd30 con 0x7ffa4406a7c0 2024-04-21T08:45:32.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.302+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa40003a50 con 0x7ffa4406a7c0 2024-04-21T08:45:32.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.302+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa441a5580 con 0x7ffa4406a7c0 2024-04-21T08:45:32.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.303+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa44103680 con 0x7ffa4406a7c0 2024-04-21T08:45:32.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.304+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7ffa40046020 con 0x7ffa4406a7c0 2024-04-21T08:45:32.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.305+0000 7ffa3affd640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ffa300a3e30 0x7ffa300a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.305+0000 7ffa49769640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ffa300a3e30 0x7ffa300a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.305+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ffa400f30d0 con 0x7ffa4406a7c0 2024-04-21T08:45:32.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.305+0000 7ffa49769640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ffa300a3e30 0x7ffa300a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa44100d00 tx=0x7ffa34005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:32.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.308+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa4004d020 con 0x7ffa4406a7c0 2024-04-21T08:45:32.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.494+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa4410a490 con 0x7ffa4406a7c0 2024-04-21T08:45:32.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.495+0000 7ffa3affd640 1 -- 172.21.15.57:0/3073303 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7ffa400bf3e0 con 0x7ffa4406a7c0 2024-04-21T08:45:32.494 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:32.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ffa300a3e30 msgr2=0x7ffa300a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ffa300a3e30 0x7ffa300a62f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa44100d00 tx=0x7ffa34005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 msgr2=0x7ffa44198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa44198740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffa40000c00 tx=0x7ffa4002c630 comp rx=0 tx=0).stop 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 shutdown_connections 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ffa300a3e30 0x7ffa300a62f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa4419cfb0 0x7ffa4419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa44107280 0x7ffa44198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 --2- 172.21.15.57:0/3073303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa4406a7c0 0x7ffa44198740 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 >> 172.21.15.57:0/3073303 conn(0x7ffa44076c50 msgr2=0x7ffa440ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 shutdown_connections 2024-04-21T08:45:32.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.497+0000 7ffa4c1ce640 1 -- 172.21.15.57:0/3073303 wait complete. 2024-04-21T08:45:32.506 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:32 smithi057 ceph-mon[27968]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.506 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:32 smithi057 ceph-mon[27968]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:32.506 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:32 smithi057 ceph-mon[27968]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.506 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:32 smithi057 ceph-mon[27968]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.507 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:32 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1765726282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:32.508 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:32.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 -- 172.21.15.57:0/3731843005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101a40 msgr2=0x7f5b58105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 --2- 172.21.15.57:0/3731843005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101a40 0x7f5b58105a80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009900 tx=0x7f5b4802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 -- 172.21.15.57:0/3731843005 shutdown_connections 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 --2- 172.21.15.57:0/3731843005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b58106160 0x7f5b5810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 --2- 172.21.15.57:0/3731843005 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101a40 0x7f5b58105a80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 --2- 172.21.15.57:0/3731843005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b58101090 0x7f5b58101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 -- 172.21.15.57:0/3731843005 >> 172.21.15.57:0/3731843005 conn(0x7f5b58077f50 msgr2=0x7f5b58078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 -- 172.21.15.57:0/3731843005 shutdown_connections 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 -- 172.21.15.57:0/3731843005 wait complete. 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 Processor -- start 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 -- start start 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.614+0000 7f5b5f488640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 0x7f5b58199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b58106160 0x7f5b58199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5819a060 0x7f5b581a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5b581110f0 con 0x7f5b58101090 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58110f70 con 0x7f5b5819a060 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5b58111270 con 0x7f5b58106160 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 0x7f5b58199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 0x7f5b58199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46638/0 (socket says 172.21.15.57:46638) 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 -- 172.21.15.57:0/4096303653 learned_addr learned my addr 172.21.15.57:0/4096303653 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:32.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5da25640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5819a060 0x7f5b581a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5ca23640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b58106160 0x7f5b58199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 -- 172.21.15.57:0/4096303653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b58106160 msgr2=0x7f5b58199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b58106160 0x7f5b58199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 -- 172.21.15.57:0/4096303653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5819a060 msgr2=0x7f5b581a4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5819a060 0x7f5b581a4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 -- 172.21.15.57:0/4096303653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b581a4e10 con 0x7f5b58101090 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5d224640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 0x7f5b58199590 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400d770 tx=0x7f5b5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54017070 con 0x7f5b58101090 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b581a5100 con 0x7f5b58101090 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b540054d0 con 0x7f5b58101090 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b54004c30 con 0x7f5b58101090 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.615+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b581a5620 con 0x7f5b58101090 2024-04-21T08:45:32.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.616+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b58068d70 con 0x7f5b58101090 2024-04-21T08:45:32.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.617+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f5b54004e10 con 0x7f5b58101090 2024-04-21T08:45:32.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.618+0000 7f5b4e7fc640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b440a3db0 0x7f5b440a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.618+0000 7f5b5ca23640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b440a3db0 0x7f5b440a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.618+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5b540d07d0 con 0x7f5b58101090 2024-04-21T08:45:32.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.619+0000 7f5b5ca23640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b440a3db0 0x7f5b440a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5b4802c240 tx=0x7f5b48005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:32.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.623+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b5409c9a0 con 0x7f5b58101090 2024-04-21T08:45:32.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:32 smithi136 ceph-mon[31791]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:32 smithi136 ceph-mon[31791]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:32 smithi136 ceph-mon[31791]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:32 smithi136 ceph-mon[31791]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:32 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1765726282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:32.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.807+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5b58109390 con 0x7f5b58101090 2024-04-21T08:45:32.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.807+0000 7f5b4e7fc640 1 -- 172.21.15.57:0/4096303653 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f5b5409c2a0 con 0x7f5b58101090 2024-04-21T08:45:32.807 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:32.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b440a3db0 msgr2=0x7f5b440a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b440a3db0 0x7f5b440a6270 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5b4802c240 tx=0x7f5b48005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 msgr2=0x7f5b58199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 0x7f5b58199590 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400d770 tx=0x7f5b5400dc30 comp rx=0 tx=0).stop 2024-04-21T08:45:32.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 shutdown_connections 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b440a3db0 0x7f5b440a6270 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b5819a060 0x7f5b581a4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b58106160 0x7f5b58199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 --2- 172.21.15.57:0/4096303653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b58101090 0x7f5b58199590 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 >> 172.21.15.57:0/4096303653 conn(0x7f5b58077f50 msgr2=0x7f5b580ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 shutdown_connections 2024-04-21T08:45:32.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.810+0000 7f5b5f488640 1 -- 172.21.15.57:0/4096303653 wait complete. 2024-04-21T08:45:32.820 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:32 smithi129 ceph-mon[31272]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:32 smithi129 ceph-mon[31272]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-21T08:45:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:32 smithi129 ceph-mon[31272]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:32 smithi129 ceph-mon[31272]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:32 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1765726282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 -- 172.21.15.57:0/3831994852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 msgr2=0x7fdcd406a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/3831994852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd406a9d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009930 tx=0x7fdcc402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 -- 172.21.15.57:0/3831994852 shutdown_connections 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/3831994852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 0x7fdcd410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/3831994852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd406a9d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/3831994852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd4108000 0x7fdcd41083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 -- 172.21.15.57:0/3831994852 >> 172.21.15.57:0/3831994852 conn(0x7fdcd4076c50 msgr2=0x7fdcd4077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 -- 172.21.15.57:0/3831994852 shutdown_connections 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.925+0000 7fdcdadf4640 1 -- 172.21.15.57:0/3831994852 wait complete. 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 Processor -- start 2024-04-21T08:45:32.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 -- start start 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd4101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 0x7fdcd41017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd4108000 0x7fdcd4101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4112190 con 0x7fdcd406a550 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4112010 con 0x7fdcd4108000 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcdadf4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4112310 con 0x7fdcd406af10 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd4101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd4101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55414/0 (socket says 172.21.15.57:55414) 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 -- 172.21.15.57:0/2332566019 learned_addr learned my addr 172.21.15.57:0/2332566019 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd9391640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd4108000 0x7fdcd4101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdccbfff640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 0x7fdcd41017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 -- 172.21.15.57:0/2332566019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 msgr2=0x7fdcd41017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 0x7fdcd41017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 -- 172.21.15.57:0/2332566019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd4108000 msgr2=0x7fdcd4101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:32.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.926+0000 7fdcd8b90640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd4108000 0x7fdcd4101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcd8b90640 1 -- 172.21.15.57:0/2332566019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcd41a6c90 con 0x7fdcd406a550 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdccbfff640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 0x7fdcd41017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcd8b90640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd4101270 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c920 tx=0x7fdcd000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0007c70 con 0x7fdcd406a550 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcd0007e10 con 0x7fdcd406a550 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd41a6f80 con 0x7fdcd406a550 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0018820 con 0x7fdcd406a550 2024-04-21T08:45:32.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.927+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41a7440 con 0x7fdcd406a550 2024-04-21T08:45:32.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.928+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd4103660 con 0x7fdcd406a550 2024-04-21T08:45:32.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.929+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fdcd000f450 con 0x7fdcd406a550 2024-04-21T08:45:32.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.930+0000 7fdcc9ffb640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdcbc0a3db0 0x7fdcbc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:32.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.930+0000 7fdccbfff640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdcbc0a3db0 0x7fdcbc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:32.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.930+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fdcd00d0df0 con 0x7fdcd406a550 2024-04-21T08:45:32.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.930+0000 7fdccbfff640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdcbc0a3db0 0x7fdcbc0a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4036040 tx=0x7fdcc4000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:32.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:32.934+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdcd0014030 con 0x7fdcd406a550 2024-04-21T08:45:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.120+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdcd402d5c0 con 0x7fdcd406a550 2024-04-21T08:45:33.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.121+0000 7fdcc9ffb640 1 -- 172.21.15.57:0/2332566019 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fdcd009cf60 con 0x7fdcd406a550 2024-04-21T08:45:33.120 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:33.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdcbc0a3db0 msgr2=0x7fdcbc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdcbc0a3db0 0x7fdcbc0a6270 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4036040 tx=0x7fdcc4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 msgr2=0x7fdcd4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd4101270 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000c920 tx=0x7fdcd000cde0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 shutdown_connections 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdcbc0a3db0 0x7fdcbc0a6270 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdcd4108000 0x7fdcd4101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdcd406af10 0x7fdcd41017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 --2- 172.21.15.57:0/2332566019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcd406a550 0x7fdcd4101270 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 >> 172.21.15.57:0/2332566019 conn(0x7fdcd4076c50 msgr2=0x7fdcd40fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 shutdown_connections 2024-04-21T08:45:33.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.123+0000 7fdcdadf4640 1 -- 172.21.15.57:0/2332566019 wait complete. 2024-04-21T08:45:33.133 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 -- 172.21.15.57:0/3012264337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c1031d0 msgr2=0x7f466c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 --2- 172.21.15.57:0/3012264337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c1031d0 0x7f466c1035b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4660009900 tx=0x7f466002bd60 comp rx=0 tx=0).stop 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 -- 172.21.15.57:0/3012264337 shutdown_connections 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 --2- 172.21.15.57:0/3012264337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c10c510 0x7f466c10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 --2- 172.21.15.57:0/3012264337 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f466c103af0 0x7f466c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 --2- 172.21.15.57:0/3012264337 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c1031d0 0x7f466c1035b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 -- 172.21.15.57:0/3012264337 >> 172.21.15.57:0/3012264337 conn(0x7f466c077f50 msgr2=0x7f466c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 -- 172.21.15.57:0/3012264337 shutdown_connections 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.239+0000 7f4672e99640 1 -- 172.21.15.57:0/3012264337 wait complete. 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 Processor -- start 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 -- start start 2024-04-21T08:45:33.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f466c1031d0 0x7f466c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c103af0 0x7f466c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 0x7f466c19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f466c112180 con 0x7f466c10c510 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f466c112000 con 0x7f466c103af0 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4672e99640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f466c112300 con 0x7f466c1031d0 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 0x7f466c19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f466bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c103af0 0x7f466c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 0x7f466c19d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55428/0 (socket says 172.21.15.57:55428) 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 -- 172.21.15.57:0/1205625917 learned_addr learned my addr 172.21.15.57:0/1205625917 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4670c35640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f466c1031d0 0x7f466c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 -- 172.21.15.57:0/1205625917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f466c1031d0 msgr2=0x7f466c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f466c1031d0 0x7f466c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 -- 172.21.15.57:0/1205625917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c103af0 msgr2=0x7f466c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c103af0 0x7f466c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f4671436640 1 -- 172.21.15.57:0/1205625917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466c19d730 con 0x7f466c10c510 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.240+0000 7f466bfff640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c103af0 0x7f466c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.241+0000 7f4671436640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 0x7f466c19d000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f466400eaa0 tx=0x7f466400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.241+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466400cdb0 con 0x7f466c10c510 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.241+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4664005020 con 0x7f466c10c510 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.241+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466c19da20 con 0x7f466c10c510 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.241+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46640116d0 con 0x7f466c10c510 2024-04-21T08:45:33.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.241+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f466c1a52b0 con 0x7f466c10c510 2024-04-21T08:45:33.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.242+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f466c068d70 con 0x7f466c10c510 2024-04-21T08:45:33.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.243+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f46640051d0 con 0x7f466c10c510 2024-04-21T08:45:33.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.244+0000 7f4669ffb640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f465c0a3db0 0x7f465c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.244+0000 7f4670c35640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f465c0a3db0 0x7f465c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.244+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f46640d0e90 con 0x7f466c10c510 2024-04-21T08:45:33.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.244+0000 7f4670c35640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f465c0a3db0 0x7f465c0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4660000c00 tx=0x7f466002c7e0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:33.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.248+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4664016030 con 0x7f466c10c510 2024-04-21T08:45:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.434+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f466c02d5c0 con 0x7f466c10c510 2024-04-21T08:45:33.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.434+0000 7f4669ffb640 1 -- 172.21.15.57:0/1205625917 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f466409d060 con 0x7f466c10c510 2024-04-21T08:45:33.434 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f465c0a3db0 msgr2=0x7f465c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f465c0a3db0 0x7f465c0a6270 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4660000c00 tx=0x7f466002c7e0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 msgr2=0x7f466c19d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 0x7f466c19d000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f466400eaa0 tx=0x7f466400ef60 comp rx=0 tx=0).stop 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 shutdown_connections 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f465c0a3db0 0x7f465c0a6270 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f466c10c510 0x7f466c19d000 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f466c103af0 0x7f466c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 --2- 172.21.15.57:0/1205625917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f466c1031d0 0x7f466c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 >> 172.21.15.57:0/1205625917 conn(0x7f466c077f50 msgr2=0x7f466c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:33.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 shutdown_connections 2024-04-21T08:45:33.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.436+0000 7f4672e99640 1 -- 172.21.15.57:0/1205625917 wait complete. 2024-04-21T08:45:33.445 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:33 smithi057 ceph-mon[27968]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:33.446 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:33 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3073303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.446 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:33 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4096303653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.446 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:33 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2332566019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.447 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:33.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 -- 172.21.15.57:0/1566478551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 msgr2=0x7f2628069540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 --2- 172.21.15.57:0/1566478551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f2628069540 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2610009ca0 tx=0x7f261002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 -- 172.21.15.57:0/1566478551 shutdown_connections 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 --2- 172.21.15.57:0/1566478551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2628069a80 0x7f262810eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 --2- 172.21.15.57:0/1566478551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f2628069540 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 --2- 172.21.15.57:0/1566478551 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 0x7f26281083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.549+0000 7f262d115640 1 -- 172.21.15.57:0/1566478551 >> 172.21.15.57:0/1566478551 conn(0x7f2628077e30 msgr2=0x7f2628078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.550+0000 7f262d115640 1 -- 172.21.15.57:0/1566478551 shutdown_connections 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.550+0000 7f262d115640 1 -- 172.21.15.57:0/1566478551 wait complete. 2024-04-21T08:45:33.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.550+0000 7f262d115640 1 Processor -- start 2024-04-21T08:45:33.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.550+0000 7f262d115640 1 -- start start 2024-04-21T08:45:33.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f262819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2628069a80 0x7f262819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 0x7f26281985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2628112160 con 0x7f26280690e0 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2628111fe0 con 0x7f2628107fe0 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f26281122e0 con 0x7f2628069a80 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f262819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f262819e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55450/0 (socket says 172.21.15.57:55450) 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 -- 172.21.15.57:0/1993520348 learned_addr learned my addr 172.21.15.57:0/1993520348 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262659c640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2628069a80 0x7f262819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262759e640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 0x7f26281985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 -- 172.21.15.57:0/1993520348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2628069a80 msgr2=0x7f262819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2628069a80 0x7f262819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 -- 172.21.15.57:0/1993520348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 msgr2=0x7f26281985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 0x7f26281985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 -- 172.21.15.57:0/1993520348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2628198d70 con 0x7f26280690e0 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262759e640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 0x7f26281985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:33.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f2626d9d640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f262819e440 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f261800e9f0 tx=0x7f261800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261800cd80 con 0x7f26280690e0 2024-04-21T08:45:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26180108e0 con 0x7f26280690e0 2024-04-21T08:45:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618018980 con 0x7f26280690e0 2024-04-21T08:45:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.551+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2628199060 con 0x7f26280690e0 2024-04-21T08:45:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.552+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26281a5300 con 0x7f26280690e0 2024-04-21T08:45:33.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.552+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2628068de0 con 0x7f26280690e0 2024-04-21T08:45:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.554+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f2618010430 con 0x7f26280690e0 2024-04-21T08:45:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.554+0000 7f261ffff640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f26140a3db0 0x7f26140a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.554+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f26180d1020 con 0x7f26280690e0 2024-04-21T08:45:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.554+0000 7f262659c640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f26140a3db0 0x7f26140a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.555+0000 7f262659c640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f26140a3db0 0x7f26140a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2610004af0 tx=0x7f2610005c50 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:33.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.559+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f261809d1f0 con 0x7f26280690e0 2024-04-21T08:45:33.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:33 smithi136 ceph-mon[31791]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:33.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:33 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3073303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:33 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4096303653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:33 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2332566019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.750+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2628199b20 con 0x7f26280690e0 2024-04-21T08:45:33.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.752+0000 7f261ffff640 1 -- 172.21.15.57:0/1993520348 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25074 (secure 0 0 0) 0x7f261809caf0 con 0x7f26280690e0 2024-04-21T08:45:33.752 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:33.752 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:45:24.897611+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:33.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f26140a3db0 msgr2=0x7f26140a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f26140a3db0 0x7f26140a6270 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2610004af0 tx=0x7f2610005c50 comp rx=0 tx=0).stop 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 msgr2=0x7f262819e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f262819e440 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f261800e9f0 tx=0x7f261800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 shutdown_connections 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f26140a3db0 0x7f26140a6270 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2628107fe0 0x7f26281985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2628069a80 0x7f262819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 --2- 172.21.15.57:0/1993520348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26280690e0 0x7f262819e440 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 >> 172.21.15.57:0/1993520348 conn(0x7f2628077e30 msgr2=0x7f2628100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 shutdown_connections 2024-04-21T08:45:33.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.754+0000 7f262d115640 1 -- 172.21.15.57:0/1993520348 wait complete. 2024-04-21T08:45:33.768 INFO:tasks.quiescer.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:33.768 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:33.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:33 smithi129 ceph-mon[31272]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:45:33.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:33 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3073303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:33 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4096303653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:33 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2332566019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:33.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 -- 172.21.15.57:0/634719758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 msgr2=0x7f129010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 --2- 172.21.15.57:0/634719758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f129010dd40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f1288009ca0 tx=0x7f128802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 -- 172.21.15.57:0/634719758 shutdown_connections 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 --2- 172.21.15.57:0/634719758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f129010dd40 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 --2- 172.21.15.57:0/634719758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1290101a50 0x7f1290105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 --2- 172.21.15.57:0/634719758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 0x7f1290101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 -- 172.21.15.57:0/634719758 >> 172.21.15.57:0/634719758 conn(0x7f1290077eb0 msgr2=0x7f12900782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 -- 172.21.15.57:0/634719758 shutdown_connections 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 -- 172.21.15.57:0/634719758 wait complete. 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 Processor -- start 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.874+0000 7f12962c1640 1 -- start start 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f12962c1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 0x7f12901986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f12962c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f1290198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f12962c1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129019cfb0 0x7f129019d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f12962c1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f12901110e0 con 0x7f1290106170 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f12962c1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1290110f60 con 0x7f129019cfb0 2024-04-21T08:45:33.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f12962c1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1290111260 con 0x7f12901010a0 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f1290198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f1290198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55466/0 (socket says 172.21.15.57:55466) 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 -- 172.21.15.57:0/4252721049 learned_addr learned my addr 172.21.15.57:0/4252721049 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128ffff640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 0x7f12901986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f129485e640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129019cfb0 0x7f129019d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 -- 172.21.15.57:0/4252721049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 msgr2=0x7f12901986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 0x7f12901986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 -- 172.21.15.57:0/4252721049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129019cfb0 msgr2=0x7f129019d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129019cfb0 0x7f129019d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 -- 172.21.15.57:0/4252721049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129019daa0 con 0x7f1290106170 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128ffff640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 0x7f12901986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.875+0000 7f128f7fe640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f1290198c30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f128000e9c0 tx=0x7f128000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.876+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128000cd80 con 0x7f1290106170 2024-04-21T08:45:33.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.876+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129019dd90 con 0x7f1290106170 2024-04-21T08:45:33.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.876+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12800054d0 con 0x7f1290106170 2024-04-21T08:45:33.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.876+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1280010430 con 0x7f1290106170 2024-04-21T08:45:33.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.876+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12901a55d0 con 0x7f1290106170 2024-04-21T08:45:33.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.877+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12900691c0 con 0x7f1290106170 2024-04-21T08:45:33.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.878+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f1280004fb0 con 0x7f1290106170 2024-04-21T08:45:33.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.879+0000 7f128d7fa640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f127c0a3db0 0x7f127c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:33.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.879+0000 7f128ffff640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f127c0a3db0 0x7f127c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:33.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.879+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f12800d13f0 con 0x7f1290106170 2024-04-21T08:45:33.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.879+0000 7f128ffff640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f127c0a3db0 0x7f127c0a6270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12780058c0 tx=0x7f1278005850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:33.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:33.882+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1280053ff0 con 0x7f1290106170 2024-04-21T08:45:34.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.069+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1290101a50 con 0x7f1290106170 2024-04-21T08:45:34.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.069+0000 7f128d7fa640 1 -- 172.21.15.57:0/4252721049 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f128009d5c0 con 0x7f1290106170 2024-04-21T08:45:34.069 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:34.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f127c0a3db0 msgr2=0x7f127c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f127c0a3db0 0x7f127c0a6270 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12780058c0 tx=0x7f1278005850 comp rx=0 tx=0).stop 2024-04-21T08:45:34.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 msgr2=0x7f1290198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f1290198c30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f128000e9c0 tx=0x7f128000ee80 comp rx=0 tx=0).stop 2024-04-21T08:45:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 shutdown_connections 2024-04-21T08:45:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f127c0a3db0 0x7f127c0a6270 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f129019cfb0 0x7f129019d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1290106170 0x7f1290198c30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 --2- 172.21.15.57:0/4252721049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12901010a0 0x7f12901986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 >> 172.21.15.57:0/4252721049 conn(0x7f1290077eb0 msgr2=0x7f129006a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:34.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 shutdown_connections 2024-04-21T08:45:34.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.071+0000 7f12962c1640 1 -- 172.21.15.57:0/4252721049 wait complete. 2024-04-21T08:45:34.082 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:34.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/1291373538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101a40 msgr2=0x7f8f74105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/1291373538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101a40 0x7f8f74105a80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8f64009900 tx=0x7f8f6402bc50 comp rx=0 tx=0).stop 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/1291373538 shutdown_connections 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/1291373538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f74106160 0x7f8f7410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/1291373538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101a40 0x7f8f74105a80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/1291373538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f74101090 0x7f8f74101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/1291373538 >> 172.21.15.57:0/1291373538 conn(0x7f8f74077f50 msgr2=0x7f8f74078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/1291373538 shutdown_connections 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/1291373538 wait complete. 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 Processor -- start 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.188+0000 7f8f7b4e4640 1 -- start start 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 0x7f8f74102b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f74106160 0x7f8f74103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f741035a0 0x7f8f741a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74111080 con 0x7f8f74101090 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74110f00 con 0x7f8f74106160 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74111200 con 0x7f8f741035a0 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 0x7f8f74102b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 0x7f8f74102b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55478/0 (socket says 172.21.15.57:55478) 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 -- 172.21.15.57:0/238017442 learned_addr learned my addr 172.21.15.57:0/238017442 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79a81640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f741035a0 0x7f8f741a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f78a7f640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f74106160 0x7f8f74103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 -- 172.21.15.57:0/238017442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f741035a0 msgr2=0x7f8f741a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f741035a0 0x7f8f741a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 -- 172.21.15.57:0/238017442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f74106160 msgr2=0x7f8f74103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f74106160 0x7f8f74103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 -- 172.21.15.57:0/238017442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f741a6fd0 con 0x7f8f74101090 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f79280640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 0x7f8f74102b20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c920 tx=0x7f8f7000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70007c70 con 0x7f8f74101090 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f70007e10 con 0x7f8f74101090 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f741a72c0 con 0x7f8f74101090 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70018820 con 0x7f8f74101090 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.189+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f741a77e0 con 0x7f8f74101090 2024-04-21T08:45:34.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.190+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f74068d70 con 0x7f8f74101090 2024-04-21T08:45:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.192+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f8f7000f450 con 0x7f8f74101090 2024-04-21T08:45:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.192+0000 7f8f6a7fc640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8f600a3de0 0x7f8f600a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.193+0000 7f8f78a7f640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8f600a3de0 0x7f8f600a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.193+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8f700d0eb0 con 0x7f8f74101090 2024-04-21T08:45:34.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.193+0000 7f8f78a7f640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8f600a3de0 0x7f8f600a62a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f64006010 tx=0x7f8f64004040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:34.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.197+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f70014030 con 0x7f8f74101090 2024-04-21T08:45:34.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.381+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8f7410a3a0 con 0x7f8f74101090 2024-04-21T08:45:34.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.382+0000 7f8f6a7fc640 1 -- 172.21.15.57:0/238017442 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f8f7001f300 con 0x7f8f74101090 2024-04-21T08:45:34.384 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8f600a3de0 msgr2=0x7f8f600a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8f600a3de0 0x7f8f600a62a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f64006010 tx=0x7f8f64004040 comp rx=0 tx=0).stop 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 msgr2=0x7f8f74102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 0x7f8f74102b20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c920 tx=0x7f8f7000cde0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 shutdown_connections 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8f600a3de0 0x7f8f600a62a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f741035a0 0x7f8f741a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f74106160 0x7f8f74103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 --2- 172.21.15.57:0/238017442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8f74101090 0x7f8f74102b20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 >> 172.21.15.57:0/238017442 conn(0x7f8f74077f50 msgr2=0x7f8f74100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:34.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 shutdown_connections 2024-04-21T08:45:34.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.384+0000 7f8f7b4e4640 1 -- 172.21.15.57:0/238017442 wait complete. 2024-04-21T08:45:34.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:34.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.500+0000 7f31a4c9a640 1 -- 172.21.15.57:0/355344975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006b1f0 msgr2=0x7f31a0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.500+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/355344975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006b1f0 0x7f31a0106b90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f3194009900 tx=0x7f319402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:34.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.500+0000 7f31a4c9a640 1 -- 172.21.15.57:0/355344975 shutdown_connections 2024-04-21T08:45:34.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.500+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/355344975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31a0107270 0x7f31a010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/355344975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006b1f0 0x7f31a0106b90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/355344975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a006a840 0x7f31a006ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 -- 172.21.15.57:0/355344975 >> 172.21.15.57:0/355344975 conn(0x7f31a0077eb0 msgr2=0x7f31a00782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:34.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 -- 172.21.15.57:0/355344975 shutdown_connections 2024-04-21T08:45:34.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 -- 172.21.15.57:0/355344975 wait complete. 2024-04-21T08:45:34.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 Processor -- start 2024-04-21T08:45:34.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 -- start start 2024-04-21T08:45:34.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 0x7f31a0198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31a0107270 0x7f31a0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.501+0000 7f31a4c9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a019cfe0 0x7f31a019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f31a4c9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0112200 con 0x7f31a006a840 2024-04-21T08:45:34.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f31a4c9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0112080 con 0x7f31a019cfe0 2024-04-21T08:45:34.508 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f31a4c9a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0112380 con 0x7f31a0107270 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 0x7f31a0198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 0x7f31a0198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55490/0 (socket says 172.21.15.57:55490) 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 -- 172.21.15.57:0/3799344842 learned_addr learned my addr 172.21.15.57:0/3799344842 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319dd9b640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31a0107270 0x7f31a0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319ed9d640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a019cfe0 0x7f31a019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 -- 172.21.15.57:0/3799344842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31a0107270 msgr2=0x7f31a0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31a0107270 0x7f31a0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 -- 172.21.15.57:0/3799344842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a019cfe0 msgr2=0x7f31a019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a019cfe0 0x7f31a019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 -- 172.21.15.57:0/3799344842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a019dad0 con 0x7f31a006a840 2024-04-21T08:45:34.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319ed9d640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a019cfe0 0x7f31a019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:34.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f319e59c640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 0x7f31a0198720 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f319000d950 tx=0x7f319000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:34.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190017070 con 0x7f31a006a840 2024-04-21T08:45:34.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3190009c30 con 0x7f31a006a840 2024-04-21T08:45:34.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a019ddc0 con 0x7f31a006a840 2024-04-21T08:45:34.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319000f620 con 0x7f31a006a840 2024-04-21T08:45:34.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.502+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01a55d0 con 0x7f31a006a840 2024-04-21T08:45:34.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.503+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a00691c0 con 0x7f31a006a840 2024-04-21T08:45:34.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.504+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f3190010460 con 0x7f31a006a840 2024-04-21T08:45:34.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.505+0000 7f318f7fe640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31880a3e30 0x7f31880a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.505+0000 7f319dd9b640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31880a3e30 0x7f31880a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.505+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f31900d09d0 con 0x7f31a006a840 2024-04-21T08:45:34.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.505+0000 7f319dd9b640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31880a3e30 0x7f31880a62f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f319402c240 tx=0x7f3194005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:34.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.510+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f319009cb40 con 0x7f31a006a840 2024-04-21T08:45:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:34 smithi136 ceph-mon[31791]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-21T08:45:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:34 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1205625917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:34 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1993520348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:34 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4252721049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:34.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.698+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f31a006b1f0 con 0x7f31a006a840 2024-04-21T08:45:34.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.698+0000 7f318f7fe640 1 -- 172.21.15.57:0/3799344842 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f3190009de0 con 0x7f31a006a840 2024-04-21T08:45:34.698 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31880a3e30 msgr2=0x7f31880a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31880a3e30 0x7f31880a62f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f319402c240 tx=0x7f3194005fd0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 msgr2=0x7f31a0198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 0x7f31a0198720 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f319000d950 tx=0x7f319000de10 comp rx=0 tx=0).stop 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 shutdown_connections 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31880a3e30 0x7f31880a62f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31a019cfe0 0x7f31a019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31a0107270 0x7f31a0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 --2- 172.21.15.57:0/3799344842 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31a006a840 0x7f31a0198720 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 >> 172.21.15.57:0/3799344842 conn(0x7f31a0077eb0 msgr2=0x7f31a0108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 shutdown_connections 2024-04-21T08:45:34.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.700+0000 7f31a4c9a640 1 -- 172.21.15.57:0/3799344842 wait complete. 2024-04-21T08:45:34.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:34 smithi057 ceph-mon[27968]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-21T08:45:34.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:34 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1205625917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:34.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:34 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1993520348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:34.709 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:34 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4252721049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:34.710 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:34.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.815+0000 7fc85d72b640 1 -- 172.21.15.57:0/1219827596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85806b1f0 msgr2=0x7fc858106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1219827596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85806b1f0 0x7fc858106b90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009900 tx=0x7fc84c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 -- 172.21.15.57:0/1219827596 shutdown_connections 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1219827596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc858107270 0x7fc85810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1219827596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc85806b1f0 0x7fc858106b90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1219827596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 0x7fc85806ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 -- 172.21.15.57:0/1219827596 >> 172.21.15.57:0/1219827596 conn(0x7fc858077eb0 msgr2=0x7fc8580782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 -- 172.21.15.57:0/1219827596 shutdown_connections 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 -- 172.21.15.57:0/1219827596 wait complete. 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 Processor -- start 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.816+0000 7fc85d72b640 1 -- start start 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 0x7fc858100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc858107270 0x7fc8581013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 0x7fc8581a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc858112200 con 0x7fc858101900 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc858112080 con 0x7fc858107270 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc858112380 con 0x7fc85806a840 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 0x7fc8581a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 0x7fc8581a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55516/0 (socket says 172.21.15.57:55516) 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc856ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 0x7fc858100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 -- 172.21.15.57:0/1927420105 learned_addr learned my addr 172.21.15.57:0/1927420105 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8567fc640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc858107270 0x7fc8581013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 -- 172.21.15.57:0/1927420105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 msgr2=0x7fc858100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 0x7fc858100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 -- 172.21.15.57:0/1927420105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc858107270 msgr2=0x7fc8581013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc858107270 0x7fc8581013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 -- 172.21.15.57:0/1927420105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8581a7010 con 0x7fc858101900 2024-04-21T08:45:34.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc856ffd640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 0x7fc858100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc8577fe640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 0x7fc8581a6910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc850009fd0 tx=0x7fc85000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850017070 con 0x7fc858101900 2024-04-21T08:45:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc850003e20 con 0x7fc858101900 2024-04-21T08:45:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8581a7300 con 0x7fc858101900 2024-04-21T08:45:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.817+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850010460 con 0x7fc858101900 2024-04-21T08:45:34.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.818+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8581a77d0 con 0x7fc858101900 2024-04-21T08:45:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.818+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8580691c0 con 0x7fc858101900 2024-04-21T08:45:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.820+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fc850003970 con 0x7fc858101900 2024-04-21T08:45:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.820+0000 7fc83ffff640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc8400a3de0 0x7fc8400a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.820+0000 7fc856ffd640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc8400a3de0 0x7fc8400a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.820+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fc8500d08e0 con 0x7fc858101900 2024-04-21T08:45:34.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.821+0000 7fc856ffd640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc8400a3de0 0x7fc8400a62a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc848009fc0 tx=0x7fc848005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:34.824+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc85009c460 con 0x7fc858101900 2024-04-21T08:45:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:34 smithi129 ceph-mon[31272]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-21T08:45:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:34 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1205625917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:34 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1993520348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:34 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4252721049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.011+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc85810a4c0 con 0x7fc858101900 2024-04-21T08:45:35.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.012+0000 7fc83ffff640 1 -- 172.21.15.57:0/1927420105 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fc8500a0760 con 0x7fc858101900 2024-04-21T08:45:35.012 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:35.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc8400a3de0 msgr2=0x7fc8400a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc8400a3de0 0x7fc8400a62a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc848009fc0 tx=0x7fc848005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:35.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 msgr2=0x7fc8581a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 0x7fc8581a6910 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc850009fd0 tx=0x7fc85000b260 comp rx=0 tx=0).stop 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 shutdown_connections 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc8400a3de0 0x7fc8400a62a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc858101900 0x7fc8581a6910 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc858107270 0x7fc8581013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 --2- 172.21.15.57:0/1927420105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc85806a840 0x7fc858100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 >> 172.21.15.57:0/1927420105 conn(0x7fc858077eb0 msgr2=0x7fc858108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 shutdown_connections 2024-04-21T08:45:35.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.014+0000 7fc85d72b640 1 -- 172.21.15.57:0/1927420105 wait complete. 2024-04-21T08:45:35.024 INFO:tasks.quiescer.ceph_manager:waiting for clean 2024-04-21T08:45:35.024 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 -- 172.21.15.57:0/668467970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 msgr2=0x7fb3f4110cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 --2- 172.21.15.57:0/668467970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f4110cf0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009ca0 tx=0x7fb3ec02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 -- 172.21.15.57:0/668467970 shutdown_connections 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 --2- 172.21.15.57:0/668467970 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f4110cf0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 --2- 172.21.15.57:0/668467970 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4101670 0x7fb3f410e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 --2- 172.21.15.57:0/668467970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f4100d50 0x7fb3f4101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 -- 172.21.15.57:0/668467970 >> 172.21.15.57:0/668467970 conn(0x7fb3f40771d0 msgr2=0x7fb3f40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 -- 172.21.15.57:0/668467970 shutdown_connections 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 -- 172.21.15.57:0/668467970 wait complete. 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 Processor -- start 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.126+0000 7fb3fb532640 1 -- start start 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3fb532640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4100d50 0x7fb3f41a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3fb532640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f4101670 0x7fb3f41a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3fb532640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3fb532640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f41143c0 con 0x7fb3f410e900 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3fb532640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f4114240 con 0x7fb3f4101670 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3fb532640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f4114540 con 0x7fb3f4100d50 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55532/0 (socket says 172.21.15.57:55532) 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 -- 172.21.15.57:0/3397520661 learned_addr learned my addr 172.21.15.57:0/3397520661 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:35.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f92ce640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4100d50 0x7fb3f41a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f8acd640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f4101670 0x7fb3f41a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 -- 172.21.15.57:0/3397520661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4100d50 msgr2=0x7fb3f41a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4100d50 0x7fb3f41a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 -- 172.21.15.57:0/3397520661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f4101670 msgr2=0x7fb3f41a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f4101670 0x7fb3f41a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 -- 172.21.15.57:0/3397520661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f419b000 con 0x7fb3f410e900 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f92ce640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4100d50 0x7fb3f41a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3f9acf640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f419a740 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec004af0 tx=0x7fb3ec034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.127+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec039040 con 0x7fb3f410e900 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.128+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3ec044070 con 0x7fb3f410e900 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.128+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f419b290 con 0x7fb3f410e900 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.128+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f41a7420 con 0x7fb3f410e900 2024-04-21T08:45:35.131 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.128+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec003e60 con 0x7fb3f410e900 2024-04-21T08:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.129+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3f406ae60 con 0x7fb3f410e900 2024-04-21T08:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.131+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fb3ec0326c0 con 0x7fb3f410e900 2024-04-21T08:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.131+0000 7fb3ea7fc640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb3e00a3db0 0x7fb3e00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.131+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fb3ec0f3ac0 con 0x7fb3f410e900 2024-04-21T08:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.131+0000 7fb3f92ce640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb3e00a3db0 0x7fb3e00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.132+0000 7fb3f92ce640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb3e00a3db0 0x7fb3e00a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40058c0 tx=0x7fb3e4005850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:35.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.135+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3ec0bfda0 con 0x7fb3f410e900 2024-04-21T08:45:35.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.317+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb3f4062390 con 0x7fb3e00a3db0 2024-04-21T08:45:35.328 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.327+0000 7fb3ea7fc640 1 -- 172.21.15.57:0/3397520661 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339645 (secure 0 0 0) 0x7fb3f4062390 con 0x7fb3e00a3db0 2024-04-21T08:45:35.332 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:35.333 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb3e00a3db0 msgr2=0x7fb3e00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb3e00a3db0 0x7fb3e00a6270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40058c0 tx=0x7fb3e4005850 comp rx=0 tx=0).stop 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 msgr2=0x7fb3f419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f419a740 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec004af0 tx=0x7fb3ec034760 comp rx=0 tx=0).stop 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 shutdown_connections 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb3e00a3db0 0x7fb3e00a6270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb3f410e900 0x7fb3f419a740 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3f4101670 0x7fb3f41a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 --2- 172.21.15.57:0/3397520661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3f4100d50 0x7fb3f41a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 >> 172.21.15.57:0/3397520661 conn(0x7fb3f40771d0 msgr2=0x7fb3f410fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:35.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 shutdown_connections 2024-04-21T08:45:35.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.331+0000 7fb3fb532640 1 -- 172.21.15.57:0/3397520661 wait complete. 2024-04-21T08:45:35.342 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-21T08:45:34.919073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382916,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":713,"num_read_kb":718,"num_write":467,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365976,"kb_used_data":12632,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575848,"statfs":{"total":1151940427776,"available":1151565668352,"internally_reserved":0,"allocated":12935168,"data_stored":10279994,"data_compressed":564019,"data_compressed_allocated":761856,"data_compressed_original":4694016,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":223,"num_read_kb":318,"num_write":84,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.988802"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993678+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:45:24.993678+0000","last_peered":"2024-04-21T08:45:24.993678+0000","last_clean":"2024-04-21T08:45:24.993678+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:45:24.993678+0000","last_undegraded":"2024-04-21T08:45:24.993678+0000","last_fullsized":"2024-04-21T08:45:24.993678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337688+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:45:25.337688+0000","last_peered":"2024-04-21T08:45:25.337688+0000","last_clean":"2024-04-21T08:45:25.337688+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:45:25.337688+0000","last_undegraded":"2024-04-21T08:45:25.337688+0000","last_fullsized":"2024-04-21T08:45:25.337688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338255+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:45:25.338255+0000","last_peered":"2024-04-21T08:45:25.338255+0000","last_clean":"2024-04-21T08:45:25.338255+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:45:25.338255+0000","last_undegraded":"2024-04-21T08:45:25.338255+0000","last_fullsized":"2024-04-21T08:45:25.338255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904383+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:45:24.904383+0000","last_peered":"2024-04-21T08:45:24.904383+0000","last_clean":"2024-04-21T08:45:24.904383+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:45:24.904383+0000","last_undegraded":"2024-04-21T08:45:24.904383+0000","last_fullsized":"2024-04-21T08:45:24.904383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991986+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:45:24.991986+0000","last_peered":"2024-04-21T08:45:24.991986+0000","last_clean":"2024-04-21T08:45:24.991986+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:45:24.991986+0000","last_undegraded":"2024-04-21T08:45:24.991986+0000","last_fullsized":"2024-04-21T08:45:24.991986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992587+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:45:24.992587+0000","last_peered":"2024-04-21T08:45:24.992587+0000","last_clean":"2024-04-21T08:45:24.992587+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:45:24.992587+0000","last_undegraded":"2024-04-21T08:45:24.992587+0000","last_fullsized":"2024-04-21T08:45:24.992587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992192+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:45:24.992192+0000","last_peered":"2024-04-21T08:45:24.992192+0000","last_clean":"2024-04-21T08:45:24.992192+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:45:24.992192+0000","last_undegraded":"2024-04-21T08:45:24.992192+0000","last_fullsized":"2024-04-21T08:45:24.992192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992575+0000","last_change":"2024-04-21T08:44:47.312984+0000","last_active":"2024-04-21T08:45:24.992575+0000","last_peered":"2024-04-21T08:45:24.992575+0000","last_clean":"2024-04-21T08:45:24.992575+0000","last_became_active":"2024-04-21T08:44:47.312465+0000","last_became_peered":"2024-04-21T08:44:47.312465+0000","last_unstale":"2024-04-21T08:45:24.992575+0000","last_undegraded":"2024-04-21T08:45:24.992575+0000","last_fullsized":"2024-04-21T08:45:24.992575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:59:51.214620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993071+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:45:24.993071+0000","last_peered":"2024-04-21T08:45:24.993071+0000","last_clean":"2024-04-21T08:45:24.993071+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:45:24.993071+0000","last_undegraded":"2024-04-21T08:45:24.993071+0000","last_fullsized":"2024-04-21T08:45:24.993071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993708+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:45:24.993708+0000","last_peered":"2024-04-21T08:45:24.993708+0000","last_clean":"2024-04-21T08:45:24.993708+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:45:24.993708+0000","last_undegraded":"2024-04-21T08:45:24.993708+0000","last_fullsized":"2024-04-21T08:45:24.993708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993237+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:45:24.993237+0000","last_peered":"2024-04-21T08:45:24.993237+0000","last_clean":"2024-04-21T08:45:24.993237+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:45:24.993237+0000","last_undegraded":"2024-04-21T08:45:24.993237+0000","last_fullsized":"2024-04-21T08:45:24.993237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904442+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:45:24.904442+0000","last_peered":"2024-04-21T08:45:24.904442+0000","last_clean":"2024-04-21T08:45:24.904442+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:45:24.904442+0000","last_undegraded":"2024-04-21T08:45:24.904442+0000","last_fullsized":"2024-04-21T08:45:24.904442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992519+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:45:24.992519+0000","last_peered":"2024-04-21T08:45:24.992519+0000","last_clean":"2024-04-21T08:45:24.992519+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:45:24.992519+0000","last_undegraded":"2024-04-21T08:45:24.992519+0000","last_fullsized":"2024-04-21T08:45:24.992519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337366+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:45:25.337366+0000","last_peered":"2024-04-21T08:45:25.337366+0000","last_clean":"2024-04-21T08:45:25.337366+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:45:25.337366+0000","last_undegraded":"2024-04-21T08:45:25.337366+0000","last_fullsized":"2024-04-21T08:45:25.337366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337428+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:45:25.337428+0000","last_peered":"2024-04-21T08:45:25.337428+0000","last_clean":"2024-04-21T08:45:25.337428+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:45:25.337428+0000","last_undegraded":"2024-04-21T08:45:25.337428+0000","last_fullsized":"2024-04-21T08:45:25.337428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991935+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:45:24.991935+0000","last_peered":"2024-04-21T08:45:24.991935+0000","last_clean":"2024-04-21T08:45:24.991935+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:45:24.991935+0000","last_undegraded":"2024-04-21T08:45:24.991935+0000","last_fullsized":"2024-04-21T08:45:24.991935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993546+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:45:24.993546+0000","last_peered":"2024-04-21T08:45:24.993546+0000","last_clean":"2024-04-21T08:45:24.993546+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:45:24.993546+0000","last_undegraded":"2024-04-21T08:45:24.993546+0000","last_fullsized":"2024-04-21T08:45:24.993546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993979+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:45:24.993979+0000","last_peered":"2024-04-21T08:45:24.993979+0000","last_clean":"2024-04-21T08:45:24.993979+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:45:24.993979+0000","last_undegraded":"2024-04-21T08:45:24.993979+0000","last_fullsized":"2024-04-21T08:45:24.993979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904038+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:45:24.904038+0000","last_peered":"2024-04-21T08:45:24.904038+0000","last_clean":"2024-04-21T08:45:24.904038+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:45:24.904038+0000","last_undegraded":"2024-04-21T08:45:24.904038+0000","last_fullsized":"2024-04-21T08:45:24.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903663+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:45:24.903663+0000","last_peered":"2024-04-21T08:45:24.903663+0000","last_clean":"2024-04-21T08:45:24.903663+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:45:24.903663+0000","last_undegraded":"2024-04-21T08:45:24.903663+0000","last_fullsized":"2024-04-21T08:45:24.903663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.994306+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:45:24.994306+0000","last_peered":"2024-04-21T08:45:24.994306+0000","last_clean":"2024-04-21T08:45:24.994306+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:45:24.994306+0000","last_undegraded":"2024-04-21T08:45:24.994306+0000","last_fullsized":"2024-04-21T08:45:24.994306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904457+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:45:24.904457+0000","last_peered":"2024-04-21T08:45:24.904457+0000","last_clean":"2024-04-21T08:45:24.904457+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:45:24.904457+0000","last_undegraded":"2024-04-21T08:45:24.904457+0000","last_fullsized":"2024-04-21T08:45:24.904457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904151+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:45:24.904151+0000","last_peered":"2024-04-21T08:45:24.904151+0000","last_clean":"2024-04-21T08:45:24.904151+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:45:24.904151+0000","last_undegraded":"2024-04-21T08:45:24.904151+0000","last_fullsized":"2024-04-21T08:45:24.904151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904639+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:45:24.904639+0000","last_peered":"2024-04-21T08:45:24.904639+0000","last_clean":"2024-04-21T08:45:24.904639+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:45:24.904639+0000","last_undegraded":"2024-04-21T08:45:24.904639+0000","last_fullsized":"2024-04-21T08:45:24.904639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338007+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:45:25.338007+0000","last_peered":"2024-04-21T08:45:25.338007+0000","last_clean":"2024-04-21T08:45:25.338007+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:45:25.338007+0000","last_undegraded":"2024-04-21T08:45:25.338007+0000","last_fullsized":"2024-04-21T08:45:25.338007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904338+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:45:24.904338+0000","last_peered":"2024-04-21T08:45:24.904338+0000","last_clean":"2024-04-21T08:45:24.904338+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:45:24.904338+0000","last_undegraded":"2024-04-21T08:45:24.904338+0000","last_fullsized":"2024-04-21T08:45:24.904338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992440+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:45:24.992440+0000","last_peered":"2024-04-21T08:45:24.992440+0000","last_clean":"2024-04-21T08:45:24.992440+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:45:24.992440+0000","last_undegraded":"2024-04-21T08:45:24.992440+0000","last_fullsized":"2024-04-21T08:45:24.992440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992517+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:45:24.992517+0000","last_peered":"2024-04-21T08:45:24.992517+0000","last_clean":"2024-04-21T08:45:24.992517+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:45:24.992517+0000","last_undegraded":"2024-04-21T08:45:24.992517+0000","last_fullsized":"2024-04-21T08:45:24.992517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904082+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:45:24.904082+0000","last_peered":"2024-04-21T08:45:24.904082+0000","last_clean":"2024-04-21T08:45:24.904082+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:45:24.904082+0000","last_undegraded":"2024-04-21T08:45:24.904082+0000","last_fullsized":"2024-04-21T08:45:24.904082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993055+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:45:24.993055+0000","last_peered":"2024-04-21T08:45:24.993055+0000","last_clean":"2024-04-21T08:45:24.993055+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:45:24.993055+0000","last_undegraded":"2024-04-21T08:45:24.993055+0000","last_fullsized":"2024-04-21T08:45:24.993055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903921+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:45:24.903921+0000","last_peered":"2024-04-21T08:45:24.903921+0000","last_clean":"2024-04-21T08:45:24.903921+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:45:24.903921+0000","last_undegraded":"2024-04-21T08:45:24.903921+0000","last_fullsized":"2024-04-21T08:45:24.903921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993496+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:45:24.993496+0000","last_peered":"2024-04-21T08:45:24.993496+0000","last_clean":"2024-04-21T08:45:24.993496+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:45:24.993496+0000","last_undegraded":"2024-04-21T08:45:24.993496+0000","last_fullsized":"2024-04-21T08:45:24.993496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992410+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:45:24.992410+0000","last_peered":"2024-04-21T08:45:24.992410+0000","last_clean":"2024-04-21T08:45:24.992410+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:45:24.992410+0000","last_undegraded":"2024-04-21T08:45:24.992410+0000","last_fullsized":"2024-04-21T08:45:24.992410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903852+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:45:24.903852+0000","last_peered":"2024-04-21T08:45:24.903852+0000","last_clean":"2024-04-21T08:45:24.903852+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:45:24.903852+0000","last_undegraded":"2024-04-21T08:45:24.903852+0000","last_fullsized":"2024-04-21T08:45:24.903852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992469+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:45:24.992469+0000","last_peered":"2024-04-21T08:45:24.992469+0000","last_clean":"2024-04-21T08:45:24.992469+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:45:24.992469+0000","last_undegraded":"2024-04-21T08:45:24.992469+0000","last_fullsized":"2024-04-21T08:45:24.992469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993353+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:45:24.993353+0000","last_peered":"2024-04-21T08:45:24.993353+0000","last_clean":"2024-04-21T08:45:24.993353+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:45:24.993353+0000","last_undegraded":"2024-04-21T08:45:24.993353+0000","last_fullsized":"2024-04-21T08:45:24.993353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993857+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:45:24.993857+0000","last_peered":"2024-04-21T08:45:24.993857+0000","last_clean":"2024-04-21T08:45:24.993857+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:45:24.993857+0000","last_undegraded":"2024-04-21T08:45:24.993857+0000","last_fullsized":"2024-04-21T08:45:24.993857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992300+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:45:24.992300+0000","last_peered":"2024-04-21T08:45:24.992300+0000","last_clean":"2024-04-21T08:45:24.992300+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:45:24.992300+0000","last_undegraded":"2024-04-21T08:45:24.992300+0000","last_fullsized":"2024-04-21T08:45:24.992300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903970+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:45:24.903970+0000","last_peered":"2024-04-21T08:45:24.903970+0000","last_clean":"2024-04-21T08:45:24.903970+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:45:24.903970+0000","last_undegraded":"2024-04-21T08:45:24.903970+0000","last_fullsized":"2024-04-21T08:45:24.903970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337570+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:45:25.337570+0000","last_peered":"2024-04-21T08:45:25.337570+0000","last_clean":"2024-04-21T08:45:25.337570+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:45:25.337570+0000","last_undegraded":"2024-04-21T08:45:25.337570+0000","last_fullsized":"2024-04-21T08:45:25.337570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992789+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:45:24.992789+0000","last_peered":"2024-04-21T08:45:24.992789+0000","last_clean":"2024-04-21T08:45:24.992789+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:45:24.992789+0000","last_undegraded":"2024-04-21T08:45:24.992789+0000","last_fullsized":"2024-04-21T08:45:24.992789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992907+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:45:24.992907+0000","last_peered":"2024-04-21T08:45:24.992907+0000","last_clean":"2024-04-21T08:45:24.992907+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:45:24.992907+0000","last_undegraded":"2024-04-21T08:45:24.992907+0000","last_fullsized":"2024-04-21T08:45:24.992907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903912+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:45:24.903912+0000","last_peered":"2024-04-21T08:45:24.903912+0000","last_clean":"2024-04-21T08:45:24.903912+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:45:24.903912+0000","last_undegraded":"2024-04-21T08:45:24.903912+0000","last_fullsized":"2024-04-21T08:45:24.903912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993629+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:45:24.993629+0000","last_peered":"2024-04-21T08:45:24.993629+0000","last_clean":"2024-04-21T08:45:24.993629+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:45:24.993629+0000","last_undegraded":"2024-04-21T08:45:24.993629+0000","last_fullsized":"2024-04-21T08:45:24.993629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992855+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:45:24.992855+0000","last_peered":"2024-04-21T08:45:24.992855+0000","last_clean":"2024-04-21T08:45:24.992855+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:45:24.992855+0000","last_undegraded":"2024-04-21T08:45:24.992855+0000","last_fullsized":"2024-04-21T08:45:24.992855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993149+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:45:24.993149+0000","last_peered":"2024-04-21T08:45:24.993149+0000","last_clean":"2024-04-21T08:45:24.993149+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:45:24.993149+0000","last_undegraded":"2024-04-21T08:45:24.993149+0000","last_fullsized":"2024-04-21T08:45:24.993149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337962+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:45:25.337962+0000","last_peered":"2024-04-21T08:45:25.337962+0000","last_clean":"2024-04-21T08:45:25.337962+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:45:25.337962+0000","last_undegraded":"2024-04-21T08:45:25.337962+0000","last_fullsized":"2024-04-21T08:45:25.337962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904574+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:45:24.904574+0000","last_peered":"2024-04-21T08:45:24.904574+0000","last_clean":"2024-04-21T08:45:24.904574+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:45:24.904574+0000","last_undegraded":"2024-04-21T08:45:24.904574+0000","last_fullsized":"2024-04-21T08:45:24.904574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993021+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:45:24.993021+0000","last_peered":"2024-04-21T08:45:24.993021+0000","last_clean":"2024-04-21T08:45:24.993021+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:45:24.993021+0000","last_undegraded":"2024-04-21T08:45:24.993021+0000","last_fullsized":"2024-04-21T08:45:24.993021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992689+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:45:24.992689+0000","last_peered":"2024-04-21T08:45:24.992689+0000","last_clean":"2024-04-21T08:45:24.992689+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:45:24.992689+0000","last_undegraded":"2024-04-21T08:45:24.992689+0000","last_fullsized":"2024-04-21T08:45:24.992689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993192+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:45:24.993192+0000","last_peered":"2024-04-21T08:45:24.993192+0000","last_clean":"2024-04-21T08:45:24.993192+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:45:24.993192+0000","last_undegraded":"2024-04-21T08:45:24.993192+0000","last_fullsized":"2024-04-21T08:45:24.993192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'37","reported_seq":339,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.639527+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:45:30.639527+0000","last_peered":"2024-04-21T08:45:30.639527+0000","last_clean":"2024-04-21T08:45:30.639527+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:45:30.639527+0000","last_undegraded":"2024-04-21T08:45:30.639527+0000","last_fullsized":"2024-04-21T08:45:30.639527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":269,"num_read_kb":204,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992174+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:45:24.992174+0000","last_peered":"2024-04-21T08:45:24.992174+0000","last_clean":"2024-04-21T08:45:24.992174+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:45:24.992174+0000","last_undegraded":"2024-04-21T08:45:24.992174+0000","last_fullsized":"2024-04-21T08:45:24.992174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993309+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:45:24.993309+0000","last_peered":"2024-04-21T08:45:24.993309+0000","last_clean":"2024-04-21T08:45:24.993309+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:45:24.993309+0000","last_undegraded":"2024-04-21T08:45:24.993309+0000","last_fullsized":"2024-04-21T08:45:24.993309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993119+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:45:24.993119+0000","last_peered":"2024-04-21T08:45:24.993119+0000","last_clean":"2024-04-21T08:45:24.993119+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:45:24.993119+0000","last_undegraded":"2024-04-21T08:45:24.993119+0000","last_fullsized":"2024-04-21T08:45:24.993119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992385+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:45:24.992385+0000","last_peered":"2024-04-21T08:45:24.992385+0000","last_clean":"2024-04-21T08:45:24.992385+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:45:24.992385+0000","last_undegraded":"2024-04-21T08:45:24.992385+0000","last_fullsized":"2024-04-21T08:45:24.992385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993270+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:45:24.993270+0000","last_peered":"2024-04-21T08:45:24.993270+0000","last_clean":"2024-04-21T08:45:24.993270+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:45:24.993270+0000","last_undegraded":"2024-04-21T08:45:24.993270+0000","last_fullsized":"2024-04-21T08:45:24.993270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992679+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:45:24.992679+0000","last_peered":"2024-04-21T08:45:24.992679+0000","last_clean":"2024-04-21T08:45:24.992679+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:45:24.992679+0000","last_undegraded":"2024-04-21T08:45:24.992679+0000","last_fullsized":"2024-04-21T08:45:24.992679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992953+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:45:24.992953+0000","last_peered":"2024-04-21T08:45:24.992953+0000","last_clean":"2024-04-21T08:45:24.992953+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:45:24.992953+0000","last_undegraded":"2024-04-21T08:45:24.992953+0000","last_fullsized":"2024-04-21T08:45:24.992953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993986+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:45:24.993986+0000","last_peered":"2024-04-21T08:45:24.993986+0000","last_clean":"2024-04-21T08:45:24.993986+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:45:24.993986+0000","last_undegraded":"2024-04-21T08:45:24.993986+0000","last_fullsized":"2024-04-21T08:45:24.993986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904172+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:45:24.904172+0000","last_peered":"2024-04-21T08:45:24.904172+0000","last_clean":"2024-04-21T08:45:24.904172+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:45:24.904172+0000","last_undegraded":"2024-04-21T08:45:24.904172+0000","last_fullsized":"2024-04-21T08:45:24.904172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992489+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:45:24.992489+0000","last_peered":"2024-04-21T08:45:24.992489+0000","last_clean":"2024-04-21T08:45:24.992489+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:45:24.992489+0000","last_undegraded":"2024-04-21T08:45:24.992489+0000","last_fullsized":"2024-04-21T08:45:24.992489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992860+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:45:24.992860+0000","last_peered":"2024-04-21T08:45:24.992860+0000","last_clean":"2024-04-21T08:45:24.992860+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:45:24.992860+0000","last_undegraded":"2024-04-21T08:45:24.992860+0000","last_fullsized":"2024-04-21T08:45:24.992860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992036+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:45:24.992036+0000","last_peered":"2024-04-21T08:45:24.992036+0000","last_clean":"2024-04-21T08:45:24.992036+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:45:24.992036+0000","last_undegraded":"2024-04-21T08:45:24.992036+0000","last_fullsized":"2024-04-21T08:45:24.992036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992792+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:45:24.992792+0000","last_peered":"2024-04-21T08:45:24.992792+0000","last_clean":"2024-04-21T08:45:24.992792+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:45:24.992792+0000","last_undegraded":"2024-04-21T08:45:24.992792+0000","last_fullsized":"2024-04-21T08:45:24.992792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992835+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:45:24.992835+0000","last_peered":"2024-04-21T08:45:24.992835+0000","last_clean":"2024-04-21T08:45:24.992835+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:45:24.992835+0000","last_undegraded":"2024-04-21T08:45:24.992835+0000","last_fullsized":"2024-04-21T08:45:24.992835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992275+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:45:24.992275+0000","last_peered":"2024-04-21T08:45:24.992275+0000","last_clean":"2024-04-21T08:45:24.992275+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:45:24.992275+0000","last_undegraded":"2024-04-21T08:45:24.992275+0000","last_fullsized":"2024-04-21T08:45:24.992275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993653+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:45:24.993653+0000","last_peered":"2024-04-21T08:45:24.993653+0000","last_clean":"2024-04-21T08:45:24.993653+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:45:24.993653+0000","last_undegraded":"2024-04-21T08:45:24.993653+0000","last_fullsized":"2024-04-21T08:45:24.993653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992241+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:45:24.992241+0000","last_peered":"2024-04-21T08:45:24.992241+0000","last_clean":"2024-04-21T08:45:24.992241+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:45:24.992241+0000","last_undegraded":"2024-04-21T08:45:24.992241+0000","last_fullsized":"2024-04-21T08:45:24.992241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993525+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:45:24.993525+0000","last_peered":"2024-04-21T08:45:24.993525+0000","last_clean":"2024-04-21T08:45:24.993525+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:45:24.993525+0000","last_undegraded":"2024-04-21T08:45:24.993525+0000","last_fullsized":"2024-04-21T08:45:24.993525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993389+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:45:24.993389+0000","last_peered":"2024-04-21T08:45:24.993389+0000","last_clean":"2024-04-21T08:45:24.993389+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:45:24.993389+0000","last_undegraded":"2024-04-21T08:45:24.993389+0000","last_fullsized":"2024-04-21T08:45:24.993389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992746+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:45:24.992746+0000","last_peered":"2024-04-21T08:45:24.992746+0000","last_clean":"2024-04-21T08:45:24.992746+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:45:24.992746+0000","last_undegraded":"2024-04-21T08:45:24.992746+0000","last_fullsized":"2024-04-21T08:45:24.992746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992763+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:45:24.992763+0000","last_peered":"2024-04-21T08:45:24.992763+0000","last_clean":"2024-04-21T08:45:24.992763+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:45:24.992763+0000","last_undegraded":"2024-04-21T08:45:24.992763+0000","last_fullsized":"2024-04-21T08:45:24.992763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992403+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:45:24.992403+0000","last_peered":"2024-04-21T08:45:24.992403+0000","last_clean":"2024-04-21T08:45:24.992403+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:45:24.992403+0000","last_undegraded":"2024-04-21T08:45:24.992403+0000","last_fullsized":"2024-04-21T08:45:24.992403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992847+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:45:24.992847+0000","last_peered":"2024-04-21T08:45:24.992847+0000","last_clean":"2024-04-21T08:45:24.992847+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:45:24.992847+0000","last_undegraded":"2024-04-21T08:45:24.992847+0000","last_fullsized":"2024-04-21T08:45:24.992847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992175+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:45:24.992175+0000","last_peered":"2024-04-21T08:45:24.992175+0000","last_clean":"2024-04-21T08:45:24.992175+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:45:24.992175+0000","last_undegraded":"2024-04-21T08:45:24.992175+0000","last_fullsized":"2024-04-21T08:45:24.992175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993373+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:45:24.993373+0000","last_peered":"2024-04-21T08:45:24.993373+0000","last_clean":"2024-04-21T08:45:24.993373+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:45:24.993373+0000","last_undegraded":"2024-04-21T08:45:24.993373+0000","last_fullsized":"2024-04-21T08:45:24.993373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904114+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:45:24.904114+0000","last_peered":"2024-04-21T08:45:24.904114+0000","last_clean":"2024-04-21T08:45:24.904114+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:45:24.904114+0000","last_undegraded":"2024-04-21T08:45:24.904114+0000","last_fullsized":"2024-04-21T08:45:24.904114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904460+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:45:24.904460+0000","last_peered":"2024-04-21T08:45:24.904460+0000","last_clean":"2024-04-21T08:45:24.904460+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:45:24.904460+0000","last_undegraded":"2024-04-21T08:45:24.904460+0000","last_fullsized":"2024-04-21T08:45:24.904460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":412,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:27.016330+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:45:27.016330+0000","last_peered":"2024-04-21T08:45:27.016330+0000","last_clean":"2024-04-21T08:45:27.016330+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:45:27.016330+0000","last_undegraded":"2024-04-21T08:45:27.016330+0000","last_fullsized":"2024-04-21T08:45:27.016330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992034+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:45:24.992034+0000","last_peered":"2024-04-21T08:45:24.992034+0000","last_clean":"2024-04-21T08:45:24.992034+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:45:24.992034+0000","last_undegraded":"2024-04-21T08:45:24.992034+0000","last_fullsized":"2024-04-21T08:45:24.992034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992367+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:45:24.992367+0000","last_peered":"2024-04-21T08:45:24.992367+0000","last_clean":"2024-04-21T08:45:24.992367+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:45:24.992367+0000","last_undegraded":"2024-04-21T08:45:24.992367+0000","last_fullsized":"2024-04-21T08:45:24.992367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993257+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:45:24.993257+0000","last_peered":"2024-04-21T08:45:24.993257+0000","last_clean":"2024-04-21T08:45:24.993257+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:45:24.993257+0000","last_undegraded":"2024-04-21T08:45:24.993257+0000","last_fullsized":"2024-04-21T08:45:24.993257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992262+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:45:24.992262+0000","last_peered":"2024-04-21T08:45:24.992262+0000","last_clean":"2024-04-21T08:45:24.992262+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:45:24.992262+0000","last_undegraded":"2024-04-21T08:45:24.992262+0000","last_fullsized":"2024-04-21T08:45:24.992262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992695+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:45:24.992695+0000","last_peered":"2024-04-21T08:45:24.992695+0000","last_clean":"2024-04-21T08:45:24.992695+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:45:24.992695+0000","last_undegraded":"2024-04-21T08:45:24.992695+0000","last_fullsized":"2024-04-21T08:45:24.992695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903778+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:45:24.903778+0000","last_peered":"2024-04-21T08:45:24.903778+0000","last_clean":"2024-04-21T08:45:24.903778+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:45:24.903778+0000","last_undegraded":"2024-04-21T08:45:24.903778+0000","last_fullsized":"2024-04-21T08:45:24.903778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'68","reported_seq":154,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.658975+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:45:30.658975+0000","last_peered":"2024-04-21T08:45:30.658975+0000","last_clean":"2024-04-21T08:45:30.658975+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:45:30.658975+0000","last_undegraded":"2024-04-21T08:45:30.658975+0000","last_fullsized":"2024-04-21T08:45:30.658975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":160,"num_write":122,"num_write_kb":1157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904474+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:45:24.904474+0000","last_peered":"2024-04-21T08:45:24.904474+0000","last_clean":"2024-04-21T08:45:24.904474+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:45:24.904474+0000","last_undegraded":"2024-04-21T08:45:24.904474+0000","last_fullsized":"2024-04-21T08:45:24.904474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903473+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:45:24.903473+0000","last_peered":"2024-04-21T08:45:24.903473+0000","last_clean":"2024-04-21T08:45:24.903473+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:45:24.903473+0000","last_undegraded":"2024-04-21T08:45:24.903473+0000","last_fullsized":"2024-04-21T08:45:24.903473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904682+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:45:24.904682+0000","last_peered":"2024-04-21T08:45:24.904682+0000","last_clean":"2024-04-21T08:45:24.904682+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:45:24.904682+0000","last_undegraded":"2024-04-21T08:45:24.904682+0000","last_fullsized":"2024-04-21T08:45:24.904682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337838+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:45:25.337838+0000","last_peered":"2024-04-21T08:45:25.337838+0000","last_clean":"2024-04-21T08:45:25.337838+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:45:25.337838+0000","last_undegraded":"2024-04-21T08:45:25.337838+0000","last_fullsized":"2024-04-21T08:45:25.337838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904527+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:45:24.904527+0000","last_peered":"2024-04-21T08:45:24.904527+0000","last_clean":"2024-04-21T08:45:24.904527+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:45:24.904527+0000","last_undegraded":"2024-04-21T08:45:24.904527+0000","last_fullsized":"2024-04-21T08:45:24.904527+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991872+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:45:24.991872+0000","last_peered":"2024-04-21T08:45:24.991872+0000","last_clean":"2024-04-21T08:45:24.991872+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:45:24.991872+0000","last_undegraded":"2024-04-21T08:45:24.991872+0000","last_fullsized":"2024-04-21T08:45:24.991872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991968+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:45:24.991968+0000","last_peered":"2024-04-21T08:45:24.991968+0000","last_clean":"2024-04-21T08:45:24.991968+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:45:24.991968+0000","last_undegraded":"2024-04-21T08:45:24.991968+0000","last_fullsized":"2024-04-21T08:45:24.991968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992202+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:45:24.992202+0000","last_peered":"2024-04-21T08:45:24.992202+0000","last_clean":"2024-04-21T08:45:24.992202+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:45:24.992202+0000","last_undegraded":"2024-04-21T08:45:24.992202+0000","last_fullsized":"2024-04-21T08:45:24.992202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903893+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:45:24.903893+0000","last_peered":"2024-04-21T08:45:24.903893+0000","last_clean":"2024-04-21T08:45:24.903893+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:45:24.903893+0000","last_undegraded":"2024-04-21T08:45:24.903893+0000","last_fullsized":"2024-04-21T08:45:24.903893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992008+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:45:24.992008+0000","last_peered":"2024-04-21T08:45:24.992008+0000","last_clean":"2024-04-21T08:45:24.992008+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:45:24.992008+0000","last_undegraded":"2024-04-21T08:45:24.992008+0000","last_fullsized":"2024-04-21T08:45:24.992008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904288+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:45:24.904288+0000","last_peered":"2024-04-21T08:45:24.904288+0000","last_clean":"2024-04-21T08:45:24.904288+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:45:24.904288+0000","last_undegraded":"2024-04-21T08:45:24.904288+0000","last_fullsized":"2024-04-21T08:45:24.904288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903726+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:45:24.903726+0000","last_peered":"2024-04-21T08:45:24.903726+0000","last_clean":"2024-04-21T08:45:24.903726+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:45:24.903726+0000","last_undegraded":"2024-04-21T08:45:24.903726+0000","last_fullsized":"2024-04-21T08:45:24.903726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903977+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:45:24.903977+0000","last_peered":"2024-04-21T08:45:24.903977+0000","last_clean":"2024-04-21T08:45:24.903977+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:45:24.903977+0000","last_undegraded":"2024-04-21T08:45:24.903977+0000","last_fullsized":"2024-04-21T08:45:24.903977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904319+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:45:24.904319+0000","last_peered":"2024-04-21T08:45:24.904319+0000","last_clean":"2024-04-21T08:45:24.904319+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:45:24.904319+0000","last_undegraded":"2024-04-21T08:45:24.904319+0000","last_fullsized":"2024-04-21T08:45:24.904319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993372+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:45:24.993372+0000","last_peered":"2024-04-21T08:45:24.993372+0000","last_clean":"2024-04-21T08:45:24.993372+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:45:24.993372+0000","last_undegraded":"2024-04-21T08:45:24.993372+0000","last_fullsized":"2024-04-21T08:45:24.993372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992918+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:45:24.992918+0000","last_peered":"2024-04-21T08:45:24.992918+0000","last_clean":"2024-04-21T08:45:24.992918+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:45:24.992918+0000","last_undegraded":"2024-04-21T08:45:24.992918+0000","last_fullsized":"2024-04-21T08:45:24.992918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992146+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:45:24.992146+0000","last_peered":"2024-04-21T08:45:24.992146+0000","last_clean":"2024-04-21T08:45:24.992146+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:45:24.992146+0000","last_undegraded":"2024-04-21T08:45:24.992146+0000","last_fullsized":"2024-04-21T08:45:24.992146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904004+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:45:24.904004+0000","last_peered":"2024-04-21T08:45:24.904004+0000","last_clean":"2024-04-21T08:45:24.904004+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:45:24.904004+0000","last_undegraded":"2024-04-21T08:45:24.904004+0000","last_fullsized":"2024-04-21T08:45:24.904004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337411+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:45:25.337411+0000","last_peered":"2024-04-21T08:45:25.337411+0000","last_clean":"2024-04-21T08:45:25.337411+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:45:25.337411+0000","last_undegraded":"2024-04-21T08:45:25.337411+0000","last_fullsized":"2024-04-21T08:45:25.337411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'57","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.654353+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:45:30.654353+0000","last_peered":"2024-04-21T08:45:30.654353+0000","last_clean":"2024-04-21T08:45:30.654353+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:45:30.654353+0000","last_undegraded":"2024-04-21T08:45:30.654353+0000","last_fullsized":"2024-04-21T08:45:30.654353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":68,"num_write_kb":807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992998+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:45:24.992998+0000","last_peered":"2024-04-21T08:45:24.992998+0000","last_clean":"2024-04-21T08:45:24.992998+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:45:24.992998+0000","last_undegraded":"2024-04-21T08:45:24.992998+0000","last_fullsized":"2024-04-21T08:45:24.992998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992640+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:45:24.992640+0000","last_peered":"2024-04-21T08:45:24.992640+0000","last_clean":"2024-04-21T08:45:24.992640+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:45:24.992640+0000","last_undegraded":"2024-04-21T08:45:24.992640+0000","last_fullsized":"2024-04-21T08:45:24.992640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992686+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:45:24.992686+0000","last_peered":"2024-04-21T08:45:24.992686+0000","last_clean":"2024-04-21T08:45:24.992686+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:45:24.992686+0000","last_undegraded":"2024-04-21T08:45:24.992686+0000","last_fullsized":"2024-04-21T08:45:24.992686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993176+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:45:24.993176+0000","last_peered":"2024-04-21T08:45:24.993176+0000","last_clean":"2024-04-21T08:45:24.993176+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:45:24.993176+0000","last_undegraded":"2024-04-21T08:45:24.993176+0000","last_fullsized":"2024-04-21T08:45:24.993176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992191+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:45:24.992191+0000","last_peered":"2024-04-21T08:45:24.992191+0000","last_clean":"2024-04-21T08:45:24.992191+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:45:24.992191+0000","last_undegraded":"2024-04-21T08:45:24.992191+0000","last_fullsized":"2024-04-21T08:45:24.992191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903722+0000","last_change":"2024-04-21T08:44:47.315087+0000","last_active":"2024-04-21T08:45:24.903722+0000","last_peered":"2024-04-21T08:45:24.903722+0000","last_clean":"2024-04-21T08:45:24.903722+0000","last_became_active":"2024-04-21T08:44:47.314647+0000","last_became_peered":"2024-04-21T08:44:47.314647+0000","last_unstale":"2024-04-21T08:45:24.903722+0000","last_undegraded":"2024-04-21T08:45:24.903722+0000","last_fullsized":"2024-04-21T08:45:24.903722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:20.425332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903546+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:45:24.903546+0000","last_peered":"2024-04-21T08:45:24.903546+0000","last_clean":"2024-04-21T08:45:24.903546+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:45:24.903546+0000","last_undegraded":"2024-04-21T08:45:24.903546+0000","last_fullsized":"2024-04-21T08:45:24.903546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337673+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:45:25.337673+0000","last_peered":"2024-04-21T08:45:25.337673+0000","last_clean":"2024-04-21T08:45:25.337673+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:45:25.337673+0000","last_undegraded":"2024-04-21T08:45:25.337673+0000","last_fullsized":"2024-04-21T08:45:25.337673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992553+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:45:24.992553+0000","last_peered":"2024-04-21T08:45:24.992553+0000","last_clean":"2024-04-21T08:45:24.992553+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:45:24.992553+0000","last_undegraded":"2024-04-21T08:45:24.992553+0000","last_fullsized":"2024-04-21T08:45:24.992553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904194+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:45:24.904194+0000","last_peered":"2024-04-21T08:45:24.904194+0000","last_clean":"2024-04-21T08:45:24.904194+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:45:24.904194+0000","last_undegraded":"2024-04-21T08:45:24.904194+0000","last_fullsized":"2024-04-21T08:45:24.904194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904716+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:45:24.904716+0000","last_peered":"2024-04-21T08:45:24.904716+0000","last_clean":"2024-04-21T08:45:24.904716+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:45:24.904716+0000","last_undegraded":"2024-04-21T08:45:24.904716+0000","last_fullsized":"2024-04-21T08:45:24.904716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992795+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:45:24.992795+0000","last_peered":"2024-04-21T08:45:24.992795+0000","last_clean":"2024-04-21T08:45:24.992795+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:45:24.992795+0000","last_undegraded":"2024-04-21T08:45:24.992795+0000","last_fullsized":"2024-04-21T08:45:24.992795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903992+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:45:24.903992+0000","last_peered":"2024-04-21T08:45:24.903992+0000","last_clean":"2024-04-21T08:45:24.903992+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:45:24.903992+0000","last_undegraded":"2024-04-21T08:45:24.903992+0000","last_fullsized":"2024-04-21T08:45:24.903992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992360+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:45:24.992360+0000","last_peered":"2024-04-21T08:45:24.992360+0000","last_clean":"2024-04-21T08:45:24.992360+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:45:24.992360+0000","last_undegraded":"2024-04-21T08:45:24.992360+0000","last_fullsized":"2024-04-21T08:45:24.992360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992585+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:45:24.992585+0000","last_peered":"2024-04-21T08:45:24.992585+0000","last_clean":"2024-04-21T08:45:24.992585+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:45:24.992585+0000","last_undegraded":"2024-04-21T08:45:24.992585+0000","last_fullsized":"2024-04-21T08:45:24.992585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993390+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:45:24.993390+0000","last_peered":"2024-04-21T08:45:24.993390+0000","last_clean":"2024-04-21T08:45:24.993390+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:45:24.993390+0000","last_undegraded":"2024-04-21T08:45:24.993390+0000","last_fullsized":"2024-04-21T08:45:24.993390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":110016,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":792314,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":476,"num_read_kb":388,"num_write":264,"num_write_kb":2062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":2376942,"data_compressed":107229,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528781,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1316881,"data_compressed":74099,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":926641,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":922567,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1123793,"data_compressed":66315,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495687,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":524651,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":524741,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1246203,"data_compressed":63758,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":528769,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1447529,"data_compressed":81155,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":721552,"data_compressed":34115,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":328414,"data_compressed":14840,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":398494,"data_compressed":16468,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:45:35.348 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 -- 172.21.15.57:0/1305822997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 msgr2=0x7f4bd810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1305822997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd810e9d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009ca0 tx=0x7f4bd002c180 comp rx=0 tx=0).stop 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 -- 172.21.15.57:0/1305822997 shutdown_connections 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1305822997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd810e9d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1305822997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd8103af0 0x7f4bd810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1305822997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bd81031d0 0x7f4bd81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 -- 172.21.15.57:0/1305822997 >> 172.21.15.57:0/1305822997 conn(0x7f4bd80771d0 msgr2=0x7f4bd80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 -- 172.21.15.57:0/1305822997 shutdown_connections 2024-04-21T08:45:35.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.483+0000 7f4bddf67640 1 -- 172.21.15.57:0/1305822997 wait complete. 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 Processor -- start 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 -- start start 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd81031d0 0x7f4bd8198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bd8103af0 0x7f4bd8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd81121a0 con 0x7f4bd810c5a0 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd8112020 con 0x7f4bd81031d0 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bddf67640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd8112320 con 0x7f4bd8103af0 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55546/0 (socket says 172.21.15.57:55546) 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bd7fff640 1 -- 172.21.15.57:0/1437656073 learned_addr learned my addr 172.21.15.57:0/1437656073 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bd77fe640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd81031d0 0x7f4bd8198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.484+0000 7f4bd6ffd640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bd8103af0 0x7f4bd8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd7fff640 1 -- 172.21.15.57:0/1437656073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bd8103af0 msgr2=0x7f4bd8198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd7fff640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bd8103af0 0x7f4bd8198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd7fff640 1 -- 172.21.15.57:0/1437656073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd81031d0 msgr2=0x7f4bd8198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd7fff640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd81031d0 0x7f4bd8198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd7fff640 1 -- 172.21.15.57:0/1437656073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd819d620 con 0x7f4bd810c5a0 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd77fe640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd81031d0 0x7f4bd8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd7fff640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd819cef0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4bd002c790 tx=0x7f4bd0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0039040 con 0x7f4bd810c5a0 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bd0044070 con 0x7f4bd810c5a0 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd819d8b0 con 0x7f4bd810c5a0 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00327c0 con 0x7f4bd810c5a0 2024-04-21T08:45:35.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.485+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd81a51a0 con 0x7f4bd810c5a0 2024-04-21T08:45:35.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.486+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd8069230 con 0x7f4bd810c5a0 2024-04-21T08:45:35.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.487+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f4bd0004040 con 0x7f4bd810c5a0 2024-04-21T08:45:35.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.488+0000 7f4bd4ff9640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4bc00a3e30 0x7f4bc00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.488+0000 7f4bd77fe640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4bc00a3e30 0x7f4bc00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.488+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f4bd00f3140 con 0x7f4bd810c5a0 2024-04-21T08:45:35.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.488+0000 7f4bd77fe640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4bc00a3e30 0x7f4bc00a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800abe0 tx=0x7f4bc800a5c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:35.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.492+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4bd00bf420 con 0x7f4bd810c5a0 2024-04-21T08:45:35.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:35 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/238017442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:35 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3799344842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:35 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1927420105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.675+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bd8062390 con 0x7f4bc00a3e30 2024-04-21T08:45:35.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.684+0000 7f4bd4ff9640 1 -- 172.21.15.57:0/1437656073 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339645 (secure 0 0 0) 0x7f4bd8062390 con 0x7f4bc00a3e30 2024-04-21T08:45:35.688 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:35.690 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4bc00a3e30 msgr2=0x7f4bc00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4bc00a3e30 0x7f4bc00a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800abe0 tx=0x7f4bc800a5c0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 msgr2=0x7f4bd819cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd819cef0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f4bd002c790 tx=0x7f4bd0003ba0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 shutdown_connections 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4bc00a3e30 0x7f4bc00a62f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4bd810c5a0 0x7f4bd819cef0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bd8103af0 0x7f4bd8198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 --2- 172.21.15.57:0/1437656073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bd81031d0 0x7f4bd8198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 >> 172.21.15.57:0/1437656073 conn(0x7f4bd80771d0 msgr2=0x7f4bd80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 shutdown_connections 2024-04-21T08:45:35.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.688+0000 7f4bddf67640 1 -- 172.21.15.57:0/1437656073 wait complete. 2024-04-21T08:45:35.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:35 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/238017442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:35 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3799344842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.698 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:35 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1927420105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.699 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-21T08:45:34.919073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382916,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":713,"num_read_kb":718,"num_write":467,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365976,"kb_used_data":12632,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575848,"statfs":{"total":1151940427776,"available":1151565668352,"internally_reserved":0,"allocated":12935168,"data_stored":10279994,"data_compressed":564019,"data_compressed_allocated":761856,"data_compressed_original":4694016,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":223,"num_read_kb":318,"num_write":84,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.988802"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993678+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:45:24.993678+0000","last_peered":"2024-04-21T08:45:24.993678+0000","last_clean":"2024-04-21T08:45:24.993678+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:45:24.993678+0000","last_undegraded":"2024-04-21T08:45:24.993678+0000","last_fullsized":"2024-04-21T08:45:24.993678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337688+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:45:25.337688+0000","last_peered":"2024-04-21T08:45:25.337688+0000","last_clean":"2024-04-21T08:45:25.337688+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:45:25.337688+0000","last_undegraded":"2024-04-21T08:45:25.337688+0000","last_fullsized":"2024-04-21T08:45:25.337688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338255+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:45:25.338255+0000","last_peered":"2024-04-21T08:45:25.338255+0000","last_clean":"2024-04-21T08:45:25.338255+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:45:25.338255+0000","last_undegraded":"2024-04-21T08:45:25.338255+0000","last_fullsized":"2024-04-21T08:45:25.338255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904383+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:45:24.904383+0000","last_peered":"2024-04-21T08:45:24.904383+0000","last_clean":"2024-04-21T08:45:24.904383+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:45:24.904383+0000","last_undegraded":"2024-04-21T08:45:24.904383+0000","last_fullsized":"2024-04-21T08:45:24.904383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991986+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:45:24.991986+0000","last_peered":"2024-04-21T08:45:24.991986+0000","last_clean":"2024-04-21T08:45:24.991986+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:45:24.991986+0000","last_undegraded":"2024-04-21T08:45:24.991986+0000","last_fullsized":"2024-04-21T08:45:24.991986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992587+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:45:24.992587+0000","last_peered":"2024-04-21T08:45:24.992587+0000","last_clean":"2024-04-21T08:45:24.992587+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:45:24.992587+0000","last_undegraded":"2024-04-21T08:45:24.992587+0000","last_fullsized":"2024-04-21T08:45:24.992587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992192+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:45:24.992192+0000","last_peered":"2024-04-21T08:45:24.992192+0000","last_clean":"2024-04-21T08:45:24.992192+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:45:24.992192+0000","last_undegraded":"2024-04-21T08:45:24.992192+0000","last_fullsized":"2024-04-21T08:45:24.992192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992575+0000","last_change":"2024-04-21T08:44:47.312984+0000","last_active":"2024-04-21T08:45:24.992575+0000","last_peered":"2024-04-21T08:45:24.992575+0000","last_clean":"2024-04-21T08:45:24.992575+0000","last_became_active":"2024-04-21T08:44:47.312465+0000","last_became_peered":"2024-04-21T08:44:47.312465+0000","last_unstale":"2024-04-21T08:45:24.992575+0000","last_undegraded":"2024-04-21T08:45:24.992575+0000","last_fullsized":"2024-04-21T08:45:24.992575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:59:51.214620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993071+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:45:24.993071+0000","last_peered":"2024-04-21T08:45:24.993071+0000","last_clean":"2024-04-21T08:45:24.993071+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:45:24.993071+0000","last_undegraded":"2024-04-21T08:45:24.993071+0000","last_fullsized":"2024-04-21T08:45:24.993071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993708+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:45:24.993708+0000","last_peered":"2024-04-21T08:45:24.993708+0000","last_clean":"2024-04-21T08:45:24.993708+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:45:24.993708+0000","last_undegraded":"2024-04-21T08:45:24.993708+0000","last_fullsized":"2024-04-21T08:45:24.993708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993237+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:45:24.993237+0000","last_peered":"2024-04-21T08:45:24.993237+0000","last_clean":"2024-04-21T08:45:24.993237+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:45:24.993237+0000","last_undegraded":"2024-04-21T08:45:24.993237+0000","last_fullsized":"2024-04-21T08:45:24.993237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904442+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:45:24.904442+0000","last_peered":"2024-04-21T08:45:24.904442+0000","last_clean":"2024-04-21T08:45:24.904442+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:45:24.904442+0000","last_undegraded":"2024-04-21T08:45:24.904442+0000","last_fullsized":"2024-04-21T08:45:24.904442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992519+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:45:24.992519+0000","last_peered":"2024-04-21T08:45:24.992519+0000","last_clean":"2024-04-21T08:45:24.992519+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:45:24.992519+0000","last_undegraded":"2024-04-21T08:45:24.992519+0000","last_fullsized":"2024-04-21T08:45:24.992519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337366+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:45:25.337366+0000","last_peered":"2024-04-21T08:45:25.337366+0000","last_clean":"2024-04-21T08:45:25.337366+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:45:25.337366+0000","last_undegraded":"2024-04-21T08:45:25.337366+0000","last_fullsized":"2024-04-21T08:45:25.337366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337428+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:45:25.337428+0000","last_peered":"2024-04-21T08:45:25.337428+0000","last_clean":"2024-04-21T08:45:25.337428+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:45:25.337428+0000","last_undegraded":"2024-04-21T08:45:25.337428+0000","last_fullsized":"2024-04-21T08:45:25.337428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991935+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:45:24.991935+0000","last_peered":"2024-04-21T08:45:24.991935+0000","last_clean":"2024-04-21T08:45:24.991935+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:45:24.991935+0000","last_undegraded":"2024-04-21T08:45:24.991935+0000","last_fullsized":"2024-04-21T08:45:24.991935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993546+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:45:24.993546+0000","last_peered":"2024-04-21T08:45:24.993546+0000","last_clean":"2024-04-21T08:45:24.993546+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:45:24.993546+0000","last_undegraded":"2024-04-21T08:45:24.993546+0000","last_fullsized":"2024-04-21T08:45:24.993546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993979+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:45:24.993979+0000","last_peered":"2024-04-21T08:45:24.993979+0000","last_clean":"2024-04-21T08:45:24.993979+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:45:24.993979+0000","last_undegraded":"2024-04-21T08:45:24.993979+0000","last_fullsized":"2024-04-21T08:45:24.993979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904038+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:45:24.904038+0000","last_peered":"2024-04-21T08:45:24.904038+0000","last_clean":"2024-04-21T08:45:24.904038+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:45:24.904038+0000","last_undegraded":"2024-04-21T08:45:24.904038+0000","last_fullsized":"2024-04-21T08:45:24.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903663+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:45:24.903663+0000","last_peered":"2024-04-21T08:45:24.903663+0000","last_clean":"2024-04-21T08:45:24.903663+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:45:24.903663+0000","last_undegraded":"2024-04-21T08:45:24.903663+0000","last_fullsized":"2024-04-21T08:45:24.903663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.994306+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:45:24.994306+0000","last_peered":"2024-04-21T08:45:24.994306+0000","last_clean":"2024-04-21T08:45:24.994306+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:45:24.994306+0000","last_undegraded":"2024-04-21T08:45:24.994306+0000","last_fullsized":"2024-04-21T08:45:24.994306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904457+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:45:24.904457+0000","last_peered":"2024-04-21T08:45:24.904457+0000","last_clean":"2024-04-21T08:45:24.904457+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:45:24.904457+0000","last_undegraded":"2024-04-21T08:45:24.904457+0000","last_fullsized":"2024-04-21T08:45:24.904457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904151+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:45:24.904151+0000","last_peered":"2024-04-21T08:45:24.904151+0000","last_clean":"2024-04-21T08:45:24.904151+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:45:24.904151+0000","last_undegraded":"2024-04-21T08:45:24.904151+0000","last_fullsized":"2024-04-21T08:45:24.904151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904639+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:45:24.904639+0000","last_peered":"2024-04-21T08:45:24.904639+0000","last_clean":"2024-04-21T08:45:24.904639+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:45:24.904639+0000","last_undegraded":"2024-04-21T08:45:24.904639+0000","last_fullsized":"2024-04-21T08:45:24.904639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338007+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:45:25.338007+0000","last_peered":"2024-04-21T08:45:25.338007+0000","last_clean":"2024-04-21T08:45:25.338007+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:45:25.338007+0000","last_undegraded":"2024-04-21T08:45:25.338007+0000","last_fullsized":"2024-04-21T08:45:25.338007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904338+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:45:24.904338+0000","last_peered":"2024-04-21T08:45:24.904338+0000","last_clean":"2024-04-21T08:45:24.904338+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:45:24.904338+0000","last_undegraded":"2024-04-21T08:45:24.904338+0000","last_fullsized":"2024-04-21T08:45:24.904338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992440+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:45:24.992440+0000","last_peered":"2024-04-21T08:45:24.992440+0000","last_clean":"2024-04-21T08:45:24.992440+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:45:24.992440+0000","last_undegraded":"2024-04-21T08:45:24.992440+0000","last_fullsized":"2024-04-21T08:45:24.992440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992517+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:45:24.992517+0000","last_peered":"2024-04-21T08:45:24.992517+0000","last_clean":"2024-04-21T08:45:24.992517+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:45:24.992517+0000","last_undegraded":"2024-04-21T08:45:24.992517+0000","last_fullsized":"2024-04-21T08:45:24.992517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904082+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:45:24.904082+0000","last_peered":"2024-04-21T08:45:24.904082+0000","last_clean":"2024-04-21T08:45:24.904082+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:45:24.904082+0000","last_undegraded":"2024-04-21T08:45:24.904082+0000","last_fullsized":"2024-04-21T08:45:24.904082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993055+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:45:24.993055+0000","last_peered":"2024-04-21T08:45:24.993055+0000","last_clean":"2024-04-21T08:45:24.993055+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:45:24.993055+0000","last_undegraded":"2024-04-21T08:45:24.993055+0000","last_fullsized":"2024-04-21T08:45:24.993055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903921+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:45:24.903921+0000","last_peered":"2024-04-21T08:45:24.903921+0000","last_clean":"2024-04-21T08:45:24.903921+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:45:24.903921+0000","last_undegraded":"2024-04-21T08:45:24.903921+0000","last_fullsized":"2024-04-21T08:45:24.903921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993496+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:45:24.993496+0000","last_peered":"2024-04-21T08:45:24.993496+0000","last_clean":"2024-04-21T08:45:24.993496+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:45:24.993496+0000","last_undegraded":"2024-04-21T08:45:24.993496+0000","last_fullsized":"2024-04-21T08:45:24.993496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992410+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:45:24.992410+0000","last_peered":"2024-04-21T08:45:24.992410+0000","last_clean":"2024-04-21T08:45:24.992410+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:45:24.992410+0000","last_undegraded":"2024-04-21T08:45:24.992410+0000","last_fullsized":"2024-04-21T08:45:24.992410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903852+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:45:24.903852+0000","last_peered":"2024-04-21T08:45:24.903852+0000","last_clean":"2024-04-21T08:45:24.903852+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:45:24.903852+0000","last_undegraded":"2024-04-21T08:45:24.903852+0000","last_fullsized":"2024-04-21T08:45:24.903852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992469+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:45:24.992469+0000","last_peered":"2024-04-21T08:45:24.992469+0000","last_clean":"2024-04-21T08:45:24.992469+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:45:24.992469+0000","last_undegraded":"2024-04-21T08:45:24.992469+0000","last_fullsized":"2024-04-21T08:45:24.992469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993353+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:45:24.993353+0000","last_peered":"2024-04-21T08:45:24.993353+0000","last_clean":"2024-04-21T08:45:24.993353+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:45:24.993353+0000","last_undegraded":"2024-04-21T08:45:24.993353+0000","last_fullsized":"2024-04-21T08:45:24.993353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993857+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:45:24.993857+0000","last_peered":"2024-04-21T08:45:24.993857+0000","last_clean":"2024-04-21T08:45:24.993857+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:45:24.993857+0000","last_undegraded":"2024-04-21T08:45:24.993857+0000","last_fullsized":"2024-04-21T08:45:24.993857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992300+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:45:24.992300+0000","last_peered":"2024-04-21T08:45:24.992300+0000","last_clean":"2024-04-21T08:45:24.992300+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:45:24.992300+0000","last_undegraded":"2024-04-21T08:45:24.992300+0000","last_fullsized":"2024-04-21T08:45:24.992300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903970+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:45:24.903970+0000","last_peered":"2024-04-21T08:45:24.903970+0000","last_clean":"2024-04-21T08:45:24.903970+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:45:24.903970+0000","last_undegraded":"2024-04-21T08:45:24.903970+0000","last_fullsized":"2024-04-21T08:45:24.903970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337570+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:45:25.337570+0000","last_peered":"2024-04-21T08:45:25.337570+0000","last_clean":"2024-04-21T08:45:25.337570+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:45:25.337570+0000","last_undegraded":"2024-04-21T08:45:25.337570+0000","last_fullsized":"2024-04-21T08:45:25.337570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992789+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:45:24.992789+0000","last_peered":"2024-04-21T08:45:24.992789+0000","last_clean":"2024-04-21T08:45:24.992789+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:45:24.992789+0000","last_undegraded":"2024-04-21T08:45:24.992789+0000","last_fullsized":"2024-04-21T08:45:24.992789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992907+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:45:24.992907+0000","last_peered":"2024-04-21T08:45:24.992907+0000","last_clean":"2024-04-21T08:45:24.992907+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:45:24.992907+0000","last_undegraded":"2024-04-21T08:45:24.992907+0000","last_fullsized":"2024-04-21T08:45:24.992907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903912+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:45:24.903912+0000","last_peered":"2024-04-21T08:45:24.903912+0000","last_clean":"2024-04-21T08:45:24.903912+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:45:24.903912+0000","last_undegraded":"2024-04-21T08:45:24.903912+0000","last_fullsized":"2024-04-21T08:45:24.903912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993629+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:45:24.993629+0000","last_peered":"2024-04-21T08:45:24.993629+0000","last_clean":"2024-04-21T08:45:24.993629+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:45:24.993629+0000","last_undegraded":"2024-04-21T08:45:24.993629+0000","last_fullsized":"2024-04-21T08:45:24.993629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992855+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:45:24.992855+0000","last_peered":"2024-04-21T08:45:24.992855+0000","last_clean":"2024-04-21T08:45:24.992855+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:45:24.992855+0000","last_undegraded":"2024-04-21T08:45:24.992855+0000","last_fullsized":"2024-04-21T08:45:24.992855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993149+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:45:24.993149+0000","last_peered":"2024-04-21T08:45:24.993149+0000","last_clean":"2024-04-21T08:45:24.993149+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:45:24.993149+0000","last_undegraded":"2024-04-21T08:45:24.993149+0000","last_fullsized":"2024-04-21T08:45:24.993149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337962+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:45:25.337962+0000","last_peered":"2024-04-21T08:45:25.337962+0000","last_clean":"2024-04-21T08:45:25.337962+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:45:25.337962+0000","last_undegraded":"2024-04-21T08:45:25.337962+0000","last_fullsized":"2024-04-21T08:45:25.337962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904574+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:45:24.904574+0000","last_peered":"2024-04-21T08:45:24.904574+0000","last_clean":"2024-04-21T08:45:24.904574+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:45:24.904574+0000","last_undegraded":"2024-04-21T08:45:24.904574+0000","last_fullsized":"2024-04-21T08:45:24.904574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993021+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:45:24.993021+0000","last_peered":"2024-04-21T08:45:24.993021+0000","last_clean":"2024-04-21T08:45:24.993021+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:45:24.993021+0000","last_undegraded":"2024-04-21T08:45:24.993021+0000","last_fullsized":"2024-04-21T08:45:24.993021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992689+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:45:24.992689+0000","last_peered":"2024-04-21T08:45:24.992689+0000","last_clean":"2024-04-21T08:45:24.992689+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:45:24.992689+0000","last_undegraded":"2024-04-21T08:45:24.992689+0000","last_fullsized":"2024-04-21T08:45:24.992689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993192+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:45:24.993192+0000","last_peered":"2024-04-21T08:45:24.993192+0000","last_clean":"2024-04-21T08:45:24.993192+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:45:24.993192+0000","last_undegraded":"2024-04-21T08:45:24.993192+0000","last_fullsized":"2024-04-21T08:45:24.993192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'37","reported_seq":339,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.639527+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:45:30.639527+0000","last_peered":"2024-04-21T08:45:30.639527+0000","last_clean":"2024-04-21T08:45:30.639527+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:45:30.639527+0000","last_undegraded":"2024-04-21T08:45:30.639527+0000","last_fullsized":"2024-04-21T08:45:30.639527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":269,"num_read_kb":204,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992174+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:45:24.992174+0000","last_peered":"2024-04-21T08:45:24.992174+0000","last_clean":"2024-04-21T08:45:24.992174+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:45:24.992174+0000","last_undegraded":"2024-04-21T08:45:24.992174+0000","last_fullsized":"2024-04-21T08:45:24.992174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993309+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:45:24.993309+0000","last_peered":"2024-04-21T08:45:24.993309+0000","last_clean":"2024-04-21T08:45:24.993309+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:45:24.993309+0000","last_undegraded":"2024-04-21T08:45:24.993309+0000","last_fullsized":"2024-04-21T08:45:24.993309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993119+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:45:24.993119+0000","last_peered":"2024-04-21T08:45:24.993119+0000","last_clean":"2024-04-21T08:45:24.993119+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:45:24.993119+0000","last_undegraded":"2024-04-21T08:45:24.993119+0000","last_fullsized":"2024-04-21T08:45:24.993119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992385+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:45:24.992385+0000","last_peered":"2024-04-21T08:45:24.992385+0000","last_clean":"2024-04-21T08:45:24.992385+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:45:24.992385+0000","last_undegraded":"2024-04-21T08:45:24.992385+0000","last_fullsized":"2024-04-21T08:45:24.992385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993270+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:45:24.993270+0000","last_peered":"2024-04-21T08:45:24.993270+0000","last_clean":"2024-04-21T08:45:24.993270+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:45:24.993270+0000","last_undegraded":"2024-04-21T08:45:24.993270+0000","last_fullsized":"2024-04-21T08:45:24.993270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992679+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:45:24.992679+0000","last_peered":"2024-04-21T08:45:24.992679+0000","last_clean":"2024-04-21T08:45:24.992679+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:45:24.992679+0000","last_undegraded":"2024-04-21T08:45:24.992679+0000","last_fullsized":"2024-04-21T08:45:24.992679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992953+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:45:24.992953+0000","last_peered":"2024-04-21T08:45:24.992953+0000","last_clean":"2024-04-21T08:45:24.992953+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:45:24.992953+0000","last_undegraded":"2024-04-21T08:45:24.992953+0000","last_fullsized":"2024-04-21T08:45:24.992953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993986+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:45:24.993986+0000","last_peered":"2024-04-21T08:45:24.993986+0000","last_clean":"2024-04-21T08:45:24.993986+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:45:24.993986+0000","last_undegraded":"2024-04-21T08:45:24.993986+0000","last_fullsized":"2024-04-21T08:45:24.993986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904172+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:45:24.904172+0000","last_peered":"2024-04-21T08:45:24.904172+0000","last_clean":"2024-04-21T08:45:24.904172+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:45:24.904172+0000","last_undegraded":"2024-04-21T08:45:24.904172+0000","last_fullsized":"2024-04-21T08:45:24.904172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992489+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:45:24.992489+0000","last_peered":"2024-04-21T08:45:24.992489+0000","last_clean":"2024-04-21T08:45:24.992489+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:45:24.992489+0000","last_undegraded":"2024-04-21T08:45:24.992489+0000","last_fullsized":"2024-04-21T08:45:24.992489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992860+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:45:24.992860+0000","last_peered":"2024-04-21T08:45:24.992860+0000","last_clean":"2024-04-21T08:45:24.992860+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:45:24.992860+0000","last_undegraded":"2024-04-21T08:45:24.992860+0000","last_fullsized":"2024-04-21T08:45:24.992860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992036+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:45:24.992036+0000","last_peered":"2024-04-21T08:45:24.992036+0000","last_clean":"2024-04-21T08:45:24.992036+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:45:24.992036+0000","last_undegraded":"2024-04-21T08:45:24.992036+0000","last_fullsized":"2024-04-21T08:45:24.992036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992792+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:45:24.992792+0000","last_peered":"2024-04-21T08:45:24.992792+0000","last_clean":"2024-04-21T08:45:24.992792+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:45:24.992792+0000","last_undegraded":"2024-04-21T08:45:24.992792+0000","last_fullsized":"2024-04-21T08:45:24.992792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992835+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:45:24.992835+0000","last_peered":"2024-04-21T08:45:24.992835+0000","last_clean":"2024-04-21T08:45:24.992835+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:45:24.992835+0000","last_undegraded":"2024-04-21T08:45:24.992835+0000","last_fullsized":"2024-04-21T08:45:24.992835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992275+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:45:24.992275+0000","last_peered":"2024-04-21T08:45:24.992275+0000","last_clean":"2024-04-21T08:45:24.992275+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:45:24.992275+0000","last_undegraded":"2024-04-21T08:45:24.992275+0000","last_fullsized":"2024-04-21T08:45:24.992275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993653+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:45:24.993653+0000","last_peered":"2024-04-21T08:45:24.993653+0000","last_clean":"2024-04-21T08:45:24.993653+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:45:24.993653+0000","last_undegraded":"2024-04-21T08:45:24.993653+0000","last_fullsized":"2024-04-21T08:45:24.993653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992241+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:45:24.992241+0000","last_peered":"2024-04-21T08:45:24.992241+0000","last_clean":"2024-04-21T08:45:24.992241+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:45:24.992241+0000","last_undegraded":"2024-04-21T08:45:24.992241+0000","last_fullsized":"2024-04-21T08:45:24.992241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993525+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:45:24.993525+0000","last_peered":"2024-04-21T08:45:24.993525+0000","last_clean":"2024-04-21T08:45:24.993525+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:45:24.993525+0000","last_undegraded":"2024-04-21T08:45:24.993525+0000","last_fullsized":"2024-04-21T08:45:24.993525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993389+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:45:24.993389+0000","last_peered":"2024-04-21T08:45:24.993389+0000","last_clean":"2024-04-21T08:45:24.993389+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:45:24.993389+0000","last_undegraded":"2024-04-21T08:45:24.993389+0000","last_fullsized":"2024-04-21T08:45:24.993389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992746+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:45:24.992746+0000","last_peered":"2024-04-21T08:45:24.992746+0000","last_clean":"2024-04-21T08:45:24.992746+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:45:24.992746+0000","last_undegraded":"2024-04-21T08:45:24.992746+0000","last_fullsized":"2024-04-21T08:45:24.992746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992763+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:45:24.992763+0000","last_peered":"2024-04-21T08:45:24.992763+0000","last_clean":"2024-04-21T08:45:24.992763+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:45:24.992763+0000","last_undegraded":"2024-04-21T08:45:24.992763+0000","last_fullsized":"2024-04-21T08:45:24.992763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992403+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:45:24.992403+0000","last_peered":"2024-04-21T08:45:24.992403+0000","last_clean":"2024-04-21T08:45:24.992403+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:45:24.992403+0000","last_undegraded":"2024-04-21T08:45:24.992403+0000","last_fullsized":"2024-04-21T08:45:24.992403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992847+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:45:24.992847+0000","last_peered":"2024-04-21T08:45:24.992847+0000","last_clean":"2024-04-21T08:45:24.992847+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:45:24.992847+0000","last_undegraded":"2024-04-21T08:45:24.992847+0000","last_fullsized":"2024-04-21T08:45:24.992847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992175+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:45:24.992175+0000","last_peered":"2024-04-21T08:45:24.992175+0000","last_clean":"2024-04-21T08:45:24.992175+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:45:24.992175+0000","last_undegraded":"2024-04-21T08:45:24.992175+0000","last_fullsized":"2024-04-21T08:45:24.992175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993373+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:45:24.993373+0000","last_peered":"2024-04-21T08:45:24.993373+0000","last_clean":"2024-04-21T08:45:24.993373+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:45:24.993373+0000","last_undegraded":"2024-04-21T08:45:24.993373+0000","last_fullsized":"2024-04-21T08:45:24.993373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904114+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:45:24.904114+0000","last_peered":"2024-04-21T08:45:24.904114+0000","last_clean":"2024-04-21T08:45:24.904114+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:45:24.904114+0000","last_undegraded":"2024-04-21T08:45:24.904114+0000","last_fullsized":"2024-04-21T08:45:24.904114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904460+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:45:24.904460+0000","last_peered":"2024-04-21T08:45:24.904460+0000","last_clean":"2024-04-21T08:45:24.904460+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:45:24.904460+0000","last_undegraded":"2024-04-21T08:45:24.904460+0000","last_fullsized":"2024-04-21T08:45:24.904460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":412,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:27.016330+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:45:27.016330+0000","last_peered":"2024-04-21T08:45:27.016330+0000","last_clean":"2024-04-21T08:45:27.016330+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:45:27.016330+0000","last_undegraded":"2024-04-21T08:45:27.016330+0000","last_fullsized":"2024-04-21T08:45:27.016330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992034+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:45:24.992034+0000","last_peered":"2024-04-21T08:45:24.992034+0000","last_clean":"2024-04-21T08:45:24.992034+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:45:24.992034+0000","last_undegraded":"2024-04-21T08:45:24.992034+0000","last_fullsized":"2024-04-21T08:45:24.992034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992367+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:45:24.992367+0000","last_peered":"2024-04-21T08:45:24.992367+0000","last_clean":"2024-04-21T08:45:24.992367+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:45:24.992367+0000","last_undegraded":"2024-04-21T08:45:24.992367+0000","last_fullsized":"2024-04-21T08:45:24.992367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993257+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:45:24.993257+0000","last_peered":"2024-04-21T08:45:24.993257+0000","last_clean":"2024-04-21T08:45:24.993257+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:45:24.993257+0000","last_undegraded":"2024-04-21T08:45:24.993257+0000","last_fullsized":"2024-04-21T08:45:24.993257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992262+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:45:24.992262+0000","last_peered":"2024-04-21T08:45:24.992262+0000","last_clean":"2024-04-21T08:45:24.992262+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:45:24.992262+0000","last_undegraded":"2024-04-21T08:45:24.992262+0000","last_fullsized":"2024-04-21T08:45:24.992262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992695+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:45:24.992695+0000","last_peered":"2024-04-21T08:45:24.992695+0000","last_clean":"2024-04-21T08:45:24.992695+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:45:24.992695+0000","last_undegraded":"2024-04-21T08:45:24.992695+0000","last_fullsized":"2024-04-21T08:45:24.992695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903778+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:45:24.903778+0000","last_peered":"2024-04-21T08:45:24.903778+0000","last_clean":"2024-04-21T08:45:24.903778+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:45:24.903778+0000","last_undegraded":"2024-04-21T08:45:24.903778+0000","last_fullsized":"2024-04-21T08:45:24.903778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'68","reported_seq":154,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.658975+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:45:30.658975+0000","last_peered":"2024-04-21T08:45:30.658975+0000","last_clean":"2024-04-21T08:45:30.658975+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:45:30.658975+0000","last_undegraded":"2024-04-21T08:45:30.658975+0000","last_fullsized":"2024-04-21T08:45:30.658975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":160,"num_write":122,"num_write_kb":1157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904474+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:45:24.904474+0000","last_peered":"2024-04-21T08:45:24.904474+0000","last_clean":"2024-04-21T08:45:24.904474+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:45:24.904474+0000","last_undegraded":"2024-04-21T08:45:24.904474+0000","last_fullsized":"2024-04-21T08:45:24.904474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903473+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:45:24.903473+0000","last_peered":"2024-04-21T08:45:24.903473+0000","last_clean":"2024-04-21T08:45:24.903473+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:45:24.903473+0000","last_undegraded":"2024-04-21T08:45:24.903473+0000","last_fullsized":"2024-04-21T08:45:24.903473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904682+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:45:24.904682+0000","last_peered":"2024-04-21T08:45:24.904682+0000","last_clean":"2024-04-21T08:45:24.904682+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:45:24.904682+0000","last_undegraded":"2024-04-21T08:45:24.904682+0000","last_fullsized":"2024-04-21T08:45:24.904682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337838+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:45:25.337838+0000","last_peered":"2024-04-21T08:45:25.337838+0000","last_clean":"2024-04-21T08:45:25.337838+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:45:25.337838+0000","last_undegraded":"2024-04-21T08:45:25.337838+0000","last_fullsized":"2024-04-21T08:45:25.337838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904527+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:45:24.904527+0000","last_peered":"2024-04-21T08:45:24.904527+0000","last_clean":"2024-04-21T08:45:24.904527+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:45:24.904527+0000","last_undegraded":"2024-04-21T08:45:24.904527+0000","last_fullsized":"2024-04-21T08:45:24.904527+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991872+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:45:24.991872+0000","last_peered":"2024-04-21T08:45:24.991872+0000","last_clean":"2024-04-21T08:45:24.991872+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:45:24.991872+0000","last_undegraded":"2024-04-21T08:45:24.991872+0000","last_fullsized":"2024-04-21T08:45:24.991872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991968+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:45:24.991968+0000","last_peered":"2024-04-21T08:45:24.991968+0000","last_clean":"2024-04-21T08:45:24.991968+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:45:24.991968+0000","last_undegraded":"2024-04-21T08:45:24.991968+0000","last_fullsized":"2024-04-21T08:45:24.991968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992202+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:45:24.992202+0000","last_peered":"2024-04-21T08:45:24.992202+0000","last_clean":"2024-04-21T08:45:24.992202+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:45:24.992202+0000","last_undegraded":"2024-04-21T08:45:24.992202+0000","last_fullsized":"2024-04-21T08:45:24.992202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903893+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:45:24.903893+0000","last_peered":"2024-04-21T08:45:24.903893+0000","last_clean":"2024-04-21T08:45:24.903893+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:45:24.903893+0000","last_undegraded":"2024-04-21T08:45:24.903893+0000","last_fullsized":"2024-04-21T08:45:24.903893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992008+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:45:24.992008+0000","last_peered":"2024-04-21T08:45:24.992008+0000","last_clean":"2024-04-21T08:45:24.992008+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:45:24.992008+0000","last_undegraded":"2024-04-21T08:45:24.992008+0000","last_fullsized":"2024-04-21T08:45:24.992008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904288+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:45:24.904288+0000","last_peered":"2024-04-21T08:45:24.904288+0000","last_clean":"2024-04-21T08:45:24.904288+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:45:24.904288+0000","last_undegraded":"2024-04-21T08:45:24.904288+0000","last_fullsized":"2024-04-21T08:45:24.904288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903726+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:45:24.903726+0000","last_peered":"2024-04-21T08:45:24.903726+0000","last_clean":"2024-04-21T08:45:24.903726+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:45:24.903726+0000","last_undegraded":"2024-04-21T08:45:24.903726+0000","last_fullsized":"2024-04-21T08:45:24.903726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903977+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:45:24.903977+0000","last_peered":"2024-04-21T08:45:24.903977+0000","last_clean":"2024-04-21T08:45:24.903977+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:45:24.903977+0000","last_undegraded":"2024-04-21T08:45:24.903977+0000","last_fullsized":"2024-04-21T08:45:24.903977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904319+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:45:24.904319+0000","last_peered":"2024-04-21T08:45:24.904319+0000","last_clean":"2024-04-21T08:45:24.904319+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:45:24.904319+0000","last_undegraded":"2024-04-21T08:45:24.904319+0000","last_fullsized":"2024-04-21T08:45:24.904319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993372+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:45:24.993372+0000","last_peered":"2024-04-21T08:45:24.993372+0000","last_clean":"2024-04-21T08:45:24.993372+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:45:24.993372+0000","last_undegraded":"2024-04-21T08:45:24.993372+0000","last_fullsized":"2024-04-21T08:45:24.993372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992918+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:45:24.992918+0000","last_peered":"2024-04-21T08:45:24.992918+0000","last_clean":"2024-04-21T08:45:24.992918+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:45:24.992918+0000","last_undegraded":"2024-04-21T08:45:24.992918+0000","last_fullsized":"2024-04-21T08:45:24.992918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992146+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:45:24.992146+0000","last_peered":"2024-04-21T08:45:24.992146+0000","last_clean":"2024-04-21T08:45:24.992146+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:45:24.992146+0000","last_undegraded":"2024-04-21T08:45:24.992146+0000","last_fullsized":"2024-04-21T08:45:24.992146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904004+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:45:24.904004+0000","last_peered":"2024-04-21T08:45:24.904004+0000","last_clean":"2024-04-21T08:45:24.904004+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:45:24.904004+0000","last_undegraded":"2024-04-21T08:45:24.904004+0000","last_fullsized":"2024-04-21T08:45:24.904004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337411+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:45:25.337411+0000","last_peered":"2024-04-21T08:45:25.337411+0000","last_clean":"2024-04-21T08:45:25.337411+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:45:25.337411+0000","last_undegraded":"2024-04-21T08:45:25.337411+0000","last_fullsized":"2024-04-21T08:45:25.337411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'57","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.654353+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:45:30.654353+0000","last_peered":"2024-04-21T08:45:30.654353+0000","last_clean":"2024-04-21T08:45:30.654353+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:45:30.654353+0000","last_undegraded":"2024-04-21T08:45:30.654353+0000","last_fullsized":"2024-04-21T08:45:30.654353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":68,"num_write_kb":807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992998+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:45:24.992998+0000","last_peered":"2024-04-21T08:45:24.992998+0000","last_clean":"2024-04-21T08:45:24.992998+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:45:24.992998+0000","last_undegraded":"2024-04-21T08:45:24.992998+0000","last_fullsized":"2024-04-21T08:45:24.992998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992640+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:45:24.992640+0000","last_peered":"2024-04-21T08:45:24.992640+0000","last_clean":"2024-04-21T08:45:24.992640+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:45:24.992640+0000","last_undegraded":"2024-04-21T08:45:24.992640+0000","last_fullsized":"2024-04-21T08:45:24.992640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992686+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:45:24.992686+0000","last_peered":"2024-04-21T08:45:24.992686+0000","last_clean":"2024-04-21T08:45:24.992686+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:45:24.992686+0000","last_undegraded":"2024-04-21T08:45:24.992686+0000","last_fullsized":"2024-04-21T08:45:24.992686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993176+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:45:24.993176+0000","last_peered":"2024-04-21T08:45:24.993176+0000","last_clean":"2024-04-21T08:45:24.993176+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:45:24.993176+0000","last_undegraded":"2024-04-21T08:45:24.993176+0000","last_fullsized":"2024-04-21T08:45:24.993176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992191+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:45:24.992191+0000","last_peered":"2024-04-21T08:45:24.992191+0000","last_clean":"2024-04-21T08:45:24.992191+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:45:24.992191+0000","last_undegraded":"2024-04-21T08:45:24.992191+0000","last_fullsized":"2024-04-21T08:45:24.992191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903722+0000","last_change":"2024-04-21T08:44:47.315087+0000","last_active":"2024-04-21T08:45:24.903722+0000","last_peered":"2024-04-21T08:45:24.903722+0000","last_clean":"2024-04-21T08:45:24.903722+0000","last_became_active":"2024-04-21T08:44:47.314647+0000","last_became_peered":"2024-04-21T08:44:47.314647+0000","last_unstale":"2024-04-21T08:45:24.903722+0000","last_undegraded":"2024-04-21T08:45:24.903722+0000","last_fullsized":"2024-04-21T08:45:24.903722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:20.425332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903546+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:45:24.903546+0000","last_peered":"2024-04-21T08:45:24.903546+0000","last_clean":"2024-04-21T08:45:24.903546+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:45:24.903546+0000","last_undegraded":"2024-04-21T08:45:24.903546+0000","last_fullsized":"2024-04-21T08:45:24.903546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337673+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:45:25.337673+0000","last_peered":"2024-04-21T08:45:25.337673+0000","last_clean":"2024-04-21T08:45:25.337673+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:45:25.337673+0000","last_undegraded":"2024-04-21T08:45:25.337673+0000","last_fullsized":"2024-04-21T08:45:25.337673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992553+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:45:24.992553+0000","last_peered":"2024-04-21T08:45:24.992553+0000","last_clean":"2024-04-21T08:45:24.992553+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:45:24.992553+0000","last_undegraded":"2024-04-21T08:45:24.992553+0000","last_fullsized":"2024-04-21T08:45:24.992553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904194+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:45:24.904194+0000","last_peered":"2024-04-21T08:45:24.904194+0000","last_clean":"2024-04-21T08:45:24.904194+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:45:24.904194+0000","last_undegraded":"2024-04-21T08:45:24.904194+0000","last_fullsized":"2024-04-21T08:45:24.904194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904716+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:45:24.904716+0000","last_peered":"2024-04-21T08:45:24.904716+0000","last_clean":"2024-04-21T08:45:24.904716+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:45:24.904716+0000","last_undegraded":"2024-04-21T08:45:24.904716+0000","last_fullsized":"2024-04-21T08:45:24.904716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992795+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:45:24.992795+0000","last_peered":"2024-04-21T08:45:24.992795+0000","last_clean":"2024-04-21T08:45:24.992795+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:45:24.992795+0000","last_undegraded":"2024-04-21T08:45:24.992795+0000","last_fullsized":"2024-04-21T08:45:24.992795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903992+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:45:24.903992+0000","last_peered":"2024-04-21T08:45:24.903992+0000","last_clean":"2024-04-21T08:45:24.903992+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:45:24.903992+0000","last_undegraded":"2024-04-21T08:45:24.903992+0000","last_fullsized":"2024-04-21T08:45:24.903992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992360+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:45:24.992360+0000","last_peered":"2024-04-21T08:45:24.992360+0000","last_clean":"2024-04-21T08:45:24.992360+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:45:24.992360+0000","last_undegraded":"2024-04-21T08:45:24.992360+0000","last_fullsized":"2024-04-21T08:45:24.992360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992585+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:45:24.992585+0000","last_peered":"2024-04-21T08:45:24.992585+0000","last_clean":"2024-04-21T08:45:24.992585+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:45:24.992585+0000","last_undegraded":"2024-04-21T08:45:24.992585+0000","last_fullsized":"2024-04-21T08:45:24.992585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993390+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:45:24.993390+0000","last_peered":"2024-04-21T08:45:24.993390+0000","last_clean":"2024-04-21T08:45:24.993390+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:45:24.993390+0000","last_undegraded":"2024-04-21T08:45:24.993390+0000","last_fullsized":"2024-04-21T08:45:24.993390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":110016,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":792314,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":476,"num_read_kb":388,"num_write":264,"num_write_kb":2062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":2376942,"data_compressed":107229,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528781,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1316881,"data_compressed":74099,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":926641,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":922567,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1123793,"data_compressed":66315,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495687,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":524651,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":524741,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1246203,"data_compressed":63758,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":528769,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1447529,"data_compressed":81155,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":721552,"data_compressed":34115,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":328414,"data_compressed":14840,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":398494,"data_compressed":16468,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:45:35.705 INFO:tasks.quiescer.ceph_manager:clean! 2024-04-21T08:45:35.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-21T08:45:35.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:35 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/238017442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:35 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3799344842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:35 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1927420105' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:35.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 -- 172.21.15.57:0/1067955362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a4106180 msgr2=0x7fe4a410dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 --2- 172.21.15.57:0/1067955362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a4106180 0x7fe4a410dd50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe49c009ca0 tx=0x7fe49c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 -- 172.21.15.57:0/1067955362 shutdown_connections 2024-04-21T08:45:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 --2- 172.21.15.57:0/1067955362 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a4106180 0x7fe4a410dd50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 --2- 172.21.15.57:0/1067955362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a4101a60 0x7fe4a4105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 --2- 172.21.15.57:0/1067955362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4a41010b0 0x7fe4a4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 -- 172.21.15.57:0/1067955362 >> 172.21.15.57:0/1067955362 conn(0x7fe4a40771d0 msgr2=0x7fe4a40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 -- 172.21.15.57:0/1067955362 shutdown_connections 2024-04-21T08:45:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.840+0000 7fe4a9954640 1 -- 172.21.15.57:0/1067955362 wait complete. 2024-04-21T08:45:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 Processor -- start 2024-04-21T08:45:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 -- start start 2024-04-21T08:45:35.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 0x7fe4a4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4a4106180 0x7fe4a4198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a419cec0 0x7fe4a419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4112200 con 0x7fe4a41010b0 2024-04-21T08:45:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4112080 con 0x7fe4a419cec0 2024-04-21T08:45:35.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.841+0000 7fe4a9954640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a4112380 con 0x7fe4a4106180 2024-04-21T08:45:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 0x7fe4a4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 0x7fe4a4198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55564/0 (socket says 172.21.15.57:55564) 2024-04-21T08:45:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 -- 172.21.15.57:0/3898646694 learned_addr learned my addr 172.21.15.57:0/3898646694 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a37fe640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a419cec0 0x7fe4a419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 -- 172.21.15.57:0/3898646694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4a4106180 msgr2=0x7fe4a4198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4a4106180 0x7fe4a4198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 -- 172.21.15.57:0/3898646694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a419cec0 msgr2=0x7fe4a419d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a419cec0 0x7fe4a419d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 -- 172.21.15.57:0/3898646694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a419d9b0 con 0x7fe4a41010b0 2024-04-21T08:45:35.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a37fe640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a419cec0 0x7fe4a419d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a2ffd640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 0x7fe4a4198600 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe49400d950 tx=0x7fe49400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494017070 con 0x7fe4a41010b0 2024-04-21T08:45:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe494009c30 con 0x7fe4a41010b0 2024-04-21T08:45:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a419dca0 con 0x7fe4a41010b0 2024-04-21T08:45:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49400f620 con 0x7fe4a41010b0 2024-04-21T08:45:35.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.842+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a41a5490 con 0x7fe4a41010b0 2024-04-21T08:45:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.843+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a4069230 con 0x7fe4a41010b0 2024-04-21T08:45:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.844+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fe494010460 con 0x7fe4a41010b0 2024-04-21T08:45:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.845+0000 7fe4a8952640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe48c0a3db0 0x7fe48c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.845+0000 7fe4a27fc640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe48c0a3db0 0x7fe48c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.845+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe4940d0a50 con 0x7fe4a41010b0 2024-04-21T08:45:35.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.846+0000 7fe4a27fc640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe48c0a3db0 0x7fe48c0a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4100660 tx=0x7fe498005c00 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:35.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:35.849+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe49409cbc0 con 0x7fe4a41010b0 2024-04-21T08:45:36.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.038+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4a4101a60 con 0x7fe48c0a3db0 2024-04-21T08:45:36.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.047+0000 7fe4a8952640 1 -- 172.21.15.57:0/3898646694 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339645 (secure 0 0 0) 0x7fe4a4101a60 con 0x7fe48c0a3db0 2024-04-21T08:45:36.052 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:36.054 INFO:teuthology.orchestra.run.smithi057.stderr:dumped all 2024-04-21T08:45:36.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe48c0a3db0 msgr2=0x7fe48c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe48c0a3db0 0x7fe48c0a6270 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4100660 tx=0x7fe498005c00 comp rx=0 tx=0).stop 2024-04-21T08:45:36.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 msgr2=0x7fe4a4198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 0x7fe4a4198600 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe49400d950 tx=0x7fe49400de10 comp rx=0 tx=0).stop 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 shutdown_connections 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe48c0a3db0 0x7fe48c0a6270 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4a419cec0 0x7fe4a419d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4a4106180 0x7fe4a4198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 --2- 172.21.15.57:0/3898646694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe4a41010b0 0x7fe4a4198600 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 >> 172.21.15.57:0/3898646694 conn(0x7fe4a40771d0 msgr2=0x7fe4a40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 shutdown_connections 2024-04-21T08:45:36.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.051+0000 7fe4a9954640 1 -- 172.21.15.57:0/3898646694 wait complete. 2024-04-21T08:45:36.063 INFO:teuthology.orchestra.run.smithi057.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-04-21T08:45:34.919073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1382916,"num_objects":28,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":713,"num_read_kb":718,"num_write":467,"num_write_kb":5002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180770,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":843,"ondisk_log_size":843,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365976,"kb_used_data":12632,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575848,"statfs":{"total":1151940427776,"available":1151565668352,"internally_reserved":0,"allocated":12935168,"data_stored":10279994,"data_compressed":564019,"data_compressed_allocated":761856,"data_compressed_original":4694016,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":223,"num_read_kb":318,"num_write":84,"num_write_kb":652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.988802"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993678+0000","last_change":"2024-04-21T08:42:57.405612+0000","last_active":"2024-04-21T08:45:24.993678+0000","last_peered":"2024-04-21T08:45:24.993678+0000","last_clean":"2024-04-21T08:45:24.993678+0000","last_became_active":"2024-04-21T08:42:57.404473+0000","last_became_peered":"2024-04-21T08:42:57.404473+0000","last_unstale":"2024-04-21T08:45:24.993678+0000","last_undegraded":"2024-04-21T08:45:24.993678+0000","last_fullsized":"2024-04-21T08:45:24.993678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:11:14.354905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337688+0000","last_change":"2024-04-21T08:42:57.399034+0000","last_active":"2024-04-21T08:45:25.337688+0000","last_peered":"2024-04-21T08:45:25.337688+0000","last_clean":"2024-04-21T08:45:25.337688+0000","last_became_active":"2024-04-21T08:42:57.398690+0000","last_became_peered":"2024-04-21T08:42:57.398690+0000","last_unstale":"2024-04-21T08:45:25.337688+0000","last_undegraded":"2024-04-21T08:45:25.337688+0000","last_fullsized":"2024-04-21T08:45:25.337688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:59:51.608367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338255+0000","last_change":"2024-04-21T08:42:57.404754+0000","last_active":"2024-04-21T08:45:25.338255+0000","last_peered":"2024-04-21T08:45:25.338255+0000","last_clean":"2024-04-21T08:45:25.338255+0000","last_became_active":"2024-04-21T08:42:57.404411+0000","last_became_peered":"2024-04-21T08:42:57.404411+0000","last_unstale":"2024-04-21T08:45:25.338255+0000","last_undegraded":"2024-04-21T08:45:25.338255+0000","last_fullsized":"2024-04-21T08:45:25.338255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:11:41.648970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904383+0000","last_change":"2024-04-21T08:42:57.411040+0000","last_active":"2024-04-21T08:45:24.904383+0000","last_peered":"2024-04-21T08:45:24.904383+0000","last_clean":"2024-04-21T08:45:24.904383+0000","last_became_active":"2024-04-21T08:42:57.410671+0000","last_became_peered":"2024-04-21T08:42:57.410671+0000","last_unstale":"2024-04-21T08:45:24.904383+0000","last_undegraded":"2024-04-21T08:45:24.904383+0000","last_fullsized":"2024-04-21T08:45:24.904383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:23.887049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991986+0000","last_change":"2024-04-21T08:42:57.403022+0000","last_active":"2024-04-21T08:45:24.991986+0000","last_peered":"2024-04-21T08:45:24.991986+0000","last_clean":"2024-04-21T08:45:24.991986+0000","last_became_active":"2024-04-21T08:42:57.402640+0000","last_became_peered":"2024-04-21T08:42:57.402640+0000","last_unstale":"2024-04-21T08:45:24.991986+0000","last_undegraded":"2024-04-21T08:45:24.991986+0000","last_fullsized":"2024-04-21T08:45:24.991986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:19:26.627656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992587+0000","last_change":"2024-04-21T08:42:57.410246+0000","last_active":"2024-04-21T08:45:24.992587+0000","last_peered":"2024-04-21T08:45:24.992587+0000","last_clean":"2024-04-21T08:45:24.992587+0000","last_became_active":"2024-04-21T08:42:57.409870+0000","last_became_peered":"2024-04-21T08:42:57.409870+0000","last_unstale":"2024-04-21T08:45:24.992587+0000","last_undegraded":"2024-04-21T08:45:24.992587+0000","last_fullsized":"2024-04-21T08:45:24.992587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:19.119109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992192+0000","last_change":"2024-04-21T08:42:57.396145+0000","last_active":"2024-04-21T08:45:24.992192+0000","last_peered":"2024-04-21T08:45:24.992192+0000","last_clean":"2024-04-21T08:45:24.992192+0000","last_became_active":"2024-04-21T08:42:57.395795+0000","last_became_peered":"2024-04-21T08:42:57.395795+0000","last_unstale":"2024-04-21T08:45:24.992192+0000","last_undegraded":"2024-04-21T08:45:24.992192+0000","last_fullsized":"2024-04-21T08:45:24.992192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:23:29.429363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992575+0000","last_change":"2024-04-21T08:44:47.312984+0000","last_active":"2024-04-21T08:45:24.992575+0000","last_peered":"2024-04-21T08:45:24.992575+0000","last_clean":"2024-04-21T08:45:24.992575+0000","last_became_active":"2024-04-21T08:44:47.312465+0000","last_became_peered":"2024-04-21T08:44:47.312465+0000","last_unstale":"2024-04-21T08:45:24.992575+0000","last_undegraded":"2024-04-21T08:45:24.992575+0000","last_fullsized":"2024-04-21T08:45:24.992575+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:59:51.214620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993071+0000","last_change":"2024-04-21T08:42:57.397688+0000","last_active":"2024-04-21T08:45:24.993071+0000","last_peered":"2024-04-21T08:45:24.993071+0000","last_clean":"2024-04-21T08:45:24.993071+0000","last_became_active":"2024-04-21T08:42:57.397156+0000","last_became_peered":"2024-04-21T08:42:57.397156+0000","last_unstale":"2024-04-21T08:45:24.993071+0000","last_undegraded":"2024-04-21T08:45:24.993071+0000","last_fullsized":"2024-04-21T08:45:24.993071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:26:18.634727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993708+0000","last_change":"2024-04-21T08:42:57.413663+0000","last_active":"2024-04-21T08:45:24.993708+0000","last_peered":"2024-04-21T08:45:24.993708+0000","last_clean":"2024-04-21T08:45:24.993708+0000","last_became_active":"2024-04-21T08:42:57.413276+0000","last_became_peered":"2024-04-21T08:42:57.413276+0000","last_unstale":"2024-04-21T08:45:24.993708+0000","last_undegraded":"2024-04-21T08:45:24.993708+0000","last_fullsized":"2024-04-21T08:45:24.993708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:22:50.744414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993237+0000","last_change":"2024-04-21T08:42:57.404870+0000","last_active":"2024-04-21T08:45:24.993237+0000","last_peered":"2024-04-21T08:45:24.993237+0000","last_clean":"2024-04-21T08:45:24.993237+0000","last_became_active":"2024-04-21T08:42:57.404420+0000","last_became_peered":"2024-04-21T08:42:57.404420+0000","last_unstale":"2024-04-21T08:45:24.993237+0000","last_undegraded":"2024-04-21T08:45:24.993237+0000","last_fullsized":"2024-04-21T08:45:24.993237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:31:28.988459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904442+0000","last_change":"2024-04-21T08:42:57.409223+0000","last_active":"2024-04-21T08:45:24.904442+0000","last_peered":"2024-04-21T08:45:24.904442+0000","last_clean":"2024-04-21T08:45:24.904442+0000","last_became_active":"2024-04-21T08:42:57.408621+0000","last_became_peered":"2024-04-21T08:42:57.408621+0000","last_unstale":"2024-04-21T08:45:24.904442+0000","last_undegraded":"2024-04-21T08:45:24.904442+0000","last_fullsized":"2024-04-21T08:45:24.904442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:06:45.809723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992519+0000","last_change":"2024-04-21T08:42:57.400357+0000","last_active":"2024-04-21T08:45:24.992519+0000","last_peered":"2024-04-21T08:45:24.992519+0000","last_clean":"2024-04-21T08:45:24.992519+0000","last_became_active":"2024-04-21T08:42:57.400044+0000","last_became_peered":"2024-04-21T08:42:57.400044+0000","last_unstale":"2024-04-21T08:45:24.992519+0000","last_undegraded":"2024-04-21T08:45:24.992519+0000","last_fullsized":"2024-04-21T08:45:24.992519+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:22.673292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337366+0000","last_change":"2024-04-21T08:42:57.400665+0000","last_active":"2024-04-21T08:45:25.337366+0000","last_peered":"2024-04-21T08:45:25.337366+0000","last_clean":"2024-04-21T08:45:25.337366+0000","last_became_active":"2024-04-21T08:42:57.400329+0000","last_became_peered":"2024-04-21T08:42:57.400329+0000","last_unstale":"2024-04-21T08:45:25.337366+0000","last_undegraded":"2024-04-21T08:45:25.337366+0000","last_fullsized":"2024-04-21T08:45:25.337366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:49:30.362661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337428+0000","last_change":"2024-04-21T08:42:57.408959+0000","last_active":"2024-04-21T08:45:25.337428+0000","last_peered":"2024-04-21T08:45:25.337428+0000","last_clean":"2024-04-21T08:45:25.337428+0000","last_became_active":"2024-04-21T08:42:57.408541+0000","last_became_peered":"2024-04-21T08:42:57.408541+0000","last_unstale":"2024-04-21T08:45:25.337428+0000","last_undegraded":"2024-04-21T08:45:25.337428+0000","last_fullsized":"2024-04-21T08:45:25.337428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:59:02.814846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991935+0000","last_change":"2024-04-21T08:42:57.401771+0000","last_active":"2024-04-21T08:45:24.991935+0000","last_peered":"2024-04-21T08:45:24.991935+0000","last_clean":"2024-04-21T08:45:24.991935+0000","last_became_active":"2024-04-21T08:42:57.398699+0000","last_became_peered":"2024-04-21T08:42:57.398699+0000","last_unstale":"2024-04-21T08:45:24.991935+0000","last_undegraded":"2024-04-21T08:45:24.991935+0000","last_fullsized":"2024-04-21T08:45:24.991935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:36:56.179497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993546+0000","last_change":"2024-04-21T08:42:57.406897+0000","last_active":"2024-04-21T08:45:24.993546+0000","last_peered":"2024-04-21T08:45:24.993546+0000","last_clean":"2024-04-21T08:45:24.993546+0000","last_became_active":"2024-04-21T08:42:57.406311+0000","last_became_peered":"2024-04-21T08:42:57.406311+0000","last_unstale":"2024-04-21T08:45:24.993546+0000","last_undegraded":"2024-04-21T08:45:24.993546+0000","last_fullsized":"2024-04-21T08:45:24.993546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:48:28.271566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993979+0000","last_change":"2024-04-21T08:42:57.402606+0000","last_active":"2024-04-21T08:45:24.993979+0000","last_peered":"2024-04-21T08:45:24.993979+0000","last_clean":"2024-04-21T08:45:24.993979+0000","last_became_active":"2024-04-21T08:42:57.402223+0000","last_became_peered":"2024-04-21T08:42:57.402223+0000","last_unstale":"2024-04-21T08:45:24.993979+0000","last_undegraded":"2024-04-21T08:45:24.993979+0000","last_fullsized":"2024-04-21T08:45:24.993979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:35:33.616482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904038+0000","last_change":"2024-04-21T08:42:57.406328+0000","last_active":"2024-04-21T08:45:24.904038+0000","last_peered":"2024-04-21T08:45:24.904038+0000","last_clean":"2024-04-21T08:45:24.904038+0000","last_became_active":"2024-04-21T08:42:57.406012+0000","last_became_peered":"2024-04-21T08:42:57.406012+0000","last_unstale":"2024-04-21T08:45:24.904038+0000","last_undegraded":"2024-04-21T08:45:24.904038+0000","last_fullsized":"2024-04-21T08:45:24.904038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:25:43.124352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903663+0000","last_change":"2024-04-21T08:43:01.689263+0000","last_active":"2024-04-21T08:45:24.903663+0000","last_peered":"2024-04-21T08:45:24.903663+0000","last_clean":"2024-04-21T08:45:24.903663+0000","last_became_active":"2024-04-21T08:43:01.688807+0000","last_became_peered":"2024-04-21T08:43:01.688807+0000","last_unstale":"2024-04-21T08:45:24.903663+0000","last_undegraded":"2024-04-21T08:45:24.903663+0000","last_fullsized":"2024-04-21T08:45:24.903663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:42:08.244309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.994306+0000","last_change":"2024-04-21T08:42:56.639163+0000","last_active":"2024-04-21T08:45:24.994306+0000","last_peered":"2024-04-21T08:45:24.994306+0000","last_clean":"2024-04-21T08:45:24.994306+0000","last_became_active":"2024-04-21T08:42:56.638797+0000","last_became_peered":"2024-04-21T08:42:56.638797+0000","last_unstale":"2024-04-21T08:45:24.994306+0000","last_undegraded":"2024-04-21T08:45:24.994306+0000","last_fullsized":"2024-04-21T08:45:24.994306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:30:42.507627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904457+0000","last_change":"2024-04-21T08:42:57.403693+0000","last_active":"2024-04-21T08:45:24.904457+0000","last_peered":"2024-04-21T08:45:24.904457+0000","last_clean":"2024-04-21T08:45:24.904457+0000","last_became_active":"2024-04-21T08:42:57.403191+0000","last_became_peered":"2024-04-21T08:42:57.403191+0000","last_unstale":"2024-04-21T08:45:24.904457+0000","last_undegraded":"2024-04-21T08:45:24.904457+0000","last_fullsized":"2024-04-21T08:45:24.904457+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:45:46.550483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904151+0000","last_change":"2024-04-21T08:43:01.680678+0000","last_active":"2024-04-21T08:45:24.904151+0000","last_peered":"2024-04-21T08:45:24.904151+0000","last_clean":"2024-04-21T08:45:24.904151+0000","last_became_active":"2024-04-21T08:43:01.679944+0000","last_became_peered":"2024-04-21T08:43:01.679944+0000","last_unstale":"2024-04-21T08:45:24.904151+0000","last_undegraded":"2024-04-21T08:45:24.904151+0000","last_fullsized":"2024-04-21T08:45:24.904151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:04:42.774088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904639+0000","last_change":"2024-04-21T08:42:56.645603+0000","last_active":"2024-04-21T08:45:24.904639+0000","last_peered":"2024-04-21T08:45:24.904639+0000","last_clean":"2024-04-21T08:45:24.904639+0000","last_became_active":"2024-04-21T08:42:56.645097+0000","last_became_peered":"2024-04-21T08:42:56.645097+0000","last_unstale":"2024-04-21T08:45:24.904639+0000","last_undegraded":"2024-04-21T08:45:24.904639+0000","last_fullsized":"2024-04-21T08:45:24.904639+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:42:06.047062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.338007+0000","last_change":"2024-04-21T08:42:57.405427+0000","last_active":"2024-04-21T08:45:25.338007+0000","last_peered":"2024-04-21T08:45:25.338007+0000","last_clean":"2024-04-21T08:45:25.338007+0000","last_became_active":"2024-04-21T08:42:57.405093+0000","last_became_peered":"2024-04-21T08:42:57.405093+0000","last_unstale":"2024-04-21T08:45:25.338007+0000","last_undegraded":"2024-04-21T08:45:25.338007+0000","last_fullsized":"2024-04-21T08:45:25.338007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:57:25.642089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904338+0000","last_change":"2024-04-21T08:43:46.474224+0000","last_active":"2024-04-21T08:45:24.904338+0000","last_peered":"2024-04-21T08:45:24.904338+0000","last_clean":"2024-04-21T08:45:24.904338+0000","last_became_active":"2024-04-21T08:43:46.473715+0000","last_became_peered":"2024-04-21T08:43:46.473715+0000","last_unstale":"2024-04-21T08:45:24.904338+0000","last_undegraded":"2024-04-21T08:45:24.904338+0000","last_fullsized":"2024-04-21T08:45:24.904338+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:12:03.768248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,10],"acting":[4,3,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992440+0000","last_change":"2024-04-21T08:42:56.639948+0000","last_active":"2024-04-21T08:45:24.992440+0000","last_peered":"2024-04-21T08:45:24.992440+0000","last_clean":"2024-04-21T08:45:24.992440+0000","last_became_active":"2024-04-21T08:42:56.639101+0000","last_became_peered":"2024-04-21T08:42:56.639101+0000","last_unstale":"2024-04-21T08:45:24.992440+0000","last_undegraded":"2024-04-21T08:45:24.992440+0000","last_fullsized":"2024-04-21T08:45:24.992440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:21:32.992410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992517+0000","last_change":"2024-04-21T08:42:57.404326+0000","last_active":"2024-04-21T08:45:24.992517+0000","last_peered":"2024-04-21T08:45:24.992517+0000","last_clean":"2024-04-21T08:45:24.992517+0000","last_became_active":"2024-04-21T08:42:57.403878+0000","last_became_peered":"2024-04-21T08:42:57.403878+0000","last_unstale":"2024-04-21T08:45:24.992517+0000","last_undegraded":"2024-04-21T08:45:24.992517+0000","last_fullsized":"2024-04-21T08:45:24.992517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:21:03.188225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904082+0000","last_change":"2024-04-21T08:43:01.678269+0000","last_active":"2024-04-21T08:45:24.904082+0000","last_peered":"2024-04-21T08:45:24.904082+0000","last_clean":"2024-04-21T08:45:24.904082+0000","last_became_active":"2024-04-21T08:43:01.677795+0000","last_became_peered":"2024-04-21T08:43:01.677795+0000","last_unstale":"2024-04-21T08:45:24.904082+0000","last_undegraded":"2024-04-21T08:45:24.904082+0000","last_fullsized":"2024-04-21T08:45:24.904082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:37:48.039248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:42:56.640041+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:42:56.639715+0000","last_became_peered":"2024-04-21T08:42:56.639715+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:22.649781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993055+0000","last_change":"2024-04-21T08:42:57.410918+0000","last_active":"2024-04-21T08:45:24.993055+0000","last_peered":"2024-04-21T08:45:24.993055+0000","last_clean":"2024-04-21T08:45:24.993055+0000","last_became_active":"2024-04-21T08:42:57.410622+0000","last_became_peered":"2024-04-21T08:42:57.410622+0000","last_unstale":"2024-04-21T08:45:24.993055+0000","last_undegraded":"2024-04-21T08:45:24.993055+0000","last_fullsized":"2024-04-21T08:45:24.993055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:39:38.491886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903921+0000","last_change":"2024-04-21T08:43:01.678289+0000","last_active":"2024-04-21T08:45:24.903921+0000","last_peered":"2024-04-21T08:45:24.903921+0000","last_clean":"2024-04-21T08:45:24.903921+0000","last_became_active":"2024-04-21T08:43:01.677855+0000","last_became_peered":"2024-04-21T08:43:01.677855+0000","last_unstale":"2024-04-21T08:45:24.903921+0000","last_undegraded":"2024-04-21T08:45:24.903921+0000","last_fullsized":"2024-04-21T08:45:24.903921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:41:48.814601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993496+0000","last_change":"2024-04-21T08:43:46.474267+0000","last_active":"2024-04-21T08:45:24.993496+0000","last_peered":"2024-04-21T08:45:24.993496+0000","last_clean":"2024-04-21T08:45:24.993496+0000","last_became_active":"2024-04-21T08:43:46.473787+0000","last_became_peered":"2024-04-21T08:43:46.473787+0000","last_unstale":"2024-04-21T08:45:24.993496+0000","last_undegraded":"2024-04-21T08:45:24.993496+0000","last_fullsized":"2024-04-21T08:45:24.993496+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:43:59.818918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992410+0000","last_change":"2024-04-21T08:42:57.393527+0000","last_active":"2024-04-21T08:45:24.992410+0000","last_peered":"2024-04-21T08:45:24.992410+0000","last_clean":"2024-04-21T08:45:24.992410+0000","last_became_active":"2024-04-21T08:42:57.393243+0000","last_became_peered":"2024-04-21T08:42:57.393243+0000","last_unstale":"2024-04-21T08:45:24.992410+0000","last_undegraded":"2024-04-21T08:45:24.992410+0000","last_fullsized":"2024-04-21T08:45:24.992410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:40:52.068262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903852+0000","last_change":"2024-04-21T08:43:01.689428+0000","last_active":"2024-04-21T08:45:24.903852+0000","last_peered":"2024-04-21T08:45:24.903852+0000","last_clean":"2024-04-21T08:45:24.903852+0000","last_became_active":"2024-04-21T08:43:01.689010+0000","last_became_peered":"2024-04-21T08:43:01.689010+0000","last_unstale":"2024-04-21T08:45:24.903852+0000","last_undegraded":"2024-04-21T08:45:24.903852+0000","last_fullsized":"2024-04-21T08:45:24.903852+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:55:47.477475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992469+0000","last_change":"2024-04-21T08:42:56.645394+0000","last_active":"2024-04-21T08:45:24.992469+0000","last_peered":"2024-04-21T08:45:24.992469+0000","last_clean":"2024-04-21T08:45:24.992469+0000","last_became_active":"2024-04-21T08:42:56.645119+0000","last_became_peered":"2024-04-21T08:42:56.645119+0000","last_unstale":"2024-04-21T08:45:24.992469+0000","last_undegraded":"2024-04-21T08:45:24.992469+0000","last_fullsized":"2024-04-21T08:45:24.992469+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:13:06.040598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993353+0000","last_change":"2024-04-21T08:42:57.410139+0000","last_active":"2024-04-21T08:45:24.993353+0000","last_peered":"2024-04-21T08:45:24.993353+0000","last_clean":"2024-04-21T08:45:24.993353+0000","last_became_active":"2024-04-21T08:42:57.409811+0000","last_became_peered":"2024-04-21T08:42:57.409811+0000","last_unstale":"2024-04-21T08:45:24.993353+0000","last_undegraded":"2024-04-21T08:45:24.993353+0000","last_fullsized":"2024-04-21T08:45:24.993353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:56:53.175669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993857+0000","last_change":"2024-04-21T08:43:01.678569+0000","last_active":"2024-04-21T08:45:24.993857+0000","last_peered":"2024-04-21T08:45:24.993857+0000","last_clean":"2024-04-21T08:45:24.993857+0000","last_became_active":"2024-04-21T08:43:01.677097+0000","last_became_peered":"2024-04-21T08:43:01.677097+0000","last_unstale":"2024-04-21T08:45:24.993857+0000","last_undegraded":"2024-04-21T08:45:24.993857+0000","last_fullsized":"2024-04-21T08:45:24.993857+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:54:38.342945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992300+0000","last_change":"2024-04-21T08:42:56.635201+0000","last_active":"2024-04-21T08:45:24.992300+0000","last_peered":"2024-04-21T08:45:24.992300+0000","last_clean":"2024-04-21T08:45:24.992300+0000","last_became_active":"2024-04-21T08:42:56.634734+0000","last_became_peered":"2024-04-21T08:42:56.634734+0000","last_unstale":"2024-04-21T08:45:24.992300+0000","last_undegraded":"2024-04-21T08:45:24.992300+0000","last_fullsized":"2024-04-21T08:45:24.992300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:58:12.177539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903970+0000","last_change":"2024-04-21T08:42:57.399201+0000","last_active":"2024-04-21T08:45:24.903970+0000","last_peered":"2024-04-21T08:45:24.903970+0000","last_clean":"2024-04-21T08:45:24.903970+0000","last_became_active":"2024-04-21T08:42:57.398890+0000","last_became_peered":"2024-04-21T08:42:57.398890+0000","last_unstale":"2024-04-21T08:45:24.903970+0000","last_undegraded":"2024-04-21T08:45:24.903970+0000","last_fullsized":"2024-04-21T08:45:24.903970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:49:43.431448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337570+0000","last_change":"2024-04-21T08:43:01.676248+0000","last_active":"2024-04-21T08:45:25.337570+0000","last_peered":"2024-04-21T08:45:25.337570+0000","last_clean":"2024-04-21T08:45:25.337570+0000","last_became_active":"2024-04-21T08:43:01.675859+0000","last_became_peered":"2024-04-21T08:43:01.675859+0000","last_unstale":"2024-04-21T08:45:25.337570+0000","last_undegraded":"2024-04-21T08:45:25.337570+0000","last_fullsized":"2024-04-21T08:45:25.337570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:57:15.496904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992789+0000","last_change":"2024-04-21T08:42:56.635676+0000","last_active":"2024-04-21T08:45:24.992789+0000","last_peered":"2024-04-21T08:45:24.992789+0000","last_clean":"2024-04-21T08:45:24.992789+0000","last_became_active":"2024-04-21T08:42:56.635374+0000","last_became_peered":"2024-04-21T08:42:56.635374+0000","last_unstale":"2024-04-21T08:45:24.992789+0000","last_undegraded":"2024-04-21T08:45:24.992789+0000","last_fullsized":"2024-04-21T08:45:24.992789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:23:57.146619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992907+0000","last_change":"2024-04-21T08:42:57.398136+0000","last_active":"2024-04-21T08:45:24.992907+0000","last_peered":"2024-04-21T08:45:24.992907+0000","last_clean":"2024-04-21T08:45:24.992907+0000","last_became_active":"2024-04-21T08:42:57.397144+0000","last_became_peered":"2024-04-21T08:42:57.397144+0000","last_unstale":"2024-04-21T08:45:24.992907+0000","last_undegraded":"2024-04-21T08:45:24.992907+0000","last_fullsized":"2024-04-21T08:45:24.992907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:28:44.903871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903912+0000","last_change":"2024-04-21T08:43:01.681278+0000","last_active":"2024-04-21T08:45:24.903912+0000","last_peered":"2024-04-21T08:45:24.903912+0000","last_clean":"2024-04-21T08:45:24.903912+0000","last_became_active":"2024-04-21T08:43:01.680885+0000","last_became_peered":"2024-04-21T08:43:01.680885+0000","last_unstale":"2024-04-21T08:45:24.903912+0000","last_undegraded":"2024-04-21T08:45:24.903912+0000","last_fullsized":"2024-04-21T08:45:24.903912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:02:01.374539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993629+0000","last_change":"2024-04-21T08:42:56.637068+0000","last_active":"2024-04-21T08:45:24.993629+0000","last_peered":"2024-04-21T08:45:24.993629+0000","last_clean":"2024-04-21T08:45:24.993629+0000","last_became_active":"2024-04-21T08:42:56.636610+0000","last_became_peered":"2024-04-21T08:42:56.636610+0000","last_unstale":"2024-04-21T08:45:24.993629+0000","last_undegraded":"2024-04-21T08:45:24.993629+0000","last_fullsized":"2024-04-21T08:45:24.993629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:54:21.721872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992855+0000","last_change":"2024-04-21T08:42:57.400063+0000","last_active":"2024-04-21T08:45:24.992855+0000","last_peered":"2024-04-21T08:45:24.992855+0000","last_clean":"2024-04-21T08:45:24.992855+0000","last_became_active":"2024-04-21T08:42:57.398407+0000","last_became_peered":"2024-04-21T08:42:57.398407+0000","last_unstale":"2024-04-21T08:45:24.992855+0000","last_undegraded":"2024-04-21T08:45:24.992855+0000","last_fullsized":"2024-04-21T08:45:24.992855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:04:24.694923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992997+0000","last_change":"2024-04-21T08:43:01.685227+0000","last_active":"2024-04-21T08:45:24.992997+0000","last_peered":"2024-04-21T08:45:24.992997+0000","last_clean":"2024-04-21T08:45:24.992997+0000","last_became_active":"2024-04-21T08:43:01.684040+0000","last_became_peered":"2024-04-21T08:43:01.684040+0000","last_unstale":"2024-04-21T08:45:24.992997+0000","last_undegraded":"2024-04-21T08:45:24.992997+0000","last_fullsized":"2024-04-21T08:45:24.992997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:03:13.099200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993149+0000","last_change":"2024-04-21T08:42:56.640126+0000","last_active":"2024-04-21T08:45:24.993149+0000","last_peered":"2024-04-21T08:45:24.993149+0000","last_clean":"2024-04-21T08:45:24.993149+0000","last_became_active":"2024-04-21T08:42:56.639723+0000","last_became_peered":"2024-04-21T08:42:56.639723+0000","last_unstale":"2024-04-21T08:45:24.993149+0000","last_undegraded":"2024-04-21T08:45:24.993149+0000","last_fullsized":"2024-04-21T08:45:24.993149+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:15:30.703694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337962+0000","last_change":"2024-04-21T08:42:57.409437+0000","last_active":"2024-04-21T08:45:25.337962+0000","last_peered":"2024-04-21T08:45:25.337962+0000","last_clean":"2024-04-21T08:45:25.337962+0000","last_became_active":"2024-04-21T08:42:57.408343+0000","last_became_peered":"2024-04-21T08:42:57.408343+0000","last_unstale":"2024-04-21T08:45:25.337962+0000","last_undegraded":"2024-04-21T08:45:25.337962+0000","last_fullsized":"2024-04-21T08:45:25.337962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:02:48.292006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904574+0000","last_change":"2024-04-21T08:43:01.680486+0000","last_active":"2024-04-21T08:45:24.904574+0000","last_peered":"2024-04-21T08:45:24.904574+0000","last_clean":"2024-04-21T08:45:24.904574+0000","last_became_active":"2024-04-21T08:43:01.680030+0000","last_became_peered":"2024-04-21T08:43:01.680030+0000","last_unstale":"2024-04-21T08:45:24.904574+0000","last_undegraded":"2024-04-21T08:45:24.904574+0000","last_fullsized":"2024-04-21T08:45:24.904574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:51:11.383710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993021+0000","last_change":"2024-04-21T08:42:56.637163+0000","last_active":"2024-04-21T08:45:24.993021+0000","last_peered":"2024-04-21T08:45:24.993021+0000","last_clean":"2024-04-21T08:45:24.993021+0000","last_became_active":"2024-04-21T08:42:56.636799+0000","last_became_peered":"2024-04-21T08:42:56.636799+0000","last_unstale":"2024-04-21T08:45:24.993021+0000","last_undegraded":"2024-04-21T08:45:24.993021+0000","last_fullsized":"2024-04-21T08:45:24.993021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:26:20.303500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992689+0000","last_change":"2024-04-21T08:42:57.405520+0000","last_active":"2024-04-21T08:45:24.992689+0000","last_peered":"2024-04-21T08:45:24.992689+0000","last_clean":"2024-04-21T08:45:24.992689+0000","last_became_active":"2024-04-21T08:42:57.405158+0000","last_became_peered":"2024-04-21T08:42:57.405158+0000","last_unstale":"2024-04-21T08:45:24.992689+0000","last_undegraded":"2024-04-21T08:45:24.992689+0000","last_fullsized":"2024-04-21T08:45:24.992689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:51:41.039769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993192+0000","last_change":"2024-04-21T08:43:01.685893+0000","last_active":"2024-04-21T08:45:24.993192+0000","last_peered":"2024-04-21T08:45:24.993192+0000","last_clean":"2024-04-21T08:45:24.993192+0000","last_became_active":"2024-04-21T08:43:01.683480+0000","last_became_peered":"2024-04-21T08:43:01.683480+0000","last_unstale":"2024-04-21T08:45:24.993192+0000","last_undegraded":"2024-04-21T08:45:24.993192+0000","last_fullsized":"2024-04-21T08:45:24.993192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:09:32.364796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'37","reported_seq":339,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.639527+0000","last_change":"2024-04-21T08:42:56.630804+0000","last_active":"2024-04-21T08:45:30.639527+0000","last_peered":"2024-04-21T08:45:30.639527+0000","last_clean":"2024-04-21T08:45:30.639527+0000","last_became_active":"2024-04-21T08:42:56.630143+0000","last_became_peered":"2024-04-21T08:42:56.630143+0000","last_unstale":"2024-04-21T08:45:30.639527+0000","last_undegraded":"2024-04-21T08:45:30.639527+0000","last_fullsized":"2024-04-21T08:45:30.639527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:17:14.253406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70068,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":269,"num_read_kb":204,"num_write":37,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992174+0000","last_change":"2024-04-21T08:42:57.401145+0000","last_active":"2024-04-21T08:45:24.992174+0000","last_peered":"2024-04-21T08:45:24.992174+0000","last_clean":"2024-04-21T08:45:24.992174+0000","last_became_active":"2024-04-21T08:42:57.400802+0000","last_became_peered":"2024-04-21T08:42:57.400802+0000","last_unstale":"2024-04-21T08:45:24.992174+0000","last_undegraded":"2024-04-21T08:45:24.992174+0000","last_fullsized":"2024-04-21T08:45:24.992174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:21:05.168388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993309+0000","last_change":"2024-04-21T08:43:01.680859+0000","last_active":"2024-04-21T08:45:24.993309+0000","last_peered":"2024-04-21T08:45:24.993309+0000","last_clean":"2024-04-21T08:45:24.993309+0000","last_became_active":"2024-04-21T08:43:01.679916+0000","last_became_peered":"2024-04-21T08:43:01.679916+0000","last_unstale":"2024-04-21T08:45:24.993309+0000","last_undegraded":"2024-04-21T08:45:24.993309+0000","last_fullsized":"2024-04-21T08:45:24.993309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:58:38.400082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993119+0000","last_change":"2024-04-21T08:43:46.480696+0000","last_active":"2024-04-21T08:45:24.993119+0000","last_peered":"2024-04-21T08:45:24.993119+0000","last_clean":"2024-04-21T08:45:24.993119+0000","last_became_active":"2024-04-21T08:43:46.472379+0000","last_became_peered":"2024-04-21T08:43:46.472379+0000","last_unstale":"2024-04-21T08:45:24.993119+0000","last_undegraded":"2024-04-21T08:45:24.993119+0000","last_fullsized":"2024-04-21T08:45:24.993119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:54:01.104633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992385+0000","last_change":"2024-04-21T08:42:57.404585+0000","last_active":"2024-04-21T08:45:24.992385+0000","last_peered":"2024-04-21T08:45:24.992385+0000","last_clean":"2024-04-21T08:45:24.992385+0000","last_became_active":"2024-04-21T08:42:57.404263+0000","last_became_peered":"2024-04-21T08:42:57.404263+0000","last_unstale":"2024-04-21T08:45:24.992385+0000","last_undegraded":"2024-04-21T08:45:24.992385+0000","last_fullsized":"2024-04-21T08:45:24.992385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:00:46.361644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993270+0000","last_change":"2024-04-21T08:43:01.677841+0000","last_active":"2024-04-21T08:45:24.993270+0000","last_peered":"2024-04-21T08:45:24.993270+0000","last_clean":"2024-04-21T08:45:24.993270+0000","last_became_active":"2024-04-21T08:43:01.677429+0000","last_became_peered":"2024-04-21T08:43:01.677429+0000","last_unstale":"2024-04-21T08:45:24.993270+0000","last_undegraded":"2024-04-21T08:45:24.993270+0000","last_fullsized":"2024-04-21T08:45:24.993270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:58:15.910126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992679+0000","last_change":"2024-04-21T08:42:56.638255+0000","last_active":"2024-04-21T08:45:24.992679+0000","last_peered":"2024-04-21T08:45:24.992679+0000","last_clean":"2024-04-21T08:45:24.992679+0000","last_became_active":"2024-04-21T08:42:56.637930+0000","last_became_peered":"2024-04-21T08:42:56.637930+0000","last_unstale":"2024-04-21T08:45:24.992679+0000","last_undegraded":"2024-04-21T08:45:24.992679+0000","last_fullsized":"2024-04-21T08:45:24.992679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:28:11.287617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992953+0000","last_change":"2024-04-21T08:42:57.412202+0000","last_active":"2024-04-21T08:45:24.992953+0000","last_peered":"2024-04-21T08:45:24.992953+0000","last_clean":"2024-04-21T08:45:24.992953+0000","last_became_active":"2024-04-21T08:42:57.411841+0000","last_became_peered":"2024-04-21T08:42:57.411841+0000","last_unstale":"2024-04-21T08:45:24.992953+0000","last_undegraded":"2024-04-21T08:45:24.992953+0000","last_fullsized":"2024-04-21T08:45:24.992953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:43.129359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993986+0000","last_change":"2024-04-21T08:43:01.687267+0000","last_active":"2024-04-21T08:45:24.993986+0000","last_peered":"2024-04-21T08:45:24.993986+0000","last_clean":"2024-04-21T08:45:24.993986+0000","last_became_active":"2024-04-21T08:43:01.686872+0000","last_became_peered":"2024-04-21T08:43:01.686872+0000","last_unstale":"2024-04-21T08:45:24.993986+0000","last_undegraded":"2024-04-21T08:45:24.993986+0000","last_fullsized":"2024-04-21T08:45:24.993986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:15:17.685123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904172+0000","last_change":"2024-04-21T08:42:56.634883+0000","last_active":"2024-04-21T08:45:24.904172+0000","last_peered":"2024-04-21T08:45:24.904172+0000","last_clean":"2024-04-21T08:45:24.904172+0000","last_became_active":"2024-04-21T08:42:56.634529+0000","last_became_peered":"2024-04-21T08:42:56.634529+0000","last_unstale":"2024-04-21T08:45:24.904172+0000","last_undegraded":"2024-04-21T08:45:24.904172+0000","last_fullsized":"2024-04-21T08:45:24.904172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:51:37.781031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992489+0000","last_change":"2024-04-21T08:43:46.472759+0000","last_active":"2024-04-21T08:45:24.992489+0000","last_peered":"2024-04-21T08:45:24.992489+0000","last_clean":"2024-04-21T08:45:24.992489+0000","last_became_active":"2024-04-21T08:43:46.472298+0000","last_became_peered":"2024-04-21T08:43:46.472298+0000","last_unstale":"2024-04-21T08:45:24.992489+0000","last_undegraded":"2024-04-21T08:45:24.992489+0000","last_fullsized":"2024-04-21T08:45:24.992489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:33.040004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992860+0000","last_change":"2024-04-21T08:43:01.686092+0000","last_active":"2024-04-21T08:45:24.992860+0000","last_peered":"2024-04-21T08:45:24.992860+0000","last_clean":"2024-04-21T08:45:24.992860+0000","last_became_active":"2024-04-21T08:43:01.685670+0000","last_became_peered":"2024-04-21T08:43:01.685670+0000","last_unstale":"2024-04-21T08:45:24.992860+0000","last_undegraded":"2024-04-21T08:45:24.992860+0000","last_fullsized":"2024-04-21T08:45:24.992860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:40:09.371714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992036+0000","last_change":"2024-04-21T08:42:56.635096+0000","last_active":"2024-04-21T08:45:24.992036+0000","last_peered":"2024-04-21T08:45:24.992036+0000","last_clean":"2024-04-21T08:45:24.992036+0000","last_became_active":"2024-04-21T08:42:56.634774+0000","last_became_peered":"2024-04-21T08:42:56.634774+0000","last_unstale":"2024-04-21T08:45:24.992036+0000","last_undegraded":"2024-04-21T08:45:24.992036+0000","last_fullsized":"2024-04-21T08:45:24.992036+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:11:16.027579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992792+0000","last_change":"2024-04-21T08:42:57.410146+0000","last_active":"2024-04-21T08:45:24.992792+0000","last_peered":"2024-04-21T08:45:24.992792+0000","last_clean":"2024-04-21T08:45:24.992792+0000","last_became_active":"2024-04-21T08:42:57.409752+0000","last_became_peered":"2024-04-21T08:42:57.409752+0000","last_unstale":"2024-04-21T08:45:24.992792+0000","last_undegraded":"2024-04-21T08:45:24.992792+0000","last_fullsized":"2024-04-21T08:45:24.992792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:39:16.834310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992835+0000","last_change":"2024-04-21T08:43:01.680000+0000","last_active":"2024-04-21T08:45:24.992835+0000","last_peered":"2024-04-21T08:45:24.992835+0000","last_clean":"2024-04-21T08:45:24.992835+0000","last_became_active":"2024-04-21T08:43:01.679516+0000","last_became_peered":"2024-04-21T08:43:01.679516+0000","last_unstale":"2024-04-21T08:45:24.992835+0000","last_undegraded":"2024-04-21T08:45:24.992835+0000","last_fullsized":"2024-04-21T08:45:24.992835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:53:37.188903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992275+0000","last_change":"2024-04-21T08:42:56.640827+0000","last_active":"2024-04-21T08:45:24.992275+0000","last_peered":"2024-04-21T08:45:24.992275+0000","last_clean":"2024-04-21T08:45:24.992275+0000","last_became_active":"2024-04-21T08:42:56.640292+0000","last_became_peered":"2024-04-21T08:42:56.640292+0000","last_unstale":"2024-04-21T08:45:24.992275+0000","last_undegraded":"2024-04-21T08:45:24.992275+0000","last_fullsized":"2024-04-21T08:45:24.992275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:12:10.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993653+0000","last_change":"2024-04-21T08:43:46.472625+0000","last_active":"2024-04-21T08:45:24.993653+0000","last_peered":"2024-04-21T08:45:24.993653+0000","last_clean":"2024-04-21T08:45:24.993653+0000","last_became_active":"2024-04-21T08:43:46.472150+0000","last_became_peered":"2024-04-21T08:43:46.472150+0000","last_unstale":"2024-04-21T08:45:24.993653+0000","last_undegraded":"2024-04-21T08:45:24.993653+0000","last_fullsized":"2024-04-21T08:45:24.993653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:25:59.242517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992241+0000","last_change":"2024-04-21T08:43:01.720950+0000","last_active":"2024-04-21T08:45:24.992241+0000","last_peered":"2024-04-21T08:45:24.992241+0000","last_clean":"2024-04-21T08:45:24.992241+0000","last_became_active":"2024-04-21T08:43:01.720536+0000","last_became_peered":"2024-04-21T08:43:01.720536+0000","last_unstale":"2024-04-21T08:45:24.992241+0000","last_undegraded":"2024-04-21T08:45:24.992241+0000","last_fullsized":"2024-04-21T08:45:24.992241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:31:03.366865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993525+0000","last_change":"2024-04-21T08:42:56.641867+0000","last_active":"2024-04-21T08:45:24.993525+0000","last_peered":"2024-04-21T08:45:24.993525+0000","last_clean":"2024-04-21T08:45:24.993525+0000","last_became_active":"2024-04-21T08:42:56.641526+0000","last_became_peered":"2024-04-21T08:42:56.641526+0000","last_unstale":"2024-04-21T08:45:24.993525+0000","last_undegraded":"2024-04-21T08:45:24.993525+0000","last_fullsized":"2024-04-21T08:45:24.993525+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:40:54.363517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993389+0000","last_change":"2024-04-21T08:42:57.408422+0000","last_active":"2024-04-21T08:45:24.993389+0000","last_peered":"2024-04-21T08:45:24.993389+0000","last_clean":"2024-04-21T08:45:24.993389+0000","last_became_active":"2024-04-21T08:42:57.408012+0000","last_became_peered":"2024-04-21T08:42:57.408012+0000","last_unstale":"2024-04-21T08:45:24.993389+0000","last_undegraded":"2024-04-21T08:45:24.993389+0000","last_fullsized":"2024-04-21T08:45:24.993389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:37:40.352449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992442+0000","last_change":"2024-04-21T08:43:01.680477+0000","last_active":"2024-04-21T08:45:24.992442+0000","last_peered":"2024-04-21T08:45:24.992442+0000","last_clean":"2024-04-21T08:45:24.992442+0000","last_became_active":"2024-04-21T08:43:01.679887+0000","last_became_peered":"2024-04-21T08:43:01.679887+0000","last_unstale":"2024-04-21T08:45:24.992442+0000","last_undegraded":"2024-04-21T08:45:24.992442+0000","last_fullsized":"2024-04-21T08:45:24.992442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:49:25.644424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992746+0000","last_change":"2024-04-21T08:42:56.643462+0000","last_active":"2024-04-21T08:45:24.992746+0000","last_peered":"2024-04-21T08:45:24.992746+0000","last_clean":"2024-04-21T08:45:24.992746+0000","last_became_active":"2024-04-21T08:42:56.643155+0000","last_became_peered":"2024-04-21T08:42:56.643155+0000","last_unstale":"2024-04-21T08:45:24.992746+0000","last_undegraded":"2024-04-21T08:45:24.992746+0000","last_fullsized":"2024-04-21T08:45:24.992746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:04:11.339143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992763+0000","last_change":"2024-04-21T08:42:57.411457+0000","last_active":"2024-04-21T08:45:24.992763+0000","last_peered":"2024-04-21T08:45:24.992763+0000","last_clean":"2024-04-21T08:45:24.992763+0000","last_became_active":"2024-04-21T08:42:57.411024+0000","last_became_peered":"2024-04-21T08:42:57.411024+0000","last_unstale":"2024-04-21T08:45:24.992763+0000","last_undegraded":"2024-04-21T08:45:24.992763+0000","last_fullsized":"2024-04-21T08:45:24.992763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:15:14.909847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992403+0000","last_change":"2024-04-21T08:43:01.720664+0000","last_active":"2024-04-21T08:45:24.992403+0000","last_peered":"2024-04-21T08:45:24.992403+0000","last_clean":"2024-04-21T08:45:24.992403+0000","last_became_active":"2024-04-21T08:43:01.720171+0000","last_became_peered":"2024-04-21T08:43:01.720171+0000","last_unstale":"2024-04-21T08:45:24.992403+0000","last_undegraded":"2024-04-21T08:45:24.992403+0000","last_fullsized":"2024-04-21T08:45:24.992403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:00:15.331828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992847+0000","last_change":"2024-04-21T08:42:57.409234+0000","last_active":"2024-04-21T08:45:24.992847+0000","last_peered":"2024-04-21T08:45:24.992847+0000","last_clean":"2024-04-21T08:45:24.992847+0000","last_became_active":"2024-04-21T08:42:57.408917+0000","last_became_peered":"2024-04-21T08:42:57.408917+0000","last_unstale":"2024-04-21T08:45:24.992847+0000","last_undegraded":"2024-04-21T08:45:24.992847+0000","last_fullsized":"2024-04-21T08:45:24.992847+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:24:34.554034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992175+0000","last_change":"2024-04-21T08:42:56.643640+0000","last_active":"2024-04-21T08:45:24.992175+0000","last_peered":"2024-04-21T08:45:24.992175+0000","last_clean":"2024-04-21T08:45:24.992175+0000","last_became_active":"2024-04-21T08:42:56.643297+0000","last_became_peered":"2024-04-21T08:42:56.643297+0000","last_unstale":"2024-04-21T08:45:24.992175+0000","last_undegraded":"2024-04-21T08:45:24.992175+0000","last_fullsized":"2024-04-21T08:45:24.992175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:57:49.536250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993373+0000","last_change":"2024-04-21T08:43:01.674292+0000","last_active":"2024-04-21T08:45:24.993373+0000","last_peered":"2024-04-21T08:45:24.993373+0000","last_clean":"2024-04-21T08:45:24.993373+0000","last_became_active":"2024-04-21T08:43:01.674004+0000","last_became_peered":"2024-04-21T08:43:01.674004+0000","last_unstale":"2024-04-21T08:45:24.993373+0000","last_undegraded":"2024-04-21T08:45:24.993373+0000","last_fullsized":"2024-04-21T08:45:24.993373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:56:36.841028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904114+0000","last_change":"2024-04-21T08:42:57.408176+0000","last_active":"2024-04-21T08:45:24.904114+0000","last_peered":"2024-04-21T08:45:24.904114+0000","last_clean":"2024-04-21T08:45:24.904114+0000","last_became_active":"2024-04-21T08:42:57.407353+0000","last_became_peered":"2024-04-21T08:42:57.407353+0000","last_unstale":"2024-04-21T08:45:24.904114+0000","last_undegraded":"2024-04-21T08:45:24.904114+0000","last_fullsized":"2024-04-21T08:45:24.904114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:29:31.280524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904460+0000","last_change":"2024-04-21T08:42:56.629629+0000","last_active":"2024-04-21T08:45:24.904460+0000","last_peered":"2024-04-21T08:45:24.904460+0000","last_clean":"2024-04-21T08:45:24.904460+0000","last_became_active":"2024-04-21T08:42:56.629249+0000","last_became_peered":"2024-04-21T08:42:56.629249+0000","last_unstale":"2024-04-21T08:45:24.904460+0000","last_undegraded":"2024-04-21T08:45:24.904460+0000","last_fullsized":"2024-04-21T08:45:24.904460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:49.820241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"94'148","reported_seq":412,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:27.016330+0000","last_change":"2024-04-21T08:41:39.277599+0000","last_active":"2024-04-21T08:45:27.016330+0000","last_peered":"2024-04-21T08:45:27.016330+0000","last_clean":"2024-04-21T08:45:27.016330+0000","last_became_active":"2024-04-21T08:41:39.277170+0000","last_became_peered":"2024-04-21T08:41:39.277170+0000","last_unstale":"2024-04-21T08:45:27.016330+0000","last_undegraded":"2024-04-21T08:45:27.016330+0000","last_fullsized":"2024-04-21T08:45:27.016330+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:38:09.371087+0000","last_clean_scrub_stamp":"2024-04-21T08:38:09.371087+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:07:46.367157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992538+0000","last_change":"2024-04-21T08:43:01.721332+0000","last_active":"2024-04-21T08:45:24.992538+0000","last_peered":"2024-04-21T08:45:24.992538+0000","last_clean":"2024-04-21T08:45:24.992538+0000","last_became_active":"2024-04-21T08:43:01.720644+0000","last_became_peered":"2024-04-21T08:43:01.720644+0000","last_unstale":"2024-04-21T08:45:24.992538+0000","last_undegraded":"2024-04-21T08:45:24.992538+0000","last_fullsized":"2024-04-21T08:45:24.992538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:32:38.217334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992034+0000","last_change":"2024-04-21T08:42:57.408592+0000","last_active":"2024-04-21T08:45:24.992034+0000","last_peered":"2024-04-21T08:45:24.992034+0000","last_clean":"2024-04-21T08:45:24.992034+0000","last_became_active":"2024-04-21T08:42:57.408198+0000","last_became_peered":"2024-04-21T08:42:57.408198+0000","last_unstale":"2024-04-21T08:45:24.992034+0000","last_undegraded":"2024-04-21T08:45:24.992034+0000","last_fullsized":"2024-04-21T08:45:24.992034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:13:49.141191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992367+0000","last_change":"2024-04-21T08:42:56.639866+0000","last_active":"2024-04-21T08:45:24.992367+0000","last_peered":"2024-04-21T08:45:24.992367+0000","last_clean":"2024-04-21T08:45:24.992367+0000","last_became_active":"2024-04-21T08:42:56.639463+0000","last_became_peered":"2024-04-21T08:42:56.639463+0000","last_unstale":"2024-04-21T08:45:24.992367+0000","last_undegraded":"2024-04-21T08:45:24.992367+0000","last_fullsized":"2024-04-21T08:45:24.992367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:42:49.131141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993257+0000","last_change":"2024-04-21T08:43:01.673865+0000","last_active":"2024-04-21T08:45:24.993257+0000","last_peered":"2024-04-21T08:45:24.993257+0000","last_clean":"2024-04-21T08:45:24.993257+0000","last_became_active":"2024-04-21T08:43:01.673442+0000","last_became_peered":"2024-04-21T08:43:01.673442+0000","last_unstale":"2024-04-21T08:45:24.993257+0000","last_undegraded":"2024-04-21T08:45:24.993257+0000","last_fullsized":"2024-04-21T08:45:24.993257+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:59:09.610323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992262+0000","last_change":"2024-04-21T08:42:56.634540+0000","last_active":"2024-04-21T08:45:24.992262+0000","last_peered":"2024-04-21T08:45:24.992262+0000","last_clean":"2024-04-21T08:45:24.992262+0000","last_became_active":"2024-04-21T08:42:56.634208+0000","last_became_peered":"2024-04-21T08:42:56.634208+0000","last_unstale":"2024-04-21T08:45:24.992262+0000","last_undegraded":"2024-04-21T08:45:24.992262+0000","last_fullsized":"2024-04-21T08:45:24.992262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:49:50.725669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992695+0000","last_change":"2024-04-21T08:42:57.396437+0000","last_active":"2024-04-21T08:45:24.992695+0000","last_peered":"2024-04-21T08:45:24.992695+0000","last_clean":"2024-04-21T08:45:24.992695+0000","last_became_active":"2024-04-21T08:42:57.395911+0000","last_became_peered":"2024-04-21T08:42:57.395911+0000","last_unstale":"2024-04-21T08:45:24.992695+0000","last_undegraded":"2024-04-21T08:45:24.992695+0000","last_fullsized":"2024-04-21T08:45:24.992695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:17.369676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903778+0000","last_change":"2024-04-21T08:43:01.720910+0000","last_active":"2024-04-21T08:45:24.903778+0000","last_peered":"2024-04-21T08:45:24.903778+0000","last_clean":"2024-04-21T08:45:24.903778+0000","last_became_active":"2024-04-21T08:43:01.720121+0000","last_became_peered":"2024-04-21T08:43:01.720121+0000","last_unstale":"2024-04-21T08:45:24.903778+0000","last_undegraded":"2024-04-21T08:45:24.903778+0000","last_fullsized":"2024-04-21T08:45:24.903778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:14:34.017444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'68","reported_seq":154,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.658975+0000","last_change":"2024-04-21T08:42:56.636784+0000","last_active":"2024-04-21T08:45:30.658975+0000","last_peered":"2024-04-21T08:45:30.658975+0000","last_clean":"2024-04-21T08:45:30.658975+0000","last_became_active":"2024-04-21T08:42:56.636400+0000","last_became_peered":"2024-04-21T08:42:56.636400+0000","last_unstale":"2024-04-21T08:45:30.658975+0000","last_undegraded":"2024-04-21T08:45:30.658975+0000","last_fullsized":"2024-04-21T08:45:30.658975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:50:23.643081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":182,"num_read_kb":160,"num_write":122,"num_write_kb":1157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904474+0000","last_change":"2024-04-21T08:42:57.402723+0000","last_active":"2024-04-21T08:45:24.904474+0000","last_peered":"2024-04-21T08:45:24.904474+0000","last_clean":"2024-04-21T08:45:24.904474+0000","last_became_active":"2024-04-21T08:42:57.402360+0000","last_became_peered":"2024-04-21T08:42:57.402360+0000","last_unstale":"2024-04-21T08:45:24.904474+0000","last_undegraded":"2024-04-21T08:45:24.904474+0000","last_fullsized":"2024-04-21T08:45:24.904474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:02:45.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903473+0000","last_change":"2024-04-21T08:43:01.677074+0000","last_active":"2024-04-21T08:45:24.903473+0000","last_peered":"2024-04-21T08:45:24.903473+0000","last_clean":"2024-04-21T08:45:24.903473+0000","last_became_active":"2024-04-21T08:43:01.676489+0000","last_became_peered":"2024-04-21T08:43:01.676489+0000","last_unstale":"2024-04-21T08:45:24.903473+0000","last_undegraded":"2024-04-21T08:45:24.903473+0000","last_fullsized":"2024-04-21T08:45:24.903473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:21:42.211396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904682+0000","last_change":"2024-04-21T08:42:56.632240+0000","last_active":"2024-04-21T08:45:24.904682+0000","last_peered":"2024-04-21T08:45:24.904682+0000","last_clean":"2024-04-21T08:45:24.904682+0000","last_became_active":"2024-04-21T08:42:56.631900+0000","last_became_peered":"2024-04-21T08:42:56.631900+0000","last_unstale":"2024-04-21T08:45:24.904682+0000","last_undegraded":"2024-04-21T08:45:24.904682+0000","last_fullsized":"2024-04-21T08:45:24.904682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:18:26.761079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337838+0000","last_change":"2024-04-21T08:42:57.412033+0000","last_active":"2024-04-21T08:45:25.337838+0000","last_peered":"2024-04-21T08:45:25.337838+0000","last_clean":"2024-04-21T08:45:25.337838+0000","last_became_active":"2024-04-21T08:42:57.411675+0000","last_became_peered":"2024-04-21T08:42:57.411675+0000","last_unstale":"2024-04-21T08:45:25.337838+0000","last_undegraded":"2024-04-21T08:45:25.337838+0000","last_fullsized":"2024-04-21T08:45:25.337838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:34:01.848176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904527+0000","last_change":"2024-04-21T08:43:46.474488+0000","last_active":"2024-04-21T08:45:24.904527+0000","last_peered":"2024-04-21T08:45:24.904527+0000","last_clean":"2024-04-21T08:45:24.904527+0000","last_became_active":"2024-04-21T08:43:46.473517+0000","last_became_peered":"2024-04-21T08:43:46.473517+0000","last_unstale":"2024-04-21T08:45:24.904527+0000","last_undegraded":"2024-04-21T08:45:24.904527+0000","last_fullsized":"2024-04-21T08:45:24.904527+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:13:38.989025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991872+0000","last_change":"2024-04-21T08:42:56.628713+0000","last_active":"2024-04-21T08:45:24.991872+0000","last_peered":"2024-04-21T08:45:24.991872+0000","last_clean":"2024-04-21T08:45:24.991872+0000","last_became_active":"2024-04-21T08:42:56.628398+0000","last_became_peered":"2024-04-21T08:42:56.628398+0000","last_unstale":"2024-04-21T08:45:24.991872+0000","last_undegraded":"2024-04-21T08:45:24.991872+0000","last_fullsized":"2024-04-21T08:45:24.991872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:53:47.713438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.991968+0000","last_change":"2024-04-21T08:42:57.398844+0000","last_active":"2024-04-21T08:45:24.991968+0000","last_peered":"2024-04-21T08:45:24.991968+0000","last_clean":"2024-04-21T08:45:24.991968+0000","last_became_active":"2024-04-21T08:42:57.398529+0000","last_became_peered":"2024-04-21T08:42:57.398529+0000","last_unstale":"2024-04-21T08:45:24.991968+0000","last_undegraded":"2024-04-21T08:45:24.991968+0000","last_fullsized":"2024-04-21T08:45:24.991968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:41:20.282108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992202+0000","last_change":"2024-04-21T08:43:46.474478+0000","last_active":"2024-04-21T08:45:24.992202+0000","last_peered":"2024-04-21T08:45:24.992202+0000","last_clean":"2024-04-21T08:45:24.992202+0000","last_became_active":"2024-04-21T08:43:46.473985+0000","last_became_peered":"2024-04-21T08:43:46.473985+0000","last_unstale":"2024-04-21T08:45:24.992202+0000","last_undegraded":"2024-04-21T08:45:24.992202+0000","last_fullsized":"2024-04-21T08:45:24.992202+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:51:07.824156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903893+0000","last_change":"2024-04-21T08:42:56.642185+0000","last_active":"2024-04-21T08:45:24.903893+0000","last_peered":"2024-04-21T08:45:24.903893+0000","last_clean":"2024-04-21T08:45:24.903893+0000","last_became_active":"2024-04-21T08:42:56.641847+0000","last_became_peered":"2024-04-21T08:42:56.641847+0000","last_unstale":"2024-04-21T08:45:24.903893+0000","last_undegraded":"2024-04-21T08:45:24.903893+0000","last_fullsized":"2024-04-21T08:45:24.903893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:43:14.183268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992008+0000","last_change":"2024-04-21T08:42:57.403298+0000","last_active":"2024-04-21T08:45:24.992008+0000","last_peered":"2024-04-21T08:45:24.992008+0000","last_clean":"2024-04-21T08:45:24.992008+0000","last_became_active":"2024-04-21T08:42:57.402966+0000","last_became_peered":"2024-04-21T08:42:57.402966+0000","last_unstale":"2024-04-21T08:45:24.992008+0000","last_undegraded":"2024-04-21T08:45:24.992008+0000","last_fullsized":"2024-04-21T08:45:24.992008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:01:31.131142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904288+0000","last_change":"2024-04-21T08:43:01.681011+0000","last_active":"2024-04-21T08:45:24.904288+0000","last_peered":"2024-04-21T08:45:24.904288+0000","last_clean":"2024-04-21T08:45:24.904288+0000","last_became_active":"2024-04-21T08:43:01.679949+0000","last_became_peered":"2024-04-21T08:43:01.679949+0000","last_unstale":"2024-04-21T08:45:24.904288+0000","last_undegraded":"2024-04-21T08:45:24.904288+0000","last_fullsized":"2024-04-21T08:45:24.904288+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:31:19.581648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903726+0000","last_change":"2024-04-21T08:42:56.632533+0000","last_active":"2024-04-21T08:45:24.903726+0000","last_peered":"2024-04-21T08:45:24.903726+0000","last_clean":"2024-04-21T08:45:24.903726+0000","last_became_active":"2024-04-21T08:42:56.632230+0000","last_became_peered":"2024-04-21T08:42:56.632230+0000","last_unstale":"2024-04-21T08:45:24.903726+0000","last_undegraded":"2024-04-21T08:45:24.903726+0000","last_fullsized":"2024-04-21T08:45:24.903726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:33:15.519506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903977+0000","last_change":"2024-04-21T08:42:57.403165+0000","last_active":"2024-04-21T08:45:24.903977+0000","last_peered":"2024-04-21T08:45:24.903977+0000","last_clean":"2024-04-21T08:45:24.903977+0000","last_became_active":"2024-04-21T08:42:57.402781+0000","last_became_peered":"2024-04-21T08:42:57.402781+0000","last_unstale":"2024-04-21T08:45:24.903977+0000","last_undegraded":"2024-04-21T08:45:24.903977+0000","last_fullsized":"2024-04-21T08:45:24.903977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:04:59.587649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904319+0000","last_change":"2024-04-21T08:43:01.720566+0000","last_active":"2024-04-21T08:45:24.904319+0000","last_peered":"2024-04-21T08:45:24.904319+0000","last_clean":"2024-04-21T08:45:24.904319+0000","last_became_active":"2024-04-21T08:43:01.720059+0000","last_became_peered":"2024-04-21T08:43:01.720059+0000","last_unstale":"2024-04-21T08:45:24.904319+0000","last_undegraded":"2024-04-21T08:45:24.904319+0000","last_fullsized":"2024-04-21T08:45:24.904319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:21:40.726729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993372+0000","last_change":"2024-04-21T08:43:46.475880+0000","last_active":"2024-04-21T08:45:24.993372+0000","last_peered":"2024-04-21T08:45:24.993372+0000","last_clean":"2024-04-21T08:45:24.993372+0000","last_became_active":"2024-04-21T08:43:46.470622+0000","last_became_peered":"2024-04-21T08:43:46.470622+0000","last_unstale":"2024-04-21T08:45:24.993372+0000","last_undegraded":"2024-04-21T08:45:24.993372+0000","last_fullsized":"2024-04-21T08:45:24.993372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:19:35.310126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992918+0000","last_change":"2024-04-21T08:42:57.406165+0000","last_active":"2024-04-21T08:45:24.992918+0000","last_peered":"2024-04-21T08:45:24.992918+0000","last_clean":"2024-04-21T08:45:24.992918+0000","last_became_active":"2024-04-21T08:42:57.405824+0000","last_became_peered":"2024-04-21T08:42:57.405824+0000","last_unstale":"2024-04-21T08:45:24.992918+0000","last_undegraded":"2024-04-21T08:45:24.992918+0000","last_fullsized":"2024-04-21T08:45:24.992918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T10:39:17.338158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992146+0000","last_change":"2024-04-21T08:43:01.682722+0000","last_active":"2024-04-21T08:45:24.992146+0000","last_peered":"2024-04-21T08:45:24.992146+0000","last_clean":"2024-04-21T08:45:24.992146+0000","last_became_active":"2024-04-21T08:43:01.681893+0000","last_became_peered":"2024-04-21T08:43:01.681893+0000","last_unstale":"2024-04-21T08:45:24.992146+0000","last_undegraded":"2024-04-21T08:45:24.992146+0000","last_fullsized":"2024-04-21T08:45:24.992146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:36:16.359654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904004+0000","last_change":"2024-04-21T08:42:56.635104+0000","last_active":"2024-04-21T08:45:24.904004+0000","last_peered":"2024-04-21T08:45:24.904004+0000","last_clean":"2024-04-21T08:45:24.904004+0000","last_became_active":"2024-04-21T08:42:56.634782+0000","last_became_peered":"2024-04-21T08:42:56.634782+0000","last_unstale":"2024-04-21T08:45:24.904004+0000","last_undegraded":"2024-04-21T08:45:24.904004+0000","last_fullsized":"2024-04-21T08:45:24.904004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:04:56.081501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.401166+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.400783+0000","last_became_peered":"2024-04-21T08:42:57.400783+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T17:59:57.526019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337411+0000","last_change":"2024-04-21T08:43:01.681976+0000","last_active":"2024-04-21T08:45:25.337411+0000","last_peered":"2024-04-21T08:45:25.337411+0000","last_clean":"2024-04-21T08:45:25.337411+0000","last_became_active":"2024-04-21T08:43:01.681530+0000","last_became_peered":"2024-04-21T08:43:01.681530+0000","last_unstale":"2024-04-21T08:45:25.337411+0000","last_undegraded":"2024-04-21T08:45:25.337411+0000","last_fullsized":"2024-04-21T08:45:25.337411+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:34:15.651085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'57","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:30.654353+0000","last_change":"2024-04-21T08:42:56.645379+0000","last_active":"2024-04-21T08:45:30.654353+0000","last_peered":"2024-04-21T08:45:30.654353+0000","last_clean":"2024-04-21T08:45:30.654353+0000","last_became_active":"2024-04-21T08:42:56.645015+0000","last_became_peered":"2024-04-21T08:42:56.645015+0000","last_unstale":"2024-04-21T08:45:30.654353+0000","last_undegraded":"2024-04-21T08:45:30.654353+0000","last_fullsized":"2024-04-21T08:45:30.654353+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:12:23.022401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":20,"num_write":68,"num_write_kb":807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992998+0000","last_change":"2024-04-21T08:42:57.407821+0000","last_active":"2024-04-21T08:45:24.992998+0000","last_peered":"2024-04-21T08:45:24.992998+0000","last_clean":"2024-04-21T08:45:24.992998+0000","last_became_active":"2024-04-21T08:42:57.407469+0000","last_became_peered":"2024-04-21T08:42:57.407469+0000","last_unstale":"2024-04-21T08:45:24.992998+0000","last_undegraded":"2024-04-21T08:45:24.992998+0000","last_fullsized":"2024-04-21T08:45:24.992998+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:20:38.590789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992640+0000","last_change":"2024-04-21T08:43:01.677461+0000","last_active":"2024-04-21T08:45:24.992640+0000","last_peered":"2024-04-21T08:45:24.992640+0000","last_clean":"2024-04-21T08:45:24.992640+0000","last_became_active":"2024-04-21T08:43:01.677032+0000","last_became_peered":"2024-04-21T08:43:01.677032+0000","last_unstale":"2024-04-21T08:45:24.992640+0000","last_undegraded":"2024-04-21T08:45:24.992640+0000","last_fullsized":"2024-04-21T08:45:24.992640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:43:00.652650+0000","last_clean_scrub_stamp":"2024-04-21T08:43:00.652650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:47:06.366658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992686+0000","last_change":"2024-04-21T08:42:56.640838+0000","last_active":"2024-04-21T08:45:24.992686+0000","last_peered":"2024-04-21T08:45:24.992686+0000","last_clean":"2024-04-21T08:45:24.992686+0000","last_became_active":"2024-04-21T08:42:56.640488+0000","last_became_peered":"2024-04-21T08:42:56.640488+0000","last_unstale":"2024-04-21T08:45:24.992686+0000","last_undegraded":"2024-04-21T08:45:24.992686+0000","last_fullsized":"2024-04-21T08:45:24.992686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:55.615694+0000","last_clean_scrub_stamp":"2024-04-21T08:42:55.615694+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:52:59.848414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993176+0000","last_change":"2024-04-21T08:42:57.403267+0000","last_active":"2024-04-21T08:45:24.993176+0000","last_peered":"2024-04-21T08:45:24.993176+0000","last_clean":"2024-04-21T08:45:24.993176+0000","last_became_active":"2024-04-21T08:42:57.402825+0000","last_became_peered":"2024-04-21T08:42:57.402825+0000","last_unstale":"2024-04-21T08:45:24.993176+0000","last_undegraded":"2024-04-21T08:45:24.993176+0000","last_fullsized":"2024-04-21T08:45:24.993176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T19:10:32.151996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992191+0000","last_change":"2024-04-21T08:42:57.398854+0000","last_active":"2024-04-21T08:45:24.992191+0000","last_peered":"2024-04-21T08:45:24.992191+0000","last_clean":"2024-04-21T08:45:24.992191+0000","last_became_active":"2024-04-21T08:42:57.398481+0000","last_became_peered":"2024-04-21T08:42:57.398481+0000","last_unstale":"2024-04-21T08:45:24.992191+0000","last_undegraded":"2024-04-21T08:45:24.992191+0000","last_fullsized":"2024-04-21T08:45:24.992191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:25:51.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":49,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903722+0000","last_change":"2024-04-21T08:44:47.315087+0000","last_active":"2024-04-21T08:45:24.903722+0000","last_peered":"2024-04-21T08:45:24.903722+0000","last_clean":"2024-04-21T08:45:24.903722+0000","last_became_active":"2024-04-21T08:44:47.314647+0000","last_became_peered":"2024-04-21T08:44:47.314647+0000","last_unstale":"2024-04-21T08:45:24.903722+0000","last_undegraded":"2024-04-21T08:45:24.903722+0000","last_fullsized":"2024-04-21T08:45:24.903722+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T20:16:20.425332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903546+0000","last_change":"2024-04-21T08:42:57.406844+0000","last_active":"2024-04-21T08:45:24.903546+0000","last_peered":"2024-04-21T08:45:24.903546+0000","last_clean":"2024-04-21T08:45:24.903546+0000","last_became_active":"2024-04-21T08:42:57.405844+0000","last_became_peered":"2024-04-21T08:42:57.405844+0000","last_unstale":"2024-04-21T08:45:24.903546+0000","last_undegraded":"2024-04-21T08:45:24.903546+0000","last_fullsized":"2024-04-21T08:45:24.903546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:02:25.419913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:25.337673+0000","last_change":"2024-04-21T08:42:57.410344+0000","last_active":"2024-04-21T08:45:25.337673+0000","last_peered":"2024-04-21T08:45:25.337673+0000","last_clean":"2024-04-21T08:45:25.337673+0000","last_became_active":"2024-04-21T08:42:57.409980+0000","last_became_peered":"2024-04-21T08:42:57.409980+0000","last_unstale":"2024-04-21T08:45:25.337673+0000","last_undegraded":"2024-04-21T08:45:25.337673+0000","last_fullsized":"2024-04-21T08:45:25.337673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:35:21.174781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992553+0000","last_change":"2024-04-21T08:43:46.544928+0000","last_active":"2024-04-21T08:45:24.992553+0000","last_peered":"2024-04-21T08:45:24.992553+0000","last_clean":"2024-04-21T08:45:24.992553+0000","last_became_active":"2024-04-21T08:43:46.544513+0000","last_became_peered":"2024-04-21T08:43:46.544513+0000","last_unstale":"2024-04-21T08:45:24.992553+0000","last_undegraded":"2024-04-21T08:45:24.992553+0000","last_fullsized":"2024-04-21T08:45:24.992553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T15:48:25.701329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993393+0000","last_change":"2024-04-21T08:42:57.398315+0000","last_active":"2024-04-21T08:45:24.993393+0000","last_peered":"2024-04-21T08:45:24.993393+0000","last_clean":"2024-04-21T08:45:24.993393+0000","last_became_active":"2024-04-21T08:42:57.397971+0000","last_became_peered":"2024-04-21T08:42:57.397971+0000","last_unstale":"2024-04-21T08:45:24.993393+0000","last_undegraded":"2024-04-21T08:45:24.993393+0000","last_fullsized":"2024-04-21T08:45:24.993393+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T14:47:53.075527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904194+0000","last_change":"2024-04-21T08:42:57.401865+0000","last_active":"2024-04-21T08:45:24.904194+0000","last_peered":"2024-04-21T08:45:24.904194+0000","last_clean":"2024-04-21T08:45:24.904194+0000","last_became_active":"2024-04-21T08:42:57.401562+0000","last_became_peered":"2024-04-21T08:42:57.401562+0000","last_unstale":"2024-04-21T08:45:24.904194+0000","last_undegraded":"2024-04-21T08:45:24.904194+0000","last_fullsized":"2024-04-21T08:45:24.904194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T13:57:30.530524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.904716+0000","last_change":"2024-04-21T08:42:57.411339+0000","last_active":"2024-04-21T08:45:24.904716+0000","last_peered":"2024-04-21T08:45:24.904716+0000","last_clean":"2024-04-21T08:45:24.904716+0000","last_became_active":"2024-04-21T08:42:57.411044+0000","last_became_peered":"2024-04-21T08:42:57.411044+0000","last_unstale":"2024-04-21T08:45:24.904716+0000","last_undegraded":"2024-04-21T08:45:24.904716+0000","last_fullsized":"2024-04-21T08:45:24.904716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T12:52:51.342672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992795+0000","last_change":"2024-04-21T08:42:57.396533+0000","last_active":"2024-04-21T08:45:24.992795+0000","last_peered":"2024-04-21T08:45:24.992795+0000","last_clean":"2024-04-21T08:45:24.992795+0000","last_became_active":"2024-04-21T08:42:57.396236+0000","last_became_peered":"2024-04-21T08:42:57.396236+0000","last_unstale":"2024-04-21T08:45:24.992795+0000","last_undegraded":"2024-04-21T08:45:24.992795+0000","last_fullsized":"2024-04-21T08:45:24.992795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:29:45.535855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.903992+0000","last_change":"2024-04-21T08:42:57.401204+0000","last_active":"2024-04-21T08:45:24.903992+0000","last_peered":"2024-04-21T08:45:24.903992+0000","last_clean":"2024-04-21T08:45:24.903992+0000","last_became_active":"2024-04-21T08:42:57.400257+0000","last_became_peered":"2024-04-21T08:42:57.400257+0000","last_unstale":"2024-04-21T08:45:24.903992+0000","last_undegraded":"2024-04-21T08:45:24.903992+0000","last_fullsized":"2024-04-21T08:45:24.903992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T11:02:28.050451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992360+0000","last_change":"2024-04-21T08:42:57.401177+0000","last_active":"2024-04-21T08:45:24.992360+0000","last_peered":"2024-04-21T08:45:24.992360+0000","last_clean":"2024-04-21T08:45:24.992360+0000","last_became_active":"2024-04-21T08:42:57.400823+0000","last_became_peered":"2024-04-21T08:42:57.400823+0000","last_unstale":"2024-04-21T08:45:24.992360+0000","last_undegraded":"2024-04-21T08:45:24.992360+0000","last_fullsized":"2024-04-21T08:45:24.992360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T16:02:43.409230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.992585+0000","last_change":"2024-04-21T08:42:57.399405+0000","last_active":"2024-04-21T08:45:24.992585+0000","last_peered":"2024-04-21T08:45:24.992585+0000","last_clean":"2024-04-21T08:45:24.992585+0000","last_became_active":"2024-04-21T08:42:57.399011+0000","last_became_peered":"2024-04-21T08:42:57.399011+0000","last_unstale":"2024-04-21T08:45:24.992585+0000","last_undegraded":"2024-04-21T08:45:24.992585+0000","last_fullsized":"2024-04-21T08:45:24.992585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T18:01:41.907871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-21T08:45:24.993390+0000","last_change":"2024-04-21T08:42:57.410706+0000","last_active":"2024-04-21T08:45:24.993390+0000","last_peered":"2024-04-21T08:45:24.993390+0000","last_clean":"2024-04-21T08:45:24.993390+0000","last_became_active":"2024-04-21T08:42:57.410387+0000","last_became_peered":"2024-04-21T08:42:57.410387+0000","last_unstale":"2024-04-21T08:45:24.993390+0000","last_undegraded":"2024-04-21T08:45:24.993390+0000","last_fullsized":"2024-04-21T08:45:24.993390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-21T08:42:56.617157+0000","last_clean_scrub_stamp":"2024-04-21T08:42:56.617157+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-22T09:12:43.078420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":110016,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":792314,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":476,"num_read_kb":388,"num_write":264,"num_write_kb":2062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":2376942,"data_compressed":107229,"data_compressed_allocated":147456,"data_compressed_original":2125824,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1032192,"data_stored":528781,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1179648,"data_stored":1316881,"data_compressed":74099,"data_compressed_allocated":98304,"data_compressed_original":802816,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":926641,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1060864,"data_stored":922567,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":1123793,"data_compressed":66315,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495687,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":524651,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":524741,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1097728,"data_stored":594719,"data_compressed":31271,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":1246203,"data_compressed":63758,"data_compressed_allocated":86016,"data_compressed_original":782336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":528769,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":1447529,"data_compressed":81155,"data_compressed_allocated":106496,"data_compressed_original":978944,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36672,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":70068,"data_compressed":1628,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":721552,"data_compressed":34115,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":328414,"data_compressed":14840,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393820,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19275,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":398494,"data_compressed":16468,"data_compressed_allocated":24576,"data_compressed_original":315392,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-21T08:45:36.068 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:36.212 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 -- 172.21.15.57:0/2841557928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 msgr2=0x7f3be8107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 --2- 172.21.15.57:0/2841557928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be8107330 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009900 tx=0x7f3be402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 -- 172.21.15.57:0/2841557928 shutdown_connections 2024-04-21T08:45:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 --2- 172.21.15.57:0/2841557928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 0x7f3be810ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 --2- 172.21.15.57:0/2841557928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 0x7f3be8069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 --2- 172.21.15.57:0/2841557928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be8107330 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 -- 172.21.15.57:0/2841557928 >> 172.21.15.57:0/2841557928 conn(0x7f3be8077e30 msgr2=0x7f3be8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 -- 172.21.15.57:0/2841557928 shutdown_connections 2024-04-21T08:45:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 -- 172.21.15.57:0/2841557928 wait complete. 2024-04-21T08:45:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 Processor -- start 2024-04-21T08:45:36.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.207+0000 7f3bef375640 1 -- start start 2024-04-21T08:45:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bef375640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 0x7f3be8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bef375640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 0x7f3be8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bef375640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bef375640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8112170 con 0x7f3be8106f50 2024-04-21T08:45:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bef375640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8111ff0 con 0x7f3be8069aa0 2024-04-21T08:45:36.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bef375640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81122f0 con 0x7f3be80690e0 2024-04-21T08:45:36.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed111640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 0x7f3be8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55582/0 (socket says 172.21.15.57:55582) 2024-04-21T08:45:36.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 -- 172.21.15.57:0/3840702206 learned_addr learned my addr 172.21.15.57:0/3840702206 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:36.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bec910640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 0x7f3be8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 -- 172.21.15.57:0/3840702206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 msgr2=0x7f3be8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 0x7f3be8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 -- 172.21.15.57:0/3840702206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 msgr2=0x7f3be8198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 0x7f3be8198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 -- 172.21.15.57:0/3840702206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be819d6d0 con 0x7f3be8106f50 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bec910640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 0x7f3be8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed111640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 0x7f3be8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:36.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.208+0000 7f3bed912640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be819cfa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009fd0 tx=0x7f3be000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.209+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0017070 con 0x7f3be8106f50 2024-04-21T08:45:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.209+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3be0003e20 con 0x7f3be8106f50 2024-04-21T08:45:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.209+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be819d9c0 con 0x7f3be8106f50 2024-04-21T08:45:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.209+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0010460 con 0x7f3be8106f50 2024-04-21T08:45:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.209+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be81a5250 con 0x7f3be8106f50 2024-04-21T08:45:36.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.210+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be8068de0 con 0x7f3be8106f50 2024-04-21T08:45:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.211+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f3be0003970 con 0x7f3be8106f50 2024-04-21T08:45:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.211+0000 7f3bde7fc640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3bd40a3db0 0x7f3bd40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.211+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f3be00d0ac0 con 0x7f3be8106f50 2024-04-21T08:45:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.212+0000 7f3bed111640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3bd40a3db0 0x7f3bd40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.212+0000 7f3bed111640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3bd40a3db0 0x7f3bd40a6270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3be4006010 tx=0x7f3be402c6c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:36.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.216+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3be009cc90 con 0x7f3be8106f50 2024-04-21T08:45:36.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.445+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be810a4e0 con 0x7f3be8106f50 2024-04-21T08:45:36.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.446+0000 7f3bde7fc640 1 -- 172.21.15.57:0/3840702206 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f3be009c590 con 0x7f3be8106f50 2024-04-21T08:45:36.446 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:36.447 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:36.447 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3bd40a3db0 msgr2=0x7f3bd40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3bd40a3db0 0x7f3bd40a6270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3be4006010 tx=0x7f3be402c6c0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 msgr2=0x7f3be819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be819cfa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3be0009fd0 tx=0x7f3be000b260 comp rx=0 tx=0).stop 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 shutdown_connections 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3bd40a3db0 0x7f3bd40a6270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3be8106f50 0x7f3be819cfa0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be8069aa0 0x7f3be8198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 --2- 172.21.15.57:0/3840702206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be80690e0 0x7f3be8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.448+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 >> 172.21.15.57:0/3840702206 conn(0x7f3be8077e30 msgr2=0x7f3be80fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:36.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.449+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 shutdown_connections 2024-04-21T08:45:36.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.449+0000 7f3bef375640 1 -- 172.21.15.57:0/3840702206 wait complete. 2024-04-21T08:45:36.464 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 -- 172.21.15.57:0/467661224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1028101b40 msgr2=0x7f102810dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 --2- 172.21.15.57:0/467661224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1028101b40 0x7f102810dcb0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1020009ca0 tx=0x7f102002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 -- 172.21.15.57:0/467661224 shutdown_connections 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 --2- 172.21.15.57:0/467661224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1028101b40 0x7f102810dcb0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 --2- 172.21.15.57:0/467661224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10281011a0 0x7f1028101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 --2- 172.21.15.57:0/467661224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028105ea0 0x7f1028106280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.569+0000 7f102d91e640 1 -- 172.21.15.57:0/467661224 >> 172.21.15.57:0/467661224 conn(0x7f1028077e30 msgr2=0x7f1028078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:36.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.570+0000 7f102d91e640 1 -- 172.21.15.57:0/467661224 shutdown_connections 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.570+0000 7f102d91e640 1 -- 172.21.15.57:0/467661224 wait complete. 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.570+0000 7f102d91e640 1 Processor -- start 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.570+0000 7f102d91e640 1 -- start start 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.570+0000 7f102d91e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 0x7f1028197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.570+0000 7f102d91e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028101b40 0x7f1028197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102d91e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1028105ea0 0x7f102819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102d91e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10281110d0 con 0x7f10281011a0 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102d91e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1028110f50 con 0x7f1028105ea0 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102d91e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1028111250 con 0x7f1028101b40 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 0x7f1028197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 0x7f1028197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55594/0 (socket says 172.21.15.57:55594) 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 -- 172.21.15.57:0/2706192531 learned_addr learned my addr 172.21.15.57:0/2706192531 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f10267fc640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028101b40 0x7f1028197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f10277fe640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1028105ea0 0x7f102819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 -- 172.21.15.57:0/2706192531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028101b40 msgr2=0x7f1028197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028101b40 0x7f1028197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 -- 172.21.15.57:0/2706192531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1028105ea0 msgr2=0x7f102819be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1028105ea0 0x7f102819be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 -- 172.21.15.57:0/2706192531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102819c560 con 0x7f10281011a0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f10277fe640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1028105ea0 0x7f102819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f10267fc640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028101b40 0x7f1028197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f1026ffd640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 0x7f1028197570 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f101800c920 tx=0x7f101800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018007c70 con 0x7f10281011a0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1018007e10 con 0x7f10281011a0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1018018820 con 0x7f10281011a0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102819c850 con 0x7f10281011a0 2024-04-21T08:45:36.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.571+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10281a40e0 con 0x7f10281011a0 2024-04-21T08:45:36.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.573+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1028069230 con 0x7f10281011a0 2024-04-21T08:45:36.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.574+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f101800f450 con 0x7f10281011a0 2024-04-21T08:45:36.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.574+0000 7f102c91c640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10100a3db0 0x7f10100a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.575+0000 7f10267fc640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10100a3db0 0x7f10100a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.575+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f10180d0e30 con 0x7f10281011a0 2024-04-21T08:45:36.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.575+0000 7f10267fc640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10100a3db0 0x7f10100a6270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f101c0058c0 tx=0x7f101c005850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:36.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.579+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1018014030 con 0x7f10281011a0 2024-04-21T08:45:36.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:36 smithi136 ceph-mon[31791]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 73 KiB/s wr, 34 op/s 2024-04-21T08:45:36.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:36 smithi136 ceph-mon[31791]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:36.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.777+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10281988c0 con 0x7f10281011a0 2024-04-21T08:45:36.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.779+0000 7f102c91c640 1 -- 172.21.15.57:0/2706192531 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25074 (secure 0 0 0) 0x7f101809d000 con 0x7f10281011a0 2024-04-21T08:45:36.780 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:36.780 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:45:24.897611+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10100a3db0 msgr2=0x7f10100a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10100a3db0 0x7f10100a6270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f101c0058c0 tx=0x7f101c005850 comp rx=0 tx=0).stop 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 msgr2=0x7f1028197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 0x7f1028197570 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f101800c920 tx=0x7f101800cde0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 shutdown_connections 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10100a3db0 0x7f10100a6270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1028105ea0 0x7f102819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1028101b40 0x7f1028197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 --2- 172.21.15.57:0/2706192531 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10281011a0 0x7f1028197570 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 >> 172.21.15.57:0/2706192531 conn(0x7f1028077e30 msgr2=0x7f10280febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:36.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 shutdown_connections 2024-04-21T08:45:36.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.782+0000 7f102d91e640 1 -- 172.21.15.57:0/2706192531 wait complete. 2024-04-21T08:45:36.793 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:36 smithi057 ceph-mon[27968]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 73 KiB/s wr, 34 op/s 2024-04-21T08:45:36.794 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:36 smithi057 ceph-mon[27968]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:36.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:45:36.796 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:45:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:36 smithi129 ceph-mon[31272]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 73 KiB/s wr, 34 op/s 2024-04-21T08:45:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:36 smithi129 ceph-mon[31272]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 -- 172.21.15.57:0/950383185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 msgr2=0x7fc23c10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 --2- 172.21.15.57:0/950383185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c10e9b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc234009ca0 tx=0x7fc23402c230 comp rx=0 tx=0).stop 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 -- 172.21.15.57:0/950383185 shutdown_connections 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 --2- 172.21.15.57:0/950383185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c10e9b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 --2- 172.21.15.57:0/950383185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c103af0 0x7fc23c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 --2- 172.21.15.57:0/950383185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc23c1031d0 0x7fc23c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 -- 172.21.15.57:0/950383185 >> 172.21.15.57:0/950383185 conn(0x7fc23c077f50 msgr2=0x7fc23c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 -- 172.21.15.57:0/950383185 shutdown_connections 2024-04-21T08:45:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.901+0000 7fc242b8f640 1 -- 172.21.15.57:0/950383185 wait complete. 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 Processor -- start 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 -- start start 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c1031d0 0x7fc23c1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc23c103af0 0x7fc23c198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c1121a0 con 0x7fc23c10c510 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c112020 con 0x7fc23c103af0 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc242b8f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c112320 con 0x7fc23c1031d0 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc24112c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc24112c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55606/0 (socket says 172.21.15.57:55606) 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc24112c640 1 -- 172.21.15.57:0/905013219 learned_addr learned my addr 172.21.15.57:0/905013219 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.902+0000 7fc24092b640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c1031d0 0x7fc23c1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc23bfff640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc23c103af0 0x7fc23c198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24112c640 1 -- 172.21.15.57:0/905013219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c1031d0 msgr2=0x7fc23c1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24112c640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c1031d0 0x7fc23c1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24112c640 1 -- 172.21.15.57:0/905013219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc23c103af0 msgr2=0x7fc23c198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24112c640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc23c103af0 0x7fc23c198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24112c640 1 -- 172.21.15.57:0/905013219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c19d6d0 con 0x7fc23c10c510 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24092b640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c1031d0 0x7fc23c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc24112c640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c19cfa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc23402c840 tx=0x7fc234003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234039040 con 0x7fc23c10c510 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc234044070 con 0x7fc23c10c510 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23c19d960 con 0x7fc23c10c510 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2340329c0 con 0x7fc23c10c510 2024-04-21T08:45:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.903+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc23c1a5250 con 0x7fc23c10c510 2024-04-21T08:45:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.905+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23c068d70 con 0x7fc23c10c510 2024-04-21T08:45:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.906+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fc234032b60 con 0x7fc23c10c510 2024-04-21T08:45:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.906+0000 7fc239ffb640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc22c0a3db0 0x7fc22c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.906+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fc2340f3060 con 0x7fc23c10c510 2024-04-21T08:45:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.906+0000 7fc24092b640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc22c0a3db0 0x7fc22c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.907+0000 7fc24092b640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc22c0a3db0 0x7fc22c0a6270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc23000bfd0 tx=0x7fc23000b390 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:36.911+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc234032e50 con 0x7fc23c10c510 2024-04-21T08:45:37.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.097+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc23c107690 con 0x7fc23c10c510 2024-04-21T08:45:37.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.098+0000 7fc239ffb640 1 -- 172.21.15.57:0/905013219 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fc234035e20 con 0x7fc23c10c510 2024-04-21T08:45:37.098 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc22c0a3db0 msgr2=0x7fc22c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc22c0a3db0 0x7fc22c0a6270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc23000bfd0 tx=0x7fc23000b390 comp rx=0 tx=0).stop 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 msgr2=0x7fc23c19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c19cfa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc23402c840 tx=0x7fc234003c20 comp rx=0 tx=0).stop 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 shutdown_connections 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc22c0a3db0 0x7fc22c0a6270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc23c10c510 0x7fc23c19cfa0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc23c103af0 0x7fc23c198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 --2- 172.21.15.57:0/905013219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c1031d0 0x7fc23c1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 >> 172.21.15.57:0/905013219 conn(0x7fc23c077f50 msgr2=0x7fc23c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 shutdown_connections 2024-04-21T08:45:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.100+0000 7fc242b8f640 1 -- 172.21.15.57:0/905013219 wait complete. 2024-04-21T08:45:37.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 -- 172.21.15.57:0/1479144494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 msgr2=0x7f03081035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 --2- 172.21.15.57:0/1479144494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f03081035b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0304009900 tx=0x7f030402bd60 comp rx=0 tx=0).stop 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 -- 172.21.15.57:0/1479144494 shutdown_connections 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 --2- 172.21.15.57:0/1479144494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f030810c510 0x7f030810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 --2- 172.21.15.57:0/1479144494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0308103af0 0x7f030810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 --2- 172.21.15.57:0/1479144494 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f03081035b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 -- 172.21.15.57:0/1479144494 >> 172.21.15.57:0/1479144494 conn(0x7f0308077f50 msgr2=0x7f0308078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 -- 172.21.15.57:0/1479144494 shutdown_connections 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.216+0000 7f031002e640 1 -- 172.21.15.57:0/1479144494 wait complete. 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 Processor -- start 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 -- start start 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f0308100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0308103af0 0x7f0308101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030810c510 0x7f03081018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0308112180 con 0x7f03081031d0 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0308112000 con 0x7f030810c510 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f031002e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0308112300 con 0x7f0308103af0 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f0308100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f0308100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55618/0 (socket says 172.21.15.57:55618) 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 -- 172.21.15.57:0/736714868 learned_addr learned my addr 172.21.15.57:0/736714868 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:37.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030e5cb640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030810c510 0x7f03081018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030d5c9640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0308103af0 0x7f0308101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 -- 172.21.15.57:0/736714868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0308103af0 msgr2=0x7f0308101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0308103af0 0x7f0308101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 -- 172.21.15.57:0/736714868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030810c510 msgr2=0x7f03081018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030810c510 0x7f03081018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.217+0000 7f030ddca640 1 -- 172.21.15.57:0/736714868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03081a6c40 con 0x7f03081031d0 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f030d5c9640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0308103af0 0x7f0308101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f030ddca640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f0308100e20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0304005940 tx=0x7f030402d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304043020 con 0x7f03081031d0 2024-04-21T08:45:37.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f030402dd20 con 0x7f03081031d0 2024-04-21T08:45:37.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03081a6ed0 con 0x7f03081031d0 2024-04-21T08:45:37.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304003b80 con 0x7f03081031d0 2024-04-21T08:45:37.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.218+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03081a7390 con 0x7f03081031d0 2024-04-21T08:45:37.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.219+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0308068d70 con 0x7f03081031d0 2024-04-21T08:45:37.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.220+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f0304046020 con 0x7f03081031d0 2024-04-21T08:45:37.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.221+0000 7f02feffd640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02f40a3db0 0x7f02f40a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.221+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f03040f3460 con 0x7f03081031d0 2024-04-21T08:45:37.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.221+0000 7f030d5c9640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02f40a3db0 0x7f02f40a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.222+0000 7f030d5c9640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02f40a3db0 0x7f02f40a6270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03081028b0 tx=0x7f02f8005e50 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:37.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.225+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f030404d020 con 0x7f03081031d0 2024-04-21T08:45:37.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.415+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0308107100 con 0x7f03081031d0 2024-04-21T08:45:37.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.415+0000 7f02feffd640 1 -- 172.21.15.57:0/736714868 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f03040bf7e0 con 0x7f03081031d0 2024-04-21T08:45:37.416 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02f40a3db0 msgr2=0x7f02f40a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02f40a3db0 0x7f02f40a6270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f03081028b0 tx=0x7f02f8005e50 comp rx=0 tx=0).stop 2024-04-21T08:45:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 msgr2=0x7f0308100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f0308100e20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0304005940 tx=0x7f030402d920 comp rx=0 tx=0).stop 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 shutdown_connections 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f02f40a3db0 0x7f02f40a6270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f030810c510 0x7f03081018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0308103af0 0x7f0308101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 --2- 172.21.15.57:0/736714868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f03081031d0 0x7f0308100e20 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.417+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 >> 172.21.15.57:0/736714868 conn(0x7f0308077f50 msgr2=0x7f03080fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.418+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 shutdown_connections 2024-04-21T08:45:37.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.418+0000 7f031002e640 1 -- 172.21.15.57:0/736714868 wait complete. 2024-04-21T08:45:37.429 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 -- 172.21.15.57:0/1686284484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7410c580 msgr2=0x7fbb7410e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/1686284484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7410c580 0x7fbb7410e9b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009ca0 tx=0x7fbb6c02c180 comp rx=0 tx=0).stop 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 -- 172.21.15.57:0/1686284484 shutdown_connections 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/1686284484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb7410c580 0x7fbb7410e9b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/1686284484 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb74103ad0 0x7fbb7410bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/1686284484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb741031b0 0x7fbb74103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 -- 172.21.15.57:0/1686284484 >> 172.21.15.57:0/1686284484 conn(0x7fbb740771d0 msgr2=0x7fbb740775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 -- 172.21.15.57:0/1686284484 shutdown_connections 2024-04-21T08:45:37.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.534+0000 7fbb7a86d640 1 -- 172.21.15.57:0/1686284484 wait complete. 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 Processor -- start 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 -- start start 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb741031b0 0x7fbb74198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 0x7fbb74198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7410c580 0x7fbb7419cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbb741121a0 con 0x7fbb74103ad0 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74112020 con 0x7fbb7410c580 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb7a86d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74112320 con 0x7fbb741031b0 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 0x7fbb74198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 0x7fbb74198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55636/0 (socket says 172.21.15.57:55636) 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb737fe640 1 -- 172.21.15.57:0/2864946584 learned_addr learned my addr 172.21.15.57:0/2864946584 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:37.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb73fff640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb741031b0 0x7fbb74198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.535+0000 7fbb78e0a640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7410c580 0x7fbb7419cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb737fe640 1 -- 172.21.15.57:0/2864946584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb741031b0 msgr2=0x7fbb74198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb737fe640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb741031b0 0x7fbb74198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb737fe640 1 -- 172.21.15.57:0/2864946584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7410c580 msgr2=0x7fbb7419cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb737fe640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7410c580 0x7fbb7419cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb737fe640 1 -- 172.21.15.57:0/2864946584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7419d6f0 con 0x7fbb74103ad0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb78e0a640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7410c580 0x7fbb7419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb737fe640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 0x7fbb74198c40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400d770 tx=0x7fbb6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64017070 con 0x7fbb74103ad0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb640054d0 con 0x7fbb74103ad0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7419d9e0 con 0x7fbb74103ad0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64004c30 con 0x7fbb74103ad0 2024-04-21T08:45:37.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.536+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb741a5270 con 0x7fbb74103ad0 2024-04-21T08:45:37.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.537+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb7406ad70 con 0x7fbb74103ad0 2024-04-21T08:45:37.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.538+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fbb64004e10 con 0x7fbb74103ad0 2024-04-21T08:45:37.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.539+0000 7fbb717fa640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbb600a3db0 0x7fbb600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.539+0000 7fbb73fff640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbb600a3db0 0x7fbb600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.539+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbb640d0760 con 0x7fbb74103ad0 2024-04-21T08:45:37.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.539+0000 7fbb73fff640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbb600a3db0 0x7fbb600a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c005d00 tx=0x7fbb5c00b410 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:37.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.544+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbb6409c930 con 0x7fbb74103ad0 2024-04-21T08:45:37.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.731+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb741076d0 con 0x7fbb74103ad0 2024-04-21T08:45:37.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.732+0000 7fbb717fa640 1 -- 172.21.15.57:0/2864946584 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fbb6409c230 con 0x7fbb74103ad0 2024-04-21T08:45:37.732 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:45:37.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbb600a3db0 msgr2=0x7fbb600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbb600a3db0 0x7fbb600a6270 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c005d00 tx=0x7fbb5c00b410 comp rx=0 tx=0).stop 2024-04-21T08:45:37.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 msgr2=0x7fbb74198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 0x7fbb74198c40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400d770 tx=0x7fbb6400dc30 comp rx=0 tx=0).stop 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 shutdown_connections 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbb600a3db0 0x7fbb600a6270 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb7410c580 0x7fbb7419cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb74103ad0 0x7fbb74198c40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 --2- 172.21.15.57:0/2864946584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb741031b0 0x7fbb74198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 >> 172.21.15.57:0/2864946584 conn(0x7fbb740771d0 msgr2=0x7fbb740fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 shutdown_connections 2024-04-21T08:45:37.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.734+0000 7fbb7a86d640 1 -- 172.21.15.57:0/2864946584 wait complete. 2024-04-21T08:45:37.744 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:37 smithi057 ceph-mon[27968]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:37.744 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:37 smithi057 ceph-mon[27968]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:37.744 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:37 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3840702206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:37.744 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:37 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2706192531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:37.744 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:37 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/905013219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:37.745 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:45:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:37 smithi129 ceph-mon[31272]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:37 smithi129 ceph-mon[31272]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:37 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3840702206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:37 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2706192531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:37 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/905013219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 -- 172.21.15.57:0/3467320527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 msgr2=0x7efcb406ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/3467320527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb406ac20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efca4009900 tx=0x7efca402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 -- 172.21.15.57:0/3467320527 shutdown_connections 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/3467320527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb4107270 0x7efcb410ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/3467320527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb406b1f0 0x7efcb4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/3467320527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb406ac20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 -- 172.21.15.57:0/3467320527 >> 172.21.15.57:0/3467320527 conn(0x7efcb4077eb0 msgr2=0x7efcb40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:37.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 -- 172.21.15.57:0/3467320527 shutdown_connections 2024-04-21T08:45:37.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.849+0000 7efcb8ad2640 1 -- 172.21.15.57:0/3467320527 wait complete. 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 Processor -- start 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 -- start start 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb4107270 0x7efcb419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb41985c0 0x7efcb4198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efcb41121e0 con 0x7efcb406a840 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efcb4112060 con 0x7efcb4107270 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb8ad2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efcb4112360 con 0x7efcb41985c0 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.850+0000 7efcb259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55662/0 (socket says 172.21.15.57:55662) 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 -- 172.21.15.57:0/813494504 learned_addr learned my addr 172.21.15.57:0/813494504 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcabfff640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb4107270 0x7efcb419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 -- 172.21.15.57:0/813494504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb41985c0 msgr2=0x7efcb4198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb2d9d640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb41985c0 0x7efcb4198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb41985c0 0x7efcb4198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 -- 172.21.15.57:0/813494504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb4107270 msgr2=0x7efcb419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb4107270 0x7efcb419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 -- 172.21.15.57:0/813494504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb41990e0 con 0x7efcb406a840 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcabfff640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb4107270 0x7efcb419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb259c640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb419e410 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efca4005940 tx=0x7efca40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4043020 con 0x7efcb406a840 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efca402caf0 con 0x7efcb406a840 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb4199370 con 0x7efcb406a840 2024-04-21T08:45:37.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca4003af0 con 0x7efcb406a840 2024-04-21T08:45:37.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.851+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb41a5610 con 0x7efcb406a840 2024-04-21T08:45:37.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.852+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb40691c0 con 0x7efcb406a840 2024-04-21T08:45:37.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.853+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7efca4046020 con 0x7efcb406a840 2024-04-21T08:45:37.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.854+0000 7efcab7fe640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efca00a3e30 0x7efca00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:37.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.854+0000 7efcabfff640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efca00a3e30 0x7efca00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:37.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.854+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7efca40f2e10 con 0x7efcb406a840 2024-04-21T08:45:37.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.855+0000 7efcabfff640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efca00a3e30 0x7efca00a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efcb41009c0 tx=0x7efc9c005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:37.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:37.858+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efca404d020 con 0x7efcb406a840 2024-04-21T08:45:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:37 smithi136 ceph-mon[31791]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:37 smithi136 ceph-mon[31791]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-21T08:45:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:37 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3840702206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:37 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2706192531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:37 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/905013219' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.042 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.042+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efcb410a500 con 0x7efcb406a840 2024-04-21T08:45:38.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.042+0000 7efcab7fe640 1 -- 172.21.15.57:0/813494504 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7efca403c350 con 0x7efcb406a840 2024-04-21T08:45:38.043 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:45:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efca00a3e30 msgr2=0x7efca00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efca00a3e30 0x7efca00a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efcb41009c0 tx=0x7efc9c005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 msgr2=0x7efcb419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb419e410 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efca4005940 tx=0x7efca40045e0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 shutdown_connections 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efca00a3e30 0x7efca00a62f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcb41985c0 0x7efcb4198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb4107270 0x7efcb419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 --2- 172.21.15.57:0/813494504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcb406a840 0x7efcb419e410 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 >> 172.21.15.57:0/813494504 conn(0x7efcb4077eb0 msgr2=0x7efcb40ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 shutdown_connections 2024-04-21T08:45:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.045+0000 7efcb8ad2640 1 -- 172.21.15.57:0/813494504 wait complete. 2024-04-21T08:45:38.056 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:38.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 -- 172.21.15.57:0/2219961713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 msgr2=0x7f30ac10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/2219961713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac10bfd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f309c009900 tx=0x7f309c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 -- 172.21.15.57:0/2219961713 shutdown_connections 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/2219961713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac10c5a0 0x7f30ac10e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/2219961713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac10bfd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/2219961713 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30ac1031d0 0x7f30ac1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.157+0000 7f30b3cfb640 1 -- 172.21.15.57:0/2219961713 >> 172.21.15.57:0/2219961713 conn(0x7f30ac0771d0 msgr2=0x7f30ac0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 -- 172.21.15.57:0/2219961713 shutdown_connections 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 -- 172.21.15.57:0/2219961713 wait complete. 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 Processor -- start 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 -- start start 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac1031d0 0x7f30ac1986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30ac10c5a0 0x7f30ac19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1121e0 con 0x7f30ac103af0 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac112060 con 0x7f30ac1031d0 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b3cfb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac112360 con 0x7f30ac10c5a0 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.158+0000 7f30b1296640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55694/0 (socket says 172.21.15.57:55694) 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 -- 172.21.15.57:0/4033770913 learned_addr learned my addr 172.21.15.57:0/4033770913 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1a97640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac1031d0 0x7f30ac1986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b2298640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30ac10c5a0 0x7f30ac19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 -- 172.21.15.57:0/4033770913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30ac10c5a0 msgr2=0x7f30ac19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30ac10c5a0 0x7f30ac19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 -- 172.21.15.57:0/4033770913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac1031d0 msgr2=0x7f30ac1986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac1031d0 0x7f30ac1986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 -- 172.21.15.57:0/4033770913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac19d6d0 con 0x7f30ac103af0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1a97640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac1031d0 0x7f30ac1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b1296640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac198c20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f309c005940 tx=0x7f309c02d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c043020 con 0x7f30ac103af0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309c02dd10 con 0x7f30ac103af0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ac19d960 con 0x7f30ac103af0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c003b80 con 0x7f30ac103af0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.159+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ac1a5250 con 0x7f30ac103af0 2024-04-21T08:45:38.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.160+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ac069230 con 0x7f30ac103af0 2024-04-21T08:45:38.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.161+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f309c046020 con 0x7f30ac103af0 2024-04-21T08:45:38.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.162+0000 7f30a2ffd640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30980a3db0 0x7f30980a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.162+0000 7f30b1a97640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30980a3db0 0x7f30980a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.162+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f309c0f2f50 con 0x7f30ac103af0 2024-04-21T08:45:38.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.163+0000 7f30b1a97640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30980a3db0 0x7f30980a6270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f30a8005fd0 tx=0x7f30a8005ea0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:38.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.167+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f309c04d020 con 0x7f30ac103af0 2024-04-21T08:45:38.396 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.395+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30ac107690 con 0x7f30ac103af0 2024-04-21T08:45:38.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.402+0000 7f30a2ffd640 1 -- 172.21.15.57:0/4033770913 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f309c0bf1d0 con 0x7f30ac103af0 2024-04-21T08:45:38.402 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:38.402 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:38.403 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:38.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30980a3db0 msgr2=0x7f30980a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30980a3db0 0x7f30980a6270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f30a8005fd0 tx=0x7f30a8005ea0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 msgr2=0x7f30ac198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac198c20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f309c005940 tx=0x7f309c02d910 comp rx=0 tx=0).stop 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 shutdown_connections 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30980a3db0 0x7f30980a6270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30ac10c5a0 0x7f30ac19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30ac103af0 0x7f30ac198c20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 --2- 172.21.15.57:0/4033770913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30ac1031d0 0x7f30ac1986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 >> 172.21.15.57:0/4033770913 conn(0x7f30ac0771d0 msgr2=0x7f30ac0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 shutdown_connections 2024-04-21T08:45:38.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.404+0000 7f30b3cfb640 1 -- 172.21.15.57:0/4033770913 wait complete. 2024-04-21T08:45:38.415 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 -- 172.21.15.57:0/2681825798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880107010 msgr2=0x7f18801073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 --2- 172.21.15.57:0/2681825798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880107010 0x7f18801073f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1870009930 tx=0x7f187002bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 -- 172.21.15.57:0/2681825798 shutdown_connections 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 --2- 172.21.15.57:0/2681825798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1880101ab0 0x7f188010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 --2- 172.21.15.57:0/2681825798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 0x7f1880101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 --2- 172.21.15.57:0/2681825798 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880107010 0x7f18801073f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 -- 172.21.15.57:0/2681825798 >> 172.21.15.57:0/2681825798 conn(0x7f1880077e30 msgr2=0x7f1880078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 -- 172.21.15.57:0/2681825798 shutdown_connections 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.517+0000 7f1884831640 1 -- 172.21.15.57:0/2681825798 wait complete. 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 Processor -- start 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 -- start start 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 0x7f1880198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 0x7f1880198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1880107010 0x7f188019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f18801110f0 con 0x7f1880101ab0 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1880110f70 con 0x7f1880107010 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f1884831640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1880111270 con 0x7f1880101110 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f187dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 0x7f1880198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f187dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 0x7f1880198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55704/0 (socket says 172.21.15.57:55704) 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f187dd9b640 1 -- 172.21.15.57:0/1633604916 learned_addr learned my addr 172.21.15.57:0/1633604916 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.518+0000 7f187ed9d640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1880107010 0x7f188019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187e59c640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 0x7f1880198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187dd9b640 1 -- 172.21.15.57:0/1633604916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 msgr2=0x7f1880198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187dd9b640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 0x7f1880198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187dd9b640 1 -- 172.21.15.57:0/1633604916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1880107010 msgr2=0x7f188019cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187dd9b640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1880107010 0x7f188019cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187dd9b640 1 -- 172.21.15.57:0/1633604916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188019d720 con 0x7f1880101ab0 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187e59c640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 0x7f1880198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f187dd9b640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 0x7f1880198cc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f187400e9f0 tx=0x7f187400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187400cd80 con 0x7f1880101ab0 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18740054d0 con 0x7f1880101ab0 2024-04-21T08:45:38.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188019da10 con 0x7f1880101ab0 2024-04-21T08:45:38.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874010690 con 0x7f1880101ab0 2024-04-21T08:45:38.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.519+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18801a52a0 con 0x7f1880101ab0 2024-04-21T08:45:38.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.520+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1880069230 con 0x7f1880101ab0 2024-04-21T08:45:38.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.521+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f1874004e90 con 0x7f1880101ab0 2024-04-21T08:45:38.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.522+0000 7f186f7fe640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f18680a3e30 0x7f18680a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.522+0000 7f187e59c640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f18680a3e30 0x7f18680a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.522+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f18740d1120 con 0x7f1880101ab0 2024-04-21T08:45:38.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.522+0000 7f187e59c640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f18680a3e30 0x7f18680a62f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1870036040 tx=0x7f1870000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:38.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.526+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f187409d2f0 con 0x7f1880101ab0 2024-04-21T08:45:38.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.716+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18801a5530 con 0x7f1880101ab0 2024-04-21T08:45:38.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.717+0000 7f186f7fe640 1 -- 172.21.15.57:0/1633604916 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25074 (secure 0 0 0) 0x7f187409cbf0 con 0x7f1880101ab0 2024-04-21T08:45:38.718 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:38.719 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":94,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:45:24.897611+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f18680a3e30 msgr2=0x7f18680a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f18680a3e30 0x7f18680a62f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1870036040 tx=0x7f1870000bc0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 msgr2=0x7f1880198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 0x7f1880198cc0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f187400e9f0 tx=0x7f187400edf0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 shutdown_connections 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f18680a3e30 0x7f18680a62f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1880107010 0x7f188019cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1880101ab0 0x7f1880198cc0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 --2- 172.21.15.57:0/1633604916 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1880101110 0x7f1880198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 >> 172.21.15.57:0/1633604916 conn(0x7f1880077e30 msgr2=0x7f1880105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 shutdown_connections 2024-04-21T08:45:38.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.719+0000 7f1884831640 1 -- 172.21.15.57:0/1633604916 wait complete. 2024-04-21T08:45:38.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:38 smithi057 ceph-mon[27968]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 83 KiB/s wr, 47 op/s 2024-04-21T08:45:38.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:38 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/736714868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:38 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2864946584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.729 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:38 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/813494504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.731 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 53117 2024-04-21T08:45:38.732 DEBUG:tasks.quiescer:Yielding 2024-04-21T08:45:38.732 INFO:teuthology.run_tasks:Running task workunit... 2024-04-21T08:45:38.749 INFO:tasks.workunit:Pulling workunits from ref 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:45:38.749 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-21T08:45:38.749 INFO:tasks.workunit:timeout=3h 2024-04-21T08:45:38.750 INFO:tasks.workunit:cleanup=True 2024-04-21T08:45:38.750 DEBUG:teuthology.orchestra.run.smithi057:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:38.751 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:45:38.766 INFO:teuthology.orchestra.run.smithi057.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:38.766 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-04-21 08:43:24.228486876 +0000 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-04-21 08:43:24.228486876 +0000 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-04-21 08:45:22.577187611 +0000 2024-04-21T08:45:38.767 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-04-21 08:43:24.228486876 +0000 2024-04-21T08:45:38.767 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-21T08:45:38.767 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-21T08:45:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:38 smithi129 ceph-mon[31272]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 83 KiB/s wr, 47 op/s 2024-04-21T08:45:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/736714868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2864946584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:38 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/813494504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.837 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/leonid-s-usov/ceph-ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4f6bc5273643aaf5660909e85400647a8951c10b 2024-04-21T08:45:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 -- 172.21.15.57:0/3251077661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 msgr2=0x7ff12010dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 --2- 172.21.15.57:0/3251077661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010dd50 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff118009ca0 tx=0x7ff11802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 -- 172.21.15.57:0/3251077661 shutdown_connections 2024-04-21T08:45:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 --2- 172.21.15.57:0/3251077661 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010dd50 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 --2- 172.21.15.57:0/3251077661 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff120101a60 0x7ff120105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 --2- 172.21.15.57:0/3251077661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1201010b0 0x7ff120101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 -- 172.21.15.57:0/3251077661 >> 172.21.15.57:0/3251077661 conn(0x7ff1200771d0 msgr2=0x7ff1200775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:38.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 -- 172.21.15.57:0/3251077661 shutdown_connections 2024-04-21T08:45:38.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.853+0000 7ff125ff4640 1 -- 172.21.15.57:0/3251077661 wait complete. 2024-04-21T08:45:38.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 Processor -- start 2024-04-21T08:45:38.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 -- start start 2024-04-21T08:45:38.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1201010b0 0x7ff12010e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.860 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff12010f160 0x7ff1201a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff120112200 con 0x7ff120106180 2024-04-21T08:45:38.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff120112080 con 0x7ff12010f160 2024-04-21T08:45:38.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff125ff4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff120112380 con 0x7ff1201010b0 2024-04-21T08:45:38.861 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55720/0 (socket says 172.21.15.57:55720) 2024-04-21T08:45:38.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 -- 172.21.15.57:0/2168463207 learned_addr learned my addr 172.21.15.57:0/2168463207 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:38.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11ffff640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff12010f160 0x7ff1201a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11f7fe640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1201010b0 0x7ff12010e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 -- 172.21.15.57:0/2168463207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1201010b0 msgr2=0x7ff12010e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1201010b0 0x7ff12010e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 -- 172.21.15.57:0/2168463207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff12010f160 msgr2=0x7ff1201a67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff12010f160 0x7ff1201a67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.854+0000 7ff11effd640 1 -- 172.21.15.57:0/2168463207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1201a6ef0 con 0x7ff120106180 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff11f7fe640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1201010b0 0x7ff12010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff11effd640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010ec20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c970 tx=0x7ff11400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114014070 con 0x7ff120106180 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1201a71e0 con 0x7ff120106180 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff114004440 con 0x7ff120106180 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114003e50 con 0x7ff120106180 2024-04-21T08:45:38.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.855+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1201a7780 con 0x7ff120106180 2024-04-21T08:45:38.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.856+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff120069230 con 0x7ff120106180 2024-04-21T08:45:38.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.858+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7ff114017070 con 0x7ff120106180 2024-04-21T08:45:38.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.858+0000 7ff11cff9640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff1080a3f80 0x7ff1080a6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:38.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.859+0000 7ff11f7fe640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff1080a3f80 0x7ff1080a6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:38.866 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.859+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff1140d0880 con 0x7ff120106180 2024-04-21T08:45:38.866 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.859+0000 7ff11f7fe640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff1080a3f80 0x7ff1080a6440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff110009b40 tx=0x7ff110005e50 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:38.866 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:38.863+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff11401e030 con 0x7ff120106180 2024-04-21T08:45:38.871 INFO:tasks.workunit.client.0.smithi057.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-21T08:45:38.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:38 smithi136 ceph-mon[31791]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 83 KiB/s wr, 47 op/s 2024-04-21T08:45:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:38 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/736714868' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:38 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2864946584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:45:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:38 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/813494504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:45:39.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.093+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff120101a60 con 0x7ff120106180 2024-04-21T08:45:39.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.093+0000 7ff11cff9640 1 -- 172.21.15.57:0/2168463207 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7ff11409ca50 con 0x7ff120106180 2024-04-21T08:45:39.094 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:45:39.094 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:45:39.094 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:45:39.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff1080a3f80 msgr2=0x7ff1080a6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff1080a3f80 0x7ff1080a6440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff110009b40 tx=0x7ff110005e50 comp rx=0 tx=0).stop 2024-04-21T08:45:39.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 msgr2=0x7ff12010ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010ec20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff11400c970 tx=0x7ff11400ce30 comp rx=0 tx=0).stop 2024-04-21T08:45:39.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 shutdown_connections 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff1080a3f80 0x7ff1080a6440 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff12010f160 0x7ff1201a67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff120106180 0x7ff12010ec20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 --2- 172.21.15.57:0/2168463207 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1201010b0 0x7ff12010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.095+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 >> 172.21.15.57:0/2168463207 conn(0x7ff1200771d0 msgr2=0x7ff1200fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.096+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 shutdown_connections 2024-04-21T08:45:39.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.096+0000 7ff125ff4640 1 -- 172.21.15.57:0/2168463207 wait complete. 2024-04-21T08:45:39.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-21T08:42:56.988323+0000', 'modified': '2024-04-21T08:43:22.134494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6839/4293224778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 4293224778}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 4293224778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6839/123336774', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6838', 'nonce': 123336774}, {'type': 'v1', 'addr': '172.21.15.129:6839', 'nonce': 123336774}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24503, 'qdb_cluster': [24503]} 2024-04-21T08:45:39.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-21T08:45:39.107 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 -- 172.21.15.57:0/1703265471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44101a50 msgr2=0x7f8a44105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/1703265471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44101a50 0x7f8a44105a90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009900 tx=0x7f8a3802bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 -- 172.21.15.57:0/1703265471 shutdown_connections 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/1703265471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a44106170 0x7f8a4410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/1703265471 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44101a50 0x7f8a44105a90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/1703265471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a441010a0 0x7f8a44101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 -- 172.21.15.57:0/1703265471 >> 172.21.15.57:0/1703265471 conn(0x7f8a44077eb0 msgr2=0x7f8a440782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 -- 172.21.15.57:0/1703265471 shutdown_connections 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 -- 172.21.15.57:0/1703265471 wait complete. 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.250+0000 7f8a4adb2640 1 Processor -- start 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 -- start start 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a441010a0 0x7f8a44198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 0x7f8a44198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a4419cfd0 0x7f8a4419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8a441110b0 con 0x7f8a44106170 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44110f30 con 0x7f8a4419cfd0 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4adb2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8a44111230 con 0x7f8a441010a0 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 0x7f8a44198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 0x7f8a44198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55736/0 (socket says 172.21.15.57:55736) 2024-04-21T08:45:39.266 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 -- 172.21.15.57:0/521891577 learned_addr learned my addr 172.21.15.57:0/521891577 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a441010a0 0x7f8a44198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a4934f640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a4419cfd0 0x7f8a4419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 -- 172.21.15.57:0/521891577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a441010a0 msgr2=0x7f8a44198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a441010a0 0x7f8a44198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 -- 172.21.15.57:0/521891577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a4419cfd0 msgr2=0x7f8a4419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a4419cfd0 0x7f8a4419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 -- 172.21.15.57:0/521891577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4419dac0 con 0x7f8a44106170 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.251+0000 7f8a43fff640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 0x7f8a44198ca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8a38036040 tx=0x7f8a380045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.252+0000 7f8a41ffb640 1 -- 172.21.15.57:0/521891577 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38043020 con 0x7f8a44106170 2024-04-21T08:45:39.267 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.252+0000 7f8a41ffb640 1 -- 172.21.15.57:0/521891577 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a3802cb70 con 0x7f8a44106170 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.252+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4419dd50 con 0x7f8a44106170 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.252+0000 7f8a41ffb640 1 -- 172.21.15.57:0/521891577 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38003bf0 con 0x7f8a44106170 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.252+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a441a55a0 con 0x7f8a44106170 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a41ffb640 1 -- 172.21.15.57:0/521891577 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f8a38046020 con 0x7f8a44106170 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 Processor -- start 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 -- start start 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e66b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e67160 0x556f05e6f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e72130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556f05dcab10 con 0x556f05e6fce0 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556f05d68240 con 0x556f05e67160 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556f05dc6ad0 con 0x556f05e667b0 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e72130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.268 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e72130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55738/0 (socket says 172.21.15.57:55738) 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a43fff640 1 -- 172.21.15.57:0/2369902804 learned_addr learned my addr 172.21.15.57:0/2369902804 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4934f640 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e66b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e67160 0x556f05e6f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a41ffb640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8a2c0a3e30 0x7f8a2c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.255+0000 7f8a4934f640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8a2c0a3e30 0x7f8a2c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a43fff640 1 -- 172.21.15.57:0/2369902804 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 msgr2=0x556f05e66b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a43fff640 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e66b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a43fff640 1 -- 172.21.15.57:0/2369902804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e67160 msgr2=0x556f05e6f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a43fff640 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e67160 0x556f05e6f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a41ffb640 1 -- 172.21.15.57:0/521891577 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8a380f2c70 con 0x7f8a44106170 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a43fff640 1 -- 172.21.15.57:0/2369902804 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f05add2d0 con 0x556f05e6fce0 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e67160 0x556f05e6f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a43fff640 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e72130 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8a380be550 tx=0x7f8a380036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da86dd342e359b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:39.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a4934f640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8a2c0a3e30 0x7f8a2c0a62f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009fd0 tx=0x7f8a3c004ab0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a2affd640 1 -- 172.21.15.57:0/2369902804 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380be390 con 0x556f05e6fce0 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a2affd640 1 -- 172.21.15.57:0/2369902804 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a380bef50 con 0x556f05e6fce0 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.256+0000 7f8a2affd640 1 -- 172.21.15.57:0/2369902804 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3803f400 con 0x556f05e6fce0 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 -- 172.21.15.57:0/2369902804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 msgr2=0x556f05e72130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e72130 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8a380be550 tx=0x7f8a380036f0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 -- 172.21.15.57:0/2369902804 shutdown_connections 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e72130 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e67160 0x556f05e6f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/2369902804 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e66b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 -- 172.21.15.57:0/2369902804 >> 172.21.15.57:0/2369902804 conn(0x556f05e60a60 msgr2=0x556f05e60e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 -- 172.21.15.57:0/2369902804 shutdown_connections 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 -- 172.21.15.57:0/2369902804 wait complete. 2024-04-21T08:45:39.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 -- start start 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 Processor -- start 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.257+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e667b0 0x556f05e6cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e6d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e6d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556f05e72a10 con 0x556f05e6fce0 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556f05d53190 con 0x556f05e667b0 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556f05e72b90 con 0x556f05e67160 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 10 client.0 _ll_register_callbacks cb 0x556f05d4e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e6d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e6d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55748/0 (socket says 172.21.15.57:55748) 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 -- 172.21.15.57:0/3217135429 learned_addr learned my addr 172.21.15.57:0/3217135429 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a43fff640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e6d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e667b0 0x556f05e6cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 -- 172.21.15.57:0/3217135429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 msgr2=0x556f05e6d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e6d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 -- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e667b0 msgr2=0x556f05e6cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e667b0 0x556f05e6cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 -- 172.21.15.57:0/3217135429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f05ba76b0 con 0x556f05e6fce0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a43fff640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e6d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4934f640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e6d6f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c012040 tx=0x7f8a3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824079f4a50daf53 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a23fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a23fff640 1 -- 172.21.15.57:0/3217135429 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c025070 con 0x556f05e6fce0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a23fff640 1 -- 172.21.15.57:0/3217135429 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8a3c010dc0 con 0x556f05e6fce0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a380bef50 con 0x556f05e6fce0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.258+0000 7f8a23fff640 1 -- 172.21.15.57:0/3217135429 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8a3c00fe40 con 0x556f05e6fce0 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 10 client.15765 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:39.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556f05e7ecb0 con 0x556f05e6fce0 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a23fff640 1 -- 172.21.15.57:0/3217135429 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a380bef50 con 0x556f05e6fce0 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a23fff640 1 -- 172.21.15.57:0/3217135429 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8a3c021440 con 0x556f05e6fce0 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 10 client.15765 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 10 client.15765 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 20 client.15765 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05e81360 0x556f05e827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 4 client.15765 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f05dd8530 con 0x556f05e81360 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.259+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05e81360 0x556f05e827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.260+0000 7f8a23fff640 10 client.15765 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.260+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05e81360 0x556f05e827d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=659b9322fb452196 server_cookie=3d04c1db1295e97 in_seq=0 out_seq=0 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.260+0000 7f8a23fff640 1 -- 172.21.15.57:0/3217135429 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f05dd8530 con 0x556f05e81360 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.260+0000 7f8a23fff640 10 client.15765 handle_command_reply: tid=0 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.260+0000 7f8a4d99f740 1 client.15765 shutdown 2024-04-21T08:45:39.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.260+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 msgr2=0x556f05e6d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e6d6f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c012040 tx=0x7f8a3c004a40 comp rx=0 tx=0).stop 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 shutdown_connections 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05e81360 0x556f05e827d0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e6fce0 0x556f05e6d6f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a23fff640 0 client.15765 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e6d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/3217135429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e667b0 0x556f05e6cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 >> 172.21.15.57:0/3217135429 conn(0x556f05e60a60 msgr2=0x556f05e62bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 shutdown_connections 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 1 -- 172.21.15.57:0/3217135429 wait complete. 2024-04-21T08:45:39.274 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.261+0000 7f8a4d99f740 20 client.15765 trim_cache size 0 max 16384 2024-04-21T08:45:39.285 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 Processor -- start 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 -- start start 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e6f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 0x556f05e7c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556f05e80b10 con 0x556f05e67160 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556f05e811e0 con 0x556f05e6fce0 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556f05e818b0 con 0x556f05e667b0 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 0x556f05e7c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 0x556f05e7c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55756/0 (socket says 172.21.15.57:55756) 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 -- 172.21.15.57:0/231467561 learned_addr learned my addr 172.21.15.57:0/231467561 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:39.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4934f640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 -- 172.21.15.57:0/231467561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 msgr2=0x556f05e6f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e6f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e6f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 -- 172.21.15.57:0/231467561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 msgr2=0x556f05e69ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a43fff640 1 -- 172.21.15.57:0/231467561 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f05e66f00 con 0x556f05e67160 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a4934f640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.279+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e6f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a43fff640 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 0x556f05e7c300 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8a380be550 tx=0x7f8a380ba430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a292c43798d7d27e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a2a7fc640 1 -- 172.21.15.57:0/231467561 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380be1b0 con 0x556f05e67160 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a2a7fc640 1 -- 172.21.15.57:0/231467561 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a380f5040 con 0x556f05e67160 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a2a7fc640 1 -- 172.21.15.57:0/231467561 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380054d0 con 0x556f05e67160 2024-04-21T08:45:39.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 -- 172.21.15.57:0/231467561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 msgr2=0x556f05e7c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 0x556f05e7c300 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f8a380be550 tx=0x7f8a380ba430 comp rx=0 tx=0).stop 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 -- 172.21.15.57:0/231467561 shutdown_connections 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e67160 0x556f05e7c300 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/231467561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e667b0 0x556f05e6f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 -- 172.21.15.57:0/231467561 >> 172.21.15.57:0/231467561 conn(0x556f05e60a60 msgr2=0x556f05e6b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 -- 172.21.15.57:0/231467561 shutdown_connections 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.280+0000 7f8a4d99f740 1 -- 172.21.15.57:0/231467561 wait complete. 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 -- start start 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 Processor -- start 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 0x556f05e8ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e698c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556f05e82650 con 0x556f05e667b0 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556f05e81f80 con 0x556f05e6fce0 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556f05e82d20 con 0x556f05e67160 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4d99f740 10 client.0 _ll_register_callbacks cb 0x7f8a3c009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a48b4e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 0x556f05e8ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a48b4e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 0x556f05e8ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55766/0 (socket says 172.21.15.57:55766) 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a48b4e640 1 -- 172.21.15.57:0/395640188 learned_addr learned my addr 172.21.15.57:0/395640188 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.281+0000 7f8a4934f640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a48b4e640 1 -- 172.21.15.57:0/395640188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 msgr2=0x556f05e698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a43fff640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a48b4e640 1 -- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 msgr2=0x556f05e69e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a48b4e640 1 -- 172.21.15.57:0/395640188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f05e6a540 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a43fff640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a48b4e640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 0x556f05e8ad50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8a34000c00 tx=0x7f8a340118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff95d6d18e6dc43 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a23fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a23fff640 1 -- 172.21.15.57:0/395640188 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3401d900 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a23fff640 1 -- 172.21.15.57:0/395640188 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8a34005020 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a380f5040 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a23fff640 1 -- 172.21.15.57:0/395640188 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8a3402c050 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a4d99f740 10 client.15771 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556f05ee7310 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a23fff640 1 -- 172.21.15.57:0/395640188 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a380f5040 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a23fff640 1 -- 172.21.15.57:0/395640188 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8a340104f0 con 0x556f05e667b0 2024-04-21T08:45:39.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a4d99f740 10 client.15771 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.282+0000 7f8a4d99f740 10 client.15771 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 20 client.15771 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05eebe60 0x556f05eeab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 4 client.15771 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "status"}) -- 0x556f05dd23a0 con 0x556f05eebe60 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a43fff640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05eebe60 0x556f05eeab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a23fff640 10 client.15771 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a43fff640 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05eebe60 0x556f05eeab70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bba02cf4b3512b78 server_cookie=1b085f769945dac8 in_seq=0 out_seq=0 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a23fff640 1 -- 172.21.15.57:0/395640188 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556f05dd23a0 con 0x556f05eebe60 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a23fff640 10 client.15771 handle_command_reply: tid=0 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 client.15771 shutdown 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 msgr2=0x556f05e8ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 0x556f05e8ad50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8a34000c00 tx=0x7f8a340118d0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 shutdown_connections 2024-04-21T08:45:39.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556f05eebe60 0x556f05eeab70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556f05e6fce0 0x556f05e69e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556f05e67160 0x556f05e698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a23fff640 0 client.15771 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:39.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 --2- 172.21.15.57:0/395640188 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556f05e667b0 0x556f05e8ad50 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.283+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 >> 172.21.15.57:0/395640188 conn(0x556f05e60a60 msgr2=0x556f05e5fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 shutdown_connections 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4d99f740 1 -- 172.21.15.57:0/395640188 wait complete. 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4d99f740 20 client.15771 trim_cache size 0 max 16384 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stdout: "cluster_fsid": "df0fb3ae-ffb9-11ee-bc93-c7b262605968", 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stdout: "whoami": 0, 2024-04-21T08:45:39.293 INFO:teuthology.orchestra.run.smithi057.stdout: "id": 24503, 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "want_state": "up:active", 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "state": "up:active", 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "fs_name": "cephfs", 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "rank_uptime": 161.66316854300001, 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "mdsmap_epoch": 21, 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch": 94, 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_epoch_barrier": 89, 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout: "uptime": 167.96600819299999 2024-04-21T08:45:39.294 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:45:39.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8a2c0a3e30 msgr2=0x7f8a2c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8a2c0a3e30 0x7f8a2c0a62f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009fd0 tx=0x7f8a3c004ab0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 msgr2=0x7f8a44198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 0x7f8a44198ca0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8a38036040 tx=0x7f8a380045e0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 shutdown_connections 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8a2c0a3e30 0x7f8a2c0a62f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a4419cfd0 0x7f8a4419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a44106170 0x7f8a44198ca0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 --2- 172.21.15.57:0/521891577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a441010a0 0x7f8a44198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 >> 172.21.15.57:0/521891577 conn(0x7f8a44077eb0 msgr2=0x7f8a4406a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 shutdown_connections 2024-04-21T08:45:39.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:45:39.284+0000 7f8a4adb2640 1 -- 172.21.15.57:0/521891577 wait complete. 2024-04-21T08:45:39.296 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 120 sec 2024-04-21T08:45:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:39 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4033770913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:39 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1633604916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:39 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2168463207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:39 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4033770913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:39 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1633604916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:39 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2168463207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:39 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4033770913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:39 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1633604916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:45:39.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:39 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2168463207' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:45:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:40 smithi129 ceph-mon[31272]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:40.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:40 smithi057 ceph-mon[27968]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:40 smithi136 ceph-mon[31791]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:42 smithi129 ceph-mon[31272]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:42.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:42 smithi057 ceph-mon[27968]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:42 smithi136 ceph-mon[31791]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 76 KiB/s wr, 43 op/s 2024-04-21T08:45:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:44 smithi129 ceph-mon[31272]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-21T08:45:44.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:44 smithi057 ceph-mon[27968]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-21T08:45:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:44 smithi136 ceph-mon[31791]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-04-21T08:45:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2277413885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e81010f0 msgr2=0x7fe0e81014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2277413885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e81010f0 0x7fe0e81014d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009900 tx=0x7fe0e402bc30 comp rx=0 tx=0).stop 2024-04-21T08:45:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2277413885 shutdown_connections 2024-04-21T08:45:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2277413885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e8106170 0x7fe0e810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2277413885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0e8101aa0 0x7fe0e8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2277413885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e81010f0 0x7fe0e81014d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2277413885 >> 172.21.15.57:0/2277413885 conn(0x7fe0e8077fd0 msgr2=0x7fe0e80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2277413885 shutdown_connections 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2277413885 wait complete. 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 Processor -- start 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.002+0000 7fe0ef3b2640 1 -- start start 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ef3b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e81010f0 0x7fe0e8198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ef3b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 0x7fe0e8198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ef3b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0e819d010 0x7fe0e819d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ef3b2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81110e0 con 0x7fe0e8106170 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ef3b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8110f60 con 0x7fe0e819d010 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ef3b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8111260 con 0x7fe0e81010f0 2024-04-21T08:45:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 0x7fe0e8198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 0x7fe0e8198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54360/0 (socket says 172.21.15.57:54360) 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2271243347 learned_addr learned my addr 172.21.15.57:0/2271243347 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0e819d010 0x7fe0e819d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e81010f0 0x7fe0e8198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2271243347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e81010f0 msgr2=0x7fe0e8198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e81010f0 0x7fe0e8198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2271243347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0e819d010 msgr2=0x7fe0e819d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0e819d010 0x7fe0e819d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2271243347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e819db00 con 0x7fe0e8106170 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.003+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 0x7fe0e8198c90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800d770 tx=0x7fe0d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.004+0000 7fe0de7fc640 1 -- 172.21.15.57:0/2271243347 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8017070 con 0x7fe0e8106170 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.004+0000 7fe0de7fc640 1 -- 172.21.15.57:0/2271243347 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d8009c30 con 0x7fe0e8106170 2024-04-21T08:45:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.004+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e819ddf0 con 0x7fe0e8106170 2024-04-21T08:45:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.004+0000 7fe0de7fc640 1 -- 172.21.15.57:0/2271243347 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d800f620 con 0x7fe0e8106170 2024-04-21T08:45:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.004+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e81a55e0 con 0x7fe0e8106170 2024-04-21T08:45:45.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.006+0000 7fe0de7fc640 1 -- 172.21.15.57:0/2271243347 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fe0d801a070 con 0x7fe0e8106170 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 Processor -- start 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 -- start start 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0de7fc640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0d40a3e60 0x7fe0d40a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ac620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b58e8118d0 con 0x55b58e8b60e0 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b58e7aee80 con 0x55b58e8acb60 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b58e80d890 con 0x55b58e8ad8d0 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0d40a3e60 0x7fe0d40a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0ed94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0de7fc640 1 -- 172.21.15.57:0/2271243347 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe0d80d07c0 con 0x7fe0e8106170 2024-04-21T08:45:45.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0ed94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54364/0 (socket says 172.21.15.57:54364) 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0ed94f640 1 -- 172.21.15.57:0/2970605506 learned_addr learned my addr 172.21.15.57:0/2970605506 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.007+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2970605506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 msgr2=0x55b58e8ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2970605506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 msgr2=0x55b58e8b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2970605506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b58e7d0320 con 0x55b58e8acb60 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0d40a3e60 0x7fe0d40a6320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0012c00 tx=0x7fe0e0009e00 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8acfe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80cdaf0 tx=0x7fe0d809bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2635daa19a7281ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0d37fe640 1 -- 172.21.15.57:0/2970605506 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8021070 con 0x55b58e8acb60 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0d37fe640 1 -- 172.21.15.57:0/2970605506 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d80af120 con 0x55b58e8acb60 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.008+0000 7fe0d37fe640 1 -- 172.21.15.57:0/2970605506 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d809a6d0 con 0x55b58e8acb60 2024-04-21T08:45:45.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2970605506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 msgr2=0x55b58e8acfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8acfe0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80cdaf0 tx=0x7fe0d809bd60 comp rx=0 tx=0).stop 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2970605506 shutdown_connections 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8acfe0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2970605506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2970605506 >> 172.21.15.57:0/2970605506 conn(0x55b58e873c90 msgr2=0x55b58e8a7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2970605506 shutdown_connections 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2970605506 wait complete. 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 -- start start 2024-04-21T08:45:45.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.009+0000 7fe0f1fa7740 1 Processor -- start 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8ad8d0 0x55b58e8b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b58e8b8db0 con 0x55b58e8b60e0 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b58e799eb0 con 0x55b58e8ad8d0 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b58e8b8f30 con 0x55b58e8acb60 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0f1fa7740 10 client.0 _ll_register_callbacks cb 0x55b58e7c6b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed14e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ec94d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8ad8d0 0x55b58e8b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed14e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:46272/0 (socket says 172.21.15.57:46272) 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed14e640 1 -- 172.21.15.57:0/2547414805 learned_addr learned my addr 172.21.15.57:0/2547414805 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:45.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed94f640 1 -- 172.21.15.57:0/2547414805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 msgr2=0x55b58e8b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed94f640 1 -- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8ad8d0 msgr2=0x55b58e8b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8ad8d0 0x55b58e8b2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ed94f640 1 -- 172.21.15.57:0/2547414805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b58e524210 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.010+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8ad8d0 0x55b58e8b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b3340 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005fd0 tx=0x7fe0e0009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3464917b9b84f93d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/2547414805 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0004b20 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/2547414805 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe0e0004cc0 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0d80af120 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/2547414805 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe0e0004510 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 10 client.15783 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b58e5f0620 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/2547414805 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d80af120 con 0x55b58e8b60e0 2024-04-21T08:45:45.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/2547414805 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe0e001f070 con 0x55b58e8b60e0 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 10 client.15783 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 20 client.15783 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e8c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 4 client.15783 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.011+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b58e825880 con 0x55b58e8c60f0 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.012+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e8c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.012+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e8c84e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3519373940c859b server_cookie=ad16c859b11e751b in_seq=0 out_seq=0 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.012+0000 7fe0d0ff9640 10 client.15783 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/2547414805 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b58e825880 con 0x55b58e8c60f0 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0d0ff9640 10 client.15783 handle_command_reply: tid=0 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 client.15783 shutdown 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 msgr2=0x55b58e8b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.018 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b3340 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005fd0 tx=0x7fe0e0009600 comp rx=0 tx=0).stop 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 shutdown_connections 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e8c84e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8b3340 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8ad8d0 0x55b58e8b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0d0ff9640 0 client.15783 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2547414805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.013+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 >> 172.21.15.57:0/2547414805 conn(0x55b58e873c90 msgr2=0x55b58e8a95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.014+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 shutdown_connections 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.014+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2547414805 wait complete. 2024-04-21T08:45:45.019 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.014+0000 7fe0f1fa7740 20 client.15783 trim_cache size 0 max 16384 2024-04-21T08:45:45.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 Processor -- start 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 -- start start 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8c1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8b60e0 0x55b58e8b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81ad160 con 0x55b58e8ad8d0 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81af9c0 con 0x55b58e8b60e0 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81ad7b0 con 0x55b58e8acb60 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0ec94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0ec94d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54374/0 (socket says 172.21.15.57:54374) 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2067577524 learned_addr learned my addr 172.21.15.57:0/2067577524 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.032+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8c1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8b60e0 0x55b58e8b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2067577524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 msgr2=0x55b58e8c1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8c1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2067577524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8b60e0 msgr2=0x55b58e8b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8b60e0 0x55b58e8b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ec94d640 1 -- 172.21.15.57:0/2067577524 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b58e8abe30 con 0x55b58e8ad8d0 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8b60e0 0x55b58e8b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ae080 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8000c00 tx=0x7fe0d8010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce8a8fa69ba6dcf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0d2ffd640 1 -- 172.21.15.57:0/2067577524 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8021040 con 0x55b58e8ad8d0 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0d2ffd640 1 -- 172.21.15.57:0/2067577524 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d8009de0 con 0x55b58e8ad8d0 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.033+0000 7fe0d2ffd640 1 -- 172.21.15.57:0/2067577524 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d80989b0 con 0x55b58e8ad8d0 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2067577524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 msgr2=0x55b58e8ae080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ae080 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8000c00 tx=0x7fe0d8010c00 comp rx=0 tx=0).stop 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2067577524 shutdown_connections 2024-04-21T08:45:45.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8b60e0 0x55b58e8b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8ad8d0 0x55b58e8ae080 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/2067577524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8acb60 0x55b58e8c1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2067577524 >> 172.21.15.57:0/2067577524 conn(0x55b58e873c90 msgr2=0x55b58e8b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2067577524 shutdown_connections 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/2067577524 wait complete. 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 -- start start 2024-04-21T08:45:45.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.034+0000 7fe0f1fa7740 1 Processor -- start 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8be490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8af690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81ade80 con 0x55b58e8b60e0 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81af2f0 con 0x55b58e8acb60 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e81ae550 con 0x55b58e8ad8d0 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0f1fa7740 10 client.0 _ll_register_callbacks cb 0x7fe0e0009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54386/0 (socket says 172.21.15.57:54386) 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 -- 172.21.15.57:0/578318710 learned_addr learned my addr 172.21.15.57:0/578318710 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ec94d640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 -- 172.21.15.57:0/578318710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 msgr2=0x55b58e8af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 -- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 msgr2=0x55b58e8be490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8be490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 -- 172.21.15.57:0/578318710 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b58e8b0290 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0ed94f640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8afbd0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005d70 tx=0x7fe0e0004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d97c59f180a91d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0d0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.035+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/578318710 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0020430 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/578318710 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe0e00120b0 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0d8009de0 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/578318710 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe0e0005040 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 10 client.15789 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b58e92c4d0 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/578318710 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d8009de0 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/578318710 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe0e000f040 con 0x55b58e8b60e0 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 10 client.15789 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 20 client.15789 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e92ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 4 client.15789 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.036+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b58e7b13d0 con 0x55b58e8c60f0 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.037+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e92ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.037+0000 7fe0ed14e640 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e92ea60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f8ac51160f6b6a1 server_cookie=148e9fd66d054e3e in_seq=0 out_seq=0 2024-04-21T08:45:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.037+0000 7fe0d0ff9640 10 client.15789 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0d0ff9640 1 -- 172.21.15.57:0/578318710 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b58e7b13d0 con 0x55b58e8c60f0 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0d0ff9640 10 client.15789 handle_command_reply: tid=0 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 client.15789 shutdown 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 msgr2=0x55b58e8afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8afbd0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005d70 tx=0x7fe0e0004410 comp rx=0 tx=0).stop 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 shutdown_connections 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b58e8c60f0 0x55b58e92ea60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b58e8b60e0 0x55b58e8afbd0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b58e8ad8d0 0x55b58e8af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0d0ff9640 0 client.15789 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:45.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 --2- 172.21.15.57:0/578318710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b58e8acb60 0x55b58e8be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 >> 172.21.15.57:0/578318710 conn(0x55b58e873c90 msgr2=0x55b58e8b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 shutdown_connections 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 1 -- 172.21.15.57:0/578318710 wait complete. 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.050+0000 7fe0f1fa7740 20 client.15789 trim_cache size 0 max 16384 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:45:45.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:45:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0d40a3e60 msgr2=0x7fe0d40a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0d40a3e60 0x7fe0d40a6320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0012c00 tx=0x7fe0e0009e00 comp rx=0 tx=0).stop 2024-04-21T08:45:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 msgr2=0x7fe0e8198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 0x7fe0e8198c90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800d770 tx=0x7fe0d800dc30 comp rx=0 tx=0).stop 2024-04-21T08:45:45.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 shutdown_connections 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0d40a3e60 0x7fe0d40a6320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0e819d010 0x7fe0e819d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0e8106170 0x7fe0e8198c90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 --2- 172.21.15.57:0/2271243347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e81010f0 0x7fe0e8198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 >> 172.21.15.57:0/2271243347 conn(0x7fe0e8077fd0 msgr2=0x7fe0e8069950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 shutdown_connections 2024-04-21T08:45:45.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:45.051+0000 7fe0ef3b2640 1 -- 172.21.15.57:0/2271243347 wait complete. 2024-04-21T08:45:46.798 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:46 smithi057 ceph-mon[27968]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:45:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:46 smithi129 ceph-mon[31272]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:45:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:46 smithi136 ceph-mon[31791]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-21T08:45:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:48 smithi129 ceph-mon[31272]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-21T08:45:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:48 smithi057 ceph-mon[27968]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-21T08:45:48.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:48 smithi136 ceph-mon[31791]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-21T08:45:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:50 smithi129 ceph-mon[31272]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T08:45:50.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:50 smithi057 ceph-mon[27968]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T08:45:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:50 smithi136 ceph-mon[31791]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T08:45:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:52 smithi129 ceph-mon[31272]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T08:45:52.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:52 smithi057 ceph-mon[27968]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T08:45:52.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:52 smithi136 ceph-mon[31791]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-21T08:45:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:54 smithi129 ceph-mon[31272]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-21T08:45:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:54 smithi057 ceph-mon[27968]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-21T08:45:54.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:54 smithi136 ceph-mon[31791]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2024-04-21T08:45:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:56 smithi129 ceph-mon[31272]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-21T08:45:56.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:56 smithi057 ceph-mon[27968]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-21T08:45:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:56 smithi136 ceph-mon[31791]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-21T08:45:58.181 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.180+0000 7fcaa2843640 1 -- 172.21.15.57:0/2102225833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c10c520 msgr2=0x7fca9c10e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.180+0000 7fcaa2843640 1 --2- 172.21.15.57:0/2102225833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c10c520 0x7fca9c10e9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fca94009ca0 tx=0x7fca9402c230 comp rx=0 tx=0).stop 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 -- 172.21.15.57:0/2102225833 shutdown_connections 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 --2- 172.21.15.57:0/2102225833 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c10c520 0x7fca9c10e9c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 --2- 172.21.15.57:0/2102225833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca9c103b00 0x7fca9c10bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 --2- 172.21.15.57:0/2102225833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca9c1031e0 0x7fca9c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 -- 172.21.15.57:0/2102225833 >> 172.21.15.57:0/2102225833 conn(0x7fca9c077fd0 msgr2=0x7fca9c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 -- 172.21.15.57:0/2102225833 shutdown_connections 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 -- 172.21.15.57:0/2102225833 wait complete. 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 Processor -- start 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.181+0000 7fcaa2843640 1 -- start start 2024-04-21T08:45:58.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa2843640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca9c1031e0 0x7fca9c1a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa2843640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 0x7fca9c1a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa2843640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca9c10c520 0x7fca9c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa2843640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c102210 con 0x7fca9c103b00 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa2843640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c102090 con 0x7fca9c1031e0 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa2843640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c102390 con 0x7fca9c10c520 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 0x7fca9c1a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 0x7fca9c1a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59208/0 (socket says 172.21.15.57:59208) 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 -- 172.21.15.57:0/3444287077 learned_addr learned my addr 172.21.15.57:0/3444287077 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca9c1031e0 0x7fca9c1a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca9c10c520 0x7fca9c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 -- 172.21.15.57:0/3444287077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca9c10c520 msgr2=0x7fca9c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca9c10c520 0x7fca9c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 -- 172.21.15.57:0/3444287077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca9c1031e0 msgr2=0x7fca9c1a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca9c1031e0 0x7fca9c1a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 -- 172.21.15.57:0/3444287077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c19af30 con 0x7fca9c103b00 2024-04-21T08:45:58.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.182+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 0x7fca9c1a0af0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fca9000d770 tx=0x7fca9000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.183+0000 7fca997fa640 1 -- 172.21.15.57:0/3444287077 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90017070 con 0x7fca9c103b00 2024-04-21T08:45:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.183+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca9c19b220 con 0x7fca9c103b00 2024-04-21T08:45:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.183+0000 7fca997fa640 1 -- 172.21.15.57:0/3444287077 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca900054d0 con 0x7fca9c103b00 2024-04-21T08:45:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.183+0000 7fca997fa640 1 -- 172.21.15.57:0/3444287077 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90004c30 con 0x7fca9c103b00 2024-04-21T08:45:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.183+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca9c1a7530 con 0x7fca9c103b00 2024-04-21T08:45:58.185 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.185+0000 7fca997fa640 1 -- 172.21.15.57:0/3444287077 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fca90004dd0 con 0x7fca9c103b00 2024-04-21T08:45:58.188 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.186+0000 7fca997fa640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fca840a3ee0 0x7fca840a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.186+0000 7fca997fa640 1 -- 172.21.15.57:0/3444287077 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fca900d0690 con 0x7fca9c103b00 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.186+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fca840a3ee0 0x7fca840a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.186+0000 7fcaa5436740 1 Processor -- start 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.186+0000 7fcaa5436740 1 -- start start 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acc17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa5436740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b54ac1d8d0 con 0x55b54acc20e0 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa5436740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b54abbae80 con 0x55b54acca8f0 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa5436740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b54ac19890 con 0x55b54acc1370 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59210/0 (socket says 172.21.15.57:59210) 2024-04-21T08:45:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 -- 172.21.15.57:0/3802203643 learned_addr learned my addr 172.21.15.57:0/3802203643 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acc17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fca840a3ee0 0x7fca840a63a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009b60 tx=0x7fca8c005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 -- 172.21.15.57:0/3802203643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 msgr2=0x55b54acc17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acc17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 -- 172.21.15.57:0/3802203643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 msgr2=0x55b54acccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 -- 172.21.15.57:0/3802203643 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54abdc320 con 0x55b54acc20e0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acc17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.187+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc0e30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0099e0 tx=0x7fca8c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4572b5a8461ec865 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.188+0000 7fca827fc640 1 -- 172.21.15.57:0/3802203643 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c027670 con 0x55b54acc20e0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.188+0000 7fca827fc640 1 -- 172.21.15.57:0/3802203643 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca8c022070 con 0x55b54acc20e0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.188+0000 7fca827fc640 1 -- 172.21.15.57:0/3802203643 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c01d720 con 0x55b54acc20e0 2024-04-21T08:45:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.188+0000 7fcaa5436740 1 -- 172.21.15.57:0/3802203643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 msgr2=0x55b54acc0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.188+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc0e30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0099e0 tx=0x7fca8c019000 comp rx=0 tx=0).stop 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 -- 172.21.15.57:0/3802203643 shutdown_connections 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acc17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3802203643 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc0e30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 -- 172.21.15.57:0/3802203643 >> 172.21.15.57:0/3802203643 conn(0x55b54ac7fc90 msgr2=0x55b54acbc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 -- 172.21.15.57:0/3802203643 shutdown_connections 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 -- 172.21.15.57:0/3802203643 wait complete. 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 -- start start 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 Processor -- start 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acca280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.189+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acc7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa5436740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b54accd620 con 0x55b54acc20e0 2024-04-21T08:45:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc7610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59218/0 (socket says 172.21.15.57:59218) 2024-04-21T08:45:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 -- 172.21.15.57:0/304233113 learned_addr learned my addr 172.21.15.57:0/304233113 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b54aba5eb0 con 0x55b54acca8f0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b54accd7a0 con 0x55b54acc1370 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa5436740 10 client.0 _ll_register_callbacks cb 0x55b54abd2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acc7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 -- 172.21.15.57:0/304233113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 msgr2=0x55b54acca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 -- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 msgr2=0x55b54acc7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acc7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 -- 172.21.15.57:0/304233113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54a930210 con 0x55b54acc20e0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc7610 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fca900cda50 tx=0x7fca900057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8561f7452fdc1d4e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca7b7fe640 1 -- 172.21.15.57:0/304233113 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca900b5060 con 0x55b54acc20e0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fca7b7fe640 1 -- 172.21.15.57:0/304233113 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fca900987d0 con 0x55b54acc20e0 2024-04-21T08:45:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.190+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca8c022070 con 0x55b54acc20e0 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fca7b7fe640 1 -- 172.21.15.57:0/304233113 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fca90020040 con 0x55b54acc20e0 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 10 client.15801 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b54a9fc620 con 0x55b54acc20e0 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fca7b7fe640 1 -- 172.21.15.57:0/304233113 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca900a4dd0 con 0x55b54acc20e0 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fca7b7fe640 1 -- 172.21.15.57:0/304233113 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fca900d02b0 con 0x55b54acc20e0 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 10 client.15801 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 20 client.15801 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54acda8f0 0x55b54acdcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 4 client.15801 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fca8c022070 con 0x55b54acda8f0 2024-04-21T08:45:58.196 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.191+0000 7fca9bfff640 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54acda8f0 0x55b54acdcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.192+0000 7fca7b7fe640 10 client.15801 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.192+0000 7fca9bfff640 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54acda8f0 0x55b54acdcce0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8149e51d2f9c618b server_cookie=63ca9d2706fe10e1 in_seq=0 out_seq=0 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fca7b7fe640 1 -- 172.21.15.57:0/304233113 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fca8c022070 con 0x55b54acda8f0 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fca7b7fe640 10 client.15801 handle_command_reply: tid=0 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 client.15801 shutdown 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 msgr2=0x55b54acc7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc7610 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fca900cda50 tx=0x7fca900057c0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 shutdown_connections 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54acda8f0 0x55b54acdcce0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fca7b7fe640 0 client.15801 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acca8f0 0x55b54acc7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc20e0 0x55b54acc7610 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 --2- 172.21.15.57:0/304233113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 >> 172.21.15.57:0/304233113 conn(0x55b54ac7fc90 msgr2=0x55b54acbddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 shutdown_connections 2024-04-21T08:45:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 1 -- 172.21.15.57:0/304233113 wait complete. 2024-04-21T08:45:58.198 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.193+0000 7fcaa5436740 20 client.15801 trim_cache size 0 max 16384 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 Processor -- start 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 -- start start 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acd6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54acc2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 0x55b54acc9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c1aaff0 con 0x55b54acca8f0 2024-04-21T08:45:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c1ad850 con 0x55b54acc20e0 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa5436740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c1ab640 con 0x55b54acc1370 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa0de0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 0x55b54acc9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa0de0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 0x55b54acc9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59224/0 (socket says 172.21.15.57:59224) 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.208+0000 7fcaa0de0640 1 -- 172.21.15.57:0/1845874530 learned_addr learned my addr 172.21.15.57:0/1845874530 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54acc2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fca9bfff640 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acd6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fcaa0de0640 1 -- 172.21.15.57:0/1845874530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 msgr2=0x55b54acd6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acd6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fcaa0de0640 1 -- 172.21.15.57:0/1845874530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 msgr2=0x55b54acc2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54acc2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fcaa0de0640 1 -- 172.21.15.57:0/1845874530 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54acbd290 con 0x55b54acca8f0 2024-04-21T08:45:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 0x55b54acc9d40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fca94004af0 tx=0x7fca94005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4faddb010314c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fca81ffb640 1 -- 172.21.15.57:0/1845874530 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94044070 con 0x55b54acca8f0 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fca81ffb640 1 -- 172.21.15.57:0/1845874530 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca94003e20 con 0x55b54acca8f0 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.209+0000 7fca81ffb640 1 -- 172.21.15.57:0/1845874530 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94039040 con 0x55b54acca8f0 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 -- 172.21.15.57:0/1845874530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 msgr2=0x55b54acc9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 0x55b54acc9d40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fca94004af0 tx=0x7fca94005d80 comp rx=0 tx=0).stop 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 -- 172.21.15.57:0/1845874530 shutdown_connections 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acca8f0 0x55b54acc9d40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54acc2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 --2- 172.21.15.57:0/1845874530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acc1370 0x55b54acd6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 -- 172.21.15.57:0/1845874530 >> 172.21.15.57:0/1845874530 conn(0x55b54ac7fc90 msgr2=0x55b54acc26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 -- 172.21.15.57:0/1845874530 shutdown_connections 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 -- 172.21.15.57:0/1845874530 wait complete. 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 -- start start 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.210+0000 7fcaa5436740 1 Processor -- start 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 0x55b54acce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54accec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acca8f0 0x55b54accf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c1abd10 con 0x55b54acc1370 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c1ad180 con 0x55b54acc20e0 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c1ac3e0 con 0x55b54acca8f0 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa5436740 10 client.0 _ll_register_callbacks cb 0x7fca900cd850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:45:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 0x55b54acce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 0x55b54acce6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59232/0 (socket says 172.21.15.57:59232) 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 -- 172.21.15.57:0/3089420649 learned_addr learned my addr 172.21.15.57:0/3089420649 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54accec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acca8f0 0x55b54accf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 -- 172.21.15.57:0/3089420649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acca8f0 msgr2=0x55b54accf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acca8f0 0x55b54accf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 -- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 msgr2=0x55b54accec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54accec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 -- 172.21.15.57:0/3089420649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b54ad400b0 con 0x55b54acc1370 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fcaa0de0640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acca8f0 0x55b54accf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca9bfff640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 0x55b54acce6f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fca8c01eee0 tx=0x7fca8c011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5936f8fa6b4d8a05 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.211+0000 7fca7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fca7b7fe640 1 -- 172.21.15.57:0/3089420649 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c034020 con 0x55b54acc1370 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fca7b7fe640 1 -- 172.21.15.57:0/3089420649 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fca8c0303a0 con 0x55b54acc1370 2024-04-21T08:45:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca94003e20 con 0x55b54acc1370 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fca7b7fe640 1 -- 172.21.15.57:0/3089420649 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fca8c03f050 con 0x55b54acc1370 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 10 client.15807 fetch_fsmap learned FSMap version 21 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b54ad413c0 con 0x55b54acc1370 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fca7b7fe640 1 -- 172.21.15.57:0/3089420649 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca94003e20 con 0x55b54acc1370 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fca7b7fe640 1 -- 172.21.15.57:0/3089420649 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fca8c01e8c0 con 0x55b54acc1370 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 10 client.15807 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 20 client.15807 populate_metadata read hostname 'smithi057' 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54ad419d0 0x55b54ad43df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 4 client.15807 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.212+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b54ac31880 con 0x55b54ad419d0 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.213+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54ad419d0 0x55b54ad43df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.213+0000 7fca7b7fe640 10 client.15807 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.213+0000 7fca9b7fe640 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54ad419d0 0x55b54ad43df0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b80480497ef57bb1 server_cookie=ac68c275f1cd5626 in_seq=0 out_seq=0 2024-04-21T08:45:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.219+0000 7fca7b7fe640 1 -- 172.21.15.57:0/3089420649 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b54ac31880 con 0x55b54ad419d0 2024-04-21T08:45:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.219+0000 7fca7b7fe640 10 client.15807 handle_command_reply: tid=0 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.219+0000 7fcaa5436740 1 client.15807 shutdown 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 msgr2=0x55b54acce6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 0x55b54acce6f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fca8c01eee0 tx=0x7fca8c011dc0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 shutdown_connections 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b54ad419d0 0x55b54ad43df0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b54acca8f0 0x55b54accf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b54acc20e0 0x55b54accec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 --2- 172.21.15.57:0/3089420649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b54acc1370 0x55b54acce6f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 >> 172.21.15.57:0/3089420649 conn(0x55b54ac7fc90 msgr2=0x55b54acb2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fca7b7fe640 0 client.15807 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 shutdown_connections 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 1 -- 172.21.15.57:0/3089420649 wait complete. 2024-04-21T08:45:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.220+0000 7fcaa5436740 20 client.15807 trim_cache size 0 max 16384 2024-04-21T08:45:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:45:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:45:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:45:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fca840a3ee0 msgr2=0x7fca840a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fca840a3ee0 0x7fca840a63a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca8c009b60 tx=0x7fca8c005f90 comp rx=0 tx=0).stop 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 msgr2=0x7fca9c1a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 0x7fca9c1a0af0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fca9000d770 tx=0x7fca9000dc30 comp rx=0 tx=0).stop 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 shutdown_connections 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fca840a3ee0 0x7fca840a63a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fca9c10c520 0x7fca9c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fca9c103b00 0x7fca9c1a0af0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 --2- 172.21.15.57:0/3444287077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca9c1031e0 0x7fca9c1a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:45:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 >> 172.21.15.57:0/3444287077 conn(0x7fca9c077fd0 msgr2=0x7fca9c103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:45:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 shutdown_connections 2024-04-21T08:45:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:45:58.221+0000 7fcaa2843640 1 -- 172.21.15.57:0/3444287077 wait complete. 2024-04-21T08:45:58.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:45:58 smithi129 ceph-mon[31272]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-21T08:45:58.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:45:58 smithi057 ceph-mon[27968]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-21T08:45:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:45:58 smithi136 ceph-mon[31791]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-21T08:46:00.452 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:46:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:46:00.113+0000 7fa23d06b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:46:00.452 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:46:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:46:00.113+0000 7fa23d06b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:46:00.452 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:46:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:46:00.113+0000 7fa23d06b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:46:00.452 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:46:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:46:00.113+0000 7fa23d06b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:46:00.452 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:46:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x[28198]: 2024-04-21T08:46:00.113+0000 7fa23d06b700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-21T08:46:00.567 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:00.567 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:00.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 -- 172.21.15.57:0/3745981917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef88101080 msgr2=0x7fef88101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/3745981917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef88101080 0x7fef88101460 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fef78009900 tx=0x7fef7802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 -- 172.21.15.57:0/3745981917 shutdown_connections 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/3745981917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88106150 0x7fef8810dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/3745981917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88101a30 0x7fef88105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/3745981917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef88101080 0x7fef88101460 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 -- 172.21.15.57:0/3745981917 >> 172.21.15.57:0/3745981917 conn(0x7fef88078000 msgr2=0x7fef88078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 -- 172.21.15.57:0/3745981917 shutdown_connections 2024-04-21T08:46:00.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.729+0000 7fef8c9b1640 1 -- 172.21.15.57:0/3745981917 wait complete. 2024-04-21T08:46:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.730+0000 7fef8c9b1640 1 Processor -- start 2024-04-21T08:46:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.730+0000 7fef8c9b1640 1 -- start start 2024-04-21T08:46:00.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8c9b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88101080 0x7fef881986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8c9b1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 0x7fef88198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8c9b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef8819cf90 0x7fef8819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8c9b1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fef881121d0 con 0x7fef8819cf90 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8c9b1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef88112050 con 0x7fef88101080 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8c9b1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fef88112350 con 0x7fef88106150 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 0x7fef88198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 0x7fef88198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:49272/0 (socket says 172.21.15.57:49272) 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88101080 0x7fef881986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 -- 172.21.15.57:0/4289541441 learned_addr learned my addr 172.21.15.57:0/4289541441 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef86d9d640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef8819cf90 0x7fef8819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 -- 172.21.15.57:0/4289541441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88101080 msgr2=0x7fef881986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88101080 0x7fef881986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 -- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef8819cf90 msgr2=0x7fef8819d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef8819cf90 0x7fef8819d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef7ffff640 1 -- 172.21.15.57:0/4289541441 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8819da80 con 0x7fef88106150 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.731+0000 7fef8659c640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88101080 0x7fef881986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.732+0000 7fef7ffff640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 0x7fef88198c10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fef7000d740 tx=0x7fef7000dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.732+0000 7fef7f7fe640 1 -- 172.21.15.57:0/4289541441 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70018070 con 0x7fef88106150 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.732+0000 7fef7f7fe640 1 -- 172.21.15.57:0/4289541441 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef7000de50 con 0x7fef88106150 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.732+0000 7fef7f7fe640 1 -- 172.21.15.57:0/4289541441 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70004ce0 con 0x7fef88106150 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.732+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8819dd70 con 0x7fef88106150 2024-04-21T08:46:00.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.732+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8810e530 con 0x7fef88106150 2024-04-21T08:46:00.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.735+0000 7fef8f59b740 1 Processor -- start 2024-04-21T08:46:00.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 -- start start 2024-04-21T08:46:00.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f2f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f38540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f3ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55eb07e941e0 con 0x55eb07f38a80 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55eb07e31390 con 0x55eb07f2f580 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8f59b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55eb07e901a0 con 0x55eb07f2fea0 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef86d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f2f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef86d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f2f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:41324/0 (socket says 172.21.15.57:41324) 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef86d9d640 1 -- 172.21.15.57:0/850653855 learned_addr learned my addr 172.21.15.57:0/850653855 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.736+0000 7fef8659c640 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f38540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef86d9d640 1 -- 172.21.15.57:0/850653855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 msgr2=0x55eb07f38540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef86d9d640 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f38540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef86d9d640 1 -- 172.21.15.57:0/850653855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 msgr2=0x55eb07f3ae70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:46:00.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef86d9d640 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f3ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef86d9d640 1 -- 172.21.15.57:0/850653855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb07c707d0 con 0x55eb07f2f580 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef8659c640 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f38540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef86d9d640 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f2f960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef80009f70 tx=0x7fef80010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b118bb3dd8f99cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef7cff9640 1 -- 172.21.15.57:0/850653855 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80016070 con 0x55eb07f2f580 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef7cff9640 1 -- 172.21.15.57:0/850653855 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef80004d20 con 0x55eb07f2f580 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef7cff9640 1 -- 172.21.15.57:0/850653855 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80012870 con 0x55eb07f2f580 2024-04-21T08:46:00.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.737+0000 7fef7f7fe640 1 -- 172.21.15.57:0/4289541441 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fef70012760 con 0x7fef88106150 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.738+0000 7fef8f59b740 1 -- 172.21.15.57:0/850653855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 msgr2=0x55eb07f2f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.738+0000 7fef8f59b740 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f2f960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fef80009f70 tx=0x7fef80010800 comp rx=0 tx=0).stop 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.738+0000 7fef7f7fe640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fef740a3db0 0x7fef740a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.738+0000 7fef8f59b740 1 -- 172.21.15.57:0/850653855 shutdown_connections 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.738+0000 7fef7f7fe640 1 -- 172.21.15.57:0/4289541441 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fef700d47a0 con 0x7fef88106150 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.738+0000 7fef8659c640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fef740a3db0 0x7fef740a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8f59b740 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f3ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8f59b740 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f38540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8f59b740 1 --2- 172.21.15.57:0/850653855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f2f960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8f59b740 1 -- 172.21.15.57:0/850653855 >> 172.21.15.57:0/850653855 conn(0x55eb07f29cb0 msgr2=0x55eb07f2a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8659c640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fef740a3db0 0x7fef740a6270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef78006010 tx=0x7fef78005e40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8f59b740 1 -- 172.21.15.57:0/850653855 shutdown_connections 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.739+0000 7fef8f59b740 1 -- 172.21.15.57:0/850653855 wait complete. 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.740+0000 7fef8f59b740 1 -- start start 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.740+0000 7fef8f59b740 1 Processor -- start 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.740+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f35b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f36050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f36590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f36590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f36590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59268/0 (socket says 172.21.15.57:59268) 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef86d9d640 1 -- 172.21.15.57:0/3543541606 learned_addr learned my addr 172.21.15.57:0/3543541606 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8659c640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f35b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55eb07f3b870 con 0x55eb07f38a80 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55eb07e4d2f0 con 0x55eb07f2f580 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef7ffff640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f36050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55eb07f3b9f0 con 0x55eb07f2fea0 2024-04-21T08:46:00.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef7ffff640 1 -- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 msgr2=0x55eb07f35b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef7ffff640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f35b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef7ffff640 1 -- 172.21.15.57:0/3543541606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 msgr2=0x55eb07f36590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef7ffff640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f36590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef7ffff640 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb07ba63f0 con 0x55eb07f2fea0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef8659c640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f35b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.741+0000 7fef86d9d640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f36590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef8f59b740 10 client.0 _ll_register_callbacks cb 0x55eb07e5a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef7ffff640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f36050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fef700d1be0 tx=0x7fef700a0a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c143e9e89e56ed70 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef80004d20 con 0x55eb07f2fea0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef6dffb640 1 -- 172.21.15.57:0/3543541606 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef700bf050 con 0x55eb07f2fea0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef6dffb640 1 -- 172.21.15.57:0/3543541606 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fef700bf1f0 con 0x55eb07f2fea0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef6dffb640 1 -- 172.21.15.57:0/3543541606 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fef700a4730 con 0x55eb07f2fea0 2024-04-21T08:46:00.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef8f59b740 10 client.25339 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.742+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55eb07b660f0 con 0x55eb07f2fea0 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef6dffb640 1 -- 172.21.15.57:0/3543541606 <== mon.1 v2:172.21.15.136:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fef70083060 con 0x55eb07f2fea0 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8f59b740 10 client.25339 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8f59b740 10 client.25339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8f59b740 20 client.25339 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8f59b740 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07f48640 0x55eb07f4aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8f59b740 4 client.25339 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb07ea1c00 con 0x55eb07f48640 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.743+0000 7fef8659c640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07f48640 0x55eb07f4aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.744+0000 7fef6dffb640 10 client.25339 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.744+0000 7fef8659c640 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07f48640 0x55eb07f4aa90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86dd51c53cd3c7d3 server_cookie=b73e2c104887f76c in_seq=0 out_seq=0 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.744+0000 7fef6dffb640 1 -- 172.21.15.57:0/3543541606 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb07ea1c00 con 0x55eb07f48640 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.744+0000 7fef6dffb640 10 client.25339 handle_command_reply: tid=0 2024-04-21T08:46:00.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.744+0000 7fef8f59b740 1 client.25339 shutdown 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 msgr2=0x55eb07f36050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f36050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fef700d1be0 tx=0x7fef700a0a80 comp rx=0 tx=0).stop 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 shutdown_connections 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07f48640 0x55eb07f4aa90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f36590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef6dffb640 0 client.25339 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2fea0 0x55eb07f36050 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 --2- 172.21.15.57:0/3543541606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2f580 0x55eb07f35b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 >> 172.21.15.57:0/3543541606 conn(0x55eb07f29cb0 msgr2=0x55eb07f2ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 shutdown_connections 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 1 -- 172.21.15.57:0/3543541606 wait complete. 2024-04-21T08:46:00.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.745+0000 7fef8f59b740 20 client.25339 trim_cache size 0 max 16384 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 Processor -- start 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 -- start start 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 0x55eb07f38240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f44730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f38a80 0x55eb07f2b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fef881ab070 con 0x55eb07f2f580 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef881ad8d0 con 0x55eb07f2fea0 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8f59b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fef881ab6c0 con 0x55eb07f38a80 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 0x55eb07f38240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 0x55eb07f38240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59274/0 (socket says 172.21.15.57:59274) 2024-04-21T08:46:00.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.779+0000 7fef8659c640 1 -- 172.21.15.57:0/1339811264 learned_addr learned my addr 172.21.15.57:0/1339811264 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef7ffff640 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f44730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef86d9d640 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f38a80 0x55eb07f2b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef8659c640 1 -- 172.21.15.57:0/1339811264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f38a80 msgr2=0x55eb07f2b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef8659c640 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f38a80 0x55eb07f2b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef8659c640 1 -- 172.21.15.57:0/1339811264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 msgr2=0x55eb07f44730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef8659c640 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f44730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef8659c640 1 -- 172.21.15.57:0/1339811264 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb07f297e0 con 0x55eb07f2f580 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef8659c640 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 0x55eb07f38240 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fef7802c170 tx=0x7fef78045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac74467159cd4ab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef6ffff640 1 -- 172.21.15.57:0/1339811264 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78045070 con 0x55eb07f2f580 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef6ffff640 1 -- 172.21.15.57:0/1339811264 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef7805b070 con 0x55eb07f2f580 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.780+0000 7fef6ffff640 1 -- 172.21.15.57:0/1339811264 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef780574f0 con 0x55eb07f2f580 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 -- 172.21.15.57:0/1339811264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 msgr2=0x55eb07f38240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 0x55eb07f38240 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fef7802c170 tx=0x7fef78045000 comp rx=0 tx=0).stop 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 -- 172.21.15.57:0/1339811264 shutdown_connections 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f38a80 0x55eb07f2b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f44730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 --2- 172.21.15.57:0/1339811264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f2f580 0x55eb07f38240 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 -- 172.21.15.57:0/1339811264 >> 172.21.15.57:0/1339811264 conn(0x55eb07f29cb0 msgr2=0x55eb07f2b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 -- 172.21.15.57:0/1339811264 shutdown_connections 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.781+0000 7fef8f59b740 1 -- 172.21.15.57:0/1339811264 wait complete. 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 -- start start 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 Processor -- start 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2f580 0x55eb07f40f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f323e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f32920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fef881abd90 con 0x55eb07f38a80 2024-04-21T08:46:00.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef881ad200 con 0x55eb07f2fea0 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fef881ac460 con 0x55eb07f2f580 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f32920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f32920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59276/0 (socket says 172.21.15.57:59276) 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef7ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef86d9d640 1 -- 172.21.15.57:0/2826615072 learned_addr learned my addr 172.21.15.57:0/2826615072 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.782+0000 7fef8f59b740 10 client.0 _ll_register_callbacks cb 0x7fef700d1a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef8659c640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2f580 0x55eb07f40f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef7ffff640 1 -- 172.21.15.57:0/2826615072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2f580 msgr2=0x55eb07f40f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef7ffff640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2f580 0x55eb07f40f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef7ffff640 1 -- 172.21.15.57:0/2826615072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 msgr2=0x55eb07f32920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef7ffff640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f32920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef7ffff640 1 -- 172.21.15.57:0/2826615072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb07f330d0 con 0x55eb07f2fea0 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef86d9d640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f32920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:00.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef7ffff640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f323e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef7009e630 tx=0x7fef70014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed3f069c549dc43b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef6dffb640 1 -- 172.21.15.57:0/2826615072 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef700bf040 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef7805b070 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.783+0000 7fef6dffb640 1 -- 172.21.15.57:0/2826615072 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fef700d4420 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef6dffb640 1 -- 172.21.15.57:0/2826615072 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fef7009fec0 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 10 client.25370 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55eb07fb7350 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef6dffb640 1 -- 172.21.15.57:0/2826615072 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef7805b070 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef6dffb640 1 -- 172.21.15.57:0/2826615072 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fef7009c0d0 con 0x55eb07f2fea0 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 10 client.25370 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 10 client.25370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:00.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 20 client.25370 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07fb7880 0x55eb07fb9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 4 client.25370 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eb07e9ba70 con 0x55eb07fb7880 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.784+0000 7fef8659c640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07fb7880 0x55eb07fb9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef6dffb640 10 client.25370 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8659c640 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07fb7880 0x55eb07fb9c70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8de1f107a1e2d45 server_cookie=566e36c361e33e44 in_seq=0 out_seq=0 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef6dffb640 1 -- 172.21.15.57:0/2826615072 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55eb07e9ba70 con 0x55eb07fb7880 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef6dffb640 10 client.25370 handle_command_reply: tid=0 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 client.25370 shutdown 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 msgr2=0x55eb07f323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f323e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fef7009e630 tx=0x7fef70014000 comp rx=0 tx=0).stop 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 shutdown_connections 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55eb07fb7880 0x55eb07fb9c70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55eb07f38a80 0x55eb07f32920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eb07f2fea0 0x55eb07f323e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 --2- 172.21.15.57:0/2826615072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55eb07f2f580 0x55eb07f40f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 >> 172.21.15.57:0/2826615072 conn(0x55eb07f29cb0 msgr2=0x55eb07f35ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef6dffb640 0 client.25370 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 shutdown_connections 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.785+0000 7fef8f59b740 1 -- 172.21.15.57:0/2826615072 wait complete. 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8f59b740 20 client.25370 trim_cache size 0 max 16384 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:46:00.792 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:46:00.793 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:46:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fef740a3db0 msgr2=0x7fef740a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fef740a3db0 0x7fef740a6270 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fef78006010 tx=0x7fef78005e40 comp rx=0 tx=0).stop 2024-04-21T08:46:00.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 msgr2=0x7fef88198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 0x7fef88198c10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fef7000d740 tx=0x7fef7000dc00 comp rx=0 tx=0).stop 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 shutdown_connections 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fef740a3db0 0x7fef740a6270 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef8819cf90 0x7fef8819d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef88106150 0x7fef88198c10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 --2- 172.21.15.57:0/4289541441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef88101080 0x7fef881986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 >> 172.21.15.57:0/4289541441 conn(0x7fef88078000 msgr2=0x7fef880ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 shutdown_connections 2024-04-21T08:46:00.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.786+0000 7fef8c9b1640 1 -- 172.21.15.57:0/4289541441 wait complete. 2024-04-21T08:46:00.799 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:00 smithi057 ceph-mon[27968]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:46:00.801 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:46:00.802 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:46:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:00 smithi129 ceph-mon[31272]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:46:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:00 smithi136 ceph-mon[31791]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:46:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.905+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/861068734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 msgr2=0x7f8e1410ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.905+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/861068734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e1410ede0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c009ca0 tx=0x7f8e0c02c120 comp rx=0 tx=0).stop 2024-04-21T08:46:00.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/861068734 shutdown_connections 2024-04-21T08:46:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/861068734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e1410ede0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/861068734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e140690e0 0x7f8e14069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/861068734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e14106f50 0x7f8e14107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/861068734 >> 172.21.15.57:0/861068734 conn(0x7f8e14077e30 msgr2=0x7f8e14078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/861068734 shutdown_connections 2024-04-21T08:46:00.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/861068734 wait complete. 2024-04-21T08:46:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 Processor -- start 2024-04-21T08:46:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.906+0000 7f8e1b0eb640 1 -- start start 2024-04-21T08:46:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e1b0eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e140690e0 0x7f8e14198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e1b0eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e14198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e1b0eb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e14106f50 0x7f8e1419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e1b0eb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8e14112150 con 0x7f8e14069aa0 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e1b0eb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e14111fd0 con 0x7f8e140690e0 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e1b0eb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8e141122d0 con 0x7f8e14106f50 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e14198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e14198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59296/0 (socket says 172.21.15.57:59296) 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 -- 172.21.15.57:0/1829783911 learned_addr learned my addr 172.21.15.57:0/1829783911 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e19688640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e14106f50 0x7f8e1419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e18e87640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e140690e0 0x7f8e14198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 -- 172.21.15.57:0/1829783911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e14106f50 msgr2=0x7f8e1419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e14106f50 0x7f8e1419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 -- 172.21.15.57:0/1829783911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e140690e0 msgr2=0x7f8e14198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e140690e0 0x7f8e14198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 -- 172.21.15.57:0/1829783911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1419d5d0 con 0x7f8e14069aa0 2024-04-21T08:46:00.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e18e87640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e140690e0 0x7f8e14198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.907+0000 7f8e13fff640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e14198b70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000e9f0 tx=0x7f8e0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.908+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000cd80 con 0x7f8e14069aa0 2024-04-21T08:46:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.908+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e000108e0 con 0x7f8e14069aa0 2024-04-21T08:46:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.908+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00018980 con 0x7f8e14069aa0 2024-04-21T08:46:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.908+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1419d8c0 con 0x7f8e14069aa0 2024-04-21T08:46:00.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.908+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e141a5150 con 0x7f8e14069aa0 2024-04-21T08:46:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.910+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e14068de0 con 0x7f8e14069aa0 2024-04-21T08:46:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.911+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f8e00010430 con 0x7f8e14069aa0 2024-04-21T08:46:00.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.911+0000 7f8e11ffb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8e040a3e30 0x7f8e040a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.912+0000 7f8e18e87640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8e040a3e30 0x7f8e040a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.912+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8e000d1250 con 0x7f8e14069aa0 2024-04-21T08:46:00.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.912+0000 7f8e18e87640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8e040a3e30 0x7f8e040a62f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009c30 tx=0x7f8e08005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:00.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:00.916+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8e0009d420 con 0x7f8e14069aa0 2024-04-21T08:46:01.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.147+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e1402d5c0 con 0x7f8e14069aa0 2024-04-21T08:46:01.147 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.148+0000 7f8e11ffb640 1 -- 172.21.15.57:0/1829783911 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f8e0009cd20 con 0x7f8e14069aa0 2024-04-21T08:46:01.147 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:46:01.148 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:46:01.148 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8e040a3e30 msgr2=0x7f8e040a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8e040a3e30 0x7f8e040a62f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009c30 tx=0x7f8e08005f90 comp rx=0 tx=0).stop 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 msgr2=0x7f8e14198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e14198b70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000e9f0 tx=0x7f8e0000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 shutdown_connections 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8e040a3e30 0x7f8e040a62f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e14106f50 0x7f8e1419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e14069aa0 0x7f8e14198b70 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 --2- 172.21.15.57:0/1829783911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e140690e0 0x7f8e14198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 >> 172.21.15.57:0/1829783911 conn(0x7f8e14077e30 msgr2=0x7f8e140fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.150 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 shutdown_connections 2024-04-21T08:46:01.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.150+0000 7f8e1b0eb640 1 -- 172.21.15.57:0/1829783911 wait complete. 2024-04-21T08:46:01.161 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:46:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.279+0000 7f94a364e640 1 -- 172.21.15.57:0/4117532620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 msgr2=0x7f949c110d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.279+0000 7f94a364e640 1 --2- 172.21.15.57:0/4117532620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c110d50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9494009ca0 tx=0x7f949402c100 comp rx=0 tx=0).stop 2024-04-21T08:46:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.279+0000 7f94a364e640 1 -- 172.21.15.57:0/4117532620 shutdown_connections 2024-04-21T08:46:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.279+0000 7f94a364e640 1 --2- 172.21.15.57:0/4117532620 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c110d50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 --2- 172.21.15.57:0/4117532620 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f949c1016d0 0x7f949c10e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 --2- 172.21.15.57:0/4117532620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f949c100db0 0x7f949c101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 -- 172.21.15.57:0/4117532620 >> 172.21.15.57:0/4117532620 conn(0x7f949c077fb0 msgr2=0x7f949c0783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 -- 172.21.15.57:0/4117532620 shutdown_connections 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 -- 172.21.15.57:0/4117532620 wait complete. 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 Processor -- start 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.280+0000 7f94a364e640 1 -- start start 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a364e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f949c100db0 0x7f949c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a364e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f949c1016d0 0x7f949c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a364e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a364e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f949c1143b0 con 0x7f949c10e960 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a364e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f949c114230 con 0x7f949c1016d0 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a364e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f949c114530 con 0x7f949c100db0 2024-04-21T08:46:01.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59326/0 (socket says 172.21.15.57:59326) 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 -- 172.21.15.57:0/3542444973 learned_addr learned my addr 172.21.15.57:0/3542444973 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f949c100db0 0x7f949c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a0be9640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f949c1016d0 0x7f949c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 -- 172.21.15.57:0/3542444973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f949c100db0 msgr2=0x7f949c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f949c100db0 0x7f949c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 -- 172.21.15.57:0/3542444973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f949c1016d0 msgr2=0x7f949c1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f949c1016d0 0x7f949c1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 -- 172.21.15.57:0/3542444973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949c19b050 con 0x7f949c10e960 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.281+0000 7f94a1beb640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c19a790 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9494005cc0 tx=0x7f9494035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.282+0000 7f94927fc640 1 -- 172.21.15.57:0/3542444973 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494039040 con 0x7f949c10e960 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.282+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949c19b2e0 con 0x7f949c10e960 2024-04-21T08:46:01.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.282+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949c1a7600 con 0x7f949c10e960 2024-04-21T08:46:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.282+0000 7f94927fc640 1 -- 172.21.15.57:0/3542444973 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9494044070 con 0x7f949c10e960 2024-04-21T08:46:01.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.282+0000 7f94927fc640 1 -- 172.21.15.57:0/3542444973 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494004310 con 0x7f949c10e960 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.284+0000 7f94927fc640 1 -- 172.21.15.57:0/3542444973 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f94940044b0 con 0x7f949c10e960 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.285+0000 7f94927fc640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f94880a3e30 0x7f94880a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.285+0000 7f94a6243740 1 Processor -- start 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.285+0000 7f94927fc640 1 -- 172.21.15.57:0/3542444973 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f94940f3070 con 0x7f949c10e960 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.285+0000 7f94a6243740 1 -- start start 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.285+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f94880a3e30 0x7f94880a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a6243740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 0x5585e29b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a6243740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29c2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a6243740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f94880a3e30 0x7f94880a62f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9498005d00 tx=0x7f949800b410 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a6243740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5585e291e1f0 con 0x5585e29b9e80 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a6243740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5585e28bb4b0 con 0x5585e29c2a00 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a6243740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5585e291a1b0 con 0x5585e29b94d0 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a0be9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 0x5585e29b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a0be9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29c2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59336/0 (socket says 172.21.15.57:59336) 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 0x5585e29b98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:49362/0 (socket says 172.21.15.57:49362) 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a0be9640 1 -- 172.21.15.57:0/3957732075 learned_addr learned my addr 172.21.15.57:0/3957732075 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a1beb640 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 -- 172.21.15.57:0/3957732075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 msgr2=0x5585e29c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 -- 172.21.15.57:0/3957732075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 msgr2=0x5585e29c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a13ea640 1 -- 172.21.15.57:0/3957732075 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e28ad1b0 con 0x5585e29b94d0 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.286+0000 7f94a0be9640 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.287+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 0x5585e29b98b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f949800bf10 tx=0x7f9498013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bcd8c3ab82d183a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.287+0000 7f94877fe640 1 -- 172.21.15.57:0/3957732075 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949801e620 con 0x5585e29b94d0 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.287+0000 7f94877fe640 1 -- 172.21.15.57:0/3957732075 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9498011e50 con 0x5585e29b94d0 2024-04-21T08:46:01.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.287+0000 7f94877fe640 1 -- 172.21.15.57:0/3957732075 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949801db50 con 0x5585e29b94d0 2024-04-21T08:46:01.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 -- 172.21.15.57:0/3957732075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 msgr2=0x5585e29b98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 0x5585e29b98b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f949800bf10 tx=0x7f9498013000 comp rx=0 tx=0).stop 2024-04-21T08:46:01.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 -- 172.21.15.57:0/3957732075 shutdown_connections 2024-04-21T08:46:01.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 --2- 172.21.15.57:0/3957732075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b94d0 0x5585e29b98b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 -- 172.21.15.57:0/3957732075 >> 172.21.15.57:0/3957732075 conn(0x5585e29b3ba0 msgr2=0x5585e29b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 -- 172.21.15.57:0/3957732075 shutdown_connections 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 -- 172.21.15.57:0/3957732075 wait complete. 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 -- start start 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.288+0000 7f94a6243740 1 Processor -- start 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29c2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b9e80 0x5585e29bf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29bfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5585e29c5970 con 0x5585e29b94d0 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5585e28d7500 con 0x5585e29b9e80 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5585e29c5af0 con 0x5585e29c2a00 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29c2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a6243740 10 client.0 _ll_register_callbacks cb 0x5585e28a5b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29c2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59350/0 (socket says 172.21.15.57:59350) 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 -- 172.21.15.57:0/1437893041 learned_addr learned my addr 172.21.15.57:0/1437893041 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a1beb640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29bfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a0be9640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b9e80 0x5585e29bf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 -- 172.21.15.57:0/1437893041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 msgr2=0x5585e29bfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29bfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 -- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b9e80 msgr2=0x5585e29bf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b9e80 0x5585e29bf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.289+0000 7f94a13ea640 1 -- 172.21.15.57:0/1437893041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e26303f0 con 0x5585e29b94d0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a13ea640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29c2270 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f949800b500 tx=0x7f949801d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f894a51bedc6f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f9484ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f9484ff9640 1 -- 172.21.15.57:0/1437893041 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498036060 con 0x5585e29b94d0 2024-04-21T08:46:01.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f9484ff9640 1 -- 172.21.15.57:0/1437893041 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f949801dd40 con 0x5585e29b94d0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9498011e50 con 0x5585e29b94d0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f9484ff9640 1 -- 172.21.15.57:0/1437893041 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f949803d050 con 0x5585e29b94d0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 10 client.15846 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5585e26fa7d0 con 0x5585e29b94d0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f9484ff9640 1 -- 172.21.15.57:0/1437893041 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f949801e8d0 con 0x5585e29b94d0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f9484ff9640 1 -- 172.21.15.57:0/1437893041 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f94980316f0 con 0x5585e29b94d0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 10 client.15846 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 20 client.15846 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e29d4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 4 client.15846 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.290+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9498011e50 con 0x5585e29d2ba0 2024-04-21T08:46:01.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.291+0000 7f94a0be9640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e29d4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.291+0000 7f9484ff9640 10 client.15846 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.291+0000 7f94a0be9640 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e29d4f90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8042b1d334adfb server_cookie=8b64ec6cf02a377 in_seq=0 out_seq=0 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f9484ff9640 1 -- 172.21.15.57:0/1437893041 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f9498011e50 con 0x5585e29d2ba0 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f9484ff9640 10 client.15846 handle_command_reply: tid=0 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 client.15846 shutdown 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 msgr2=0x5585e29c2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29c2270 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f949800b500 tx=0x7f949801d850 comp rx=0 tx=0).stop 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 shutdown_connections 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e29d4f90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f9484ff9640 0 client.15846 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29bfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b9e80 0x5585e29bf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 --2- 172.21.15.57:0/1437893041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29c2270 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 >> 172.21.15.57:0/1437893041 conn(0x5585e29b3ba0 msgr2=0x5585e29b58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 shutdown_connections 2024-04-21T08:46:01.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 1 -- 172.21.15.57:0/1437893041 wait complete. 2024-04-21T08:46:01.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.292+0000 7f94a6243740 20 client.15846 trim_cache size 0 max 16384 2024-04-21T08:46:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 Processor -- start 2024-04-21T08:46:01.306 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 -- start start 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b94d0 0x5585e29ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29bad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f948000ddf0 con 0x5585e29b9e80 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9480010650 con 0x5585e29b94d0 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a6243740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f948000e440 con 0x5585e29c2a00 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a0be9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29bad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a0be9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29bad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59352/0 (socket says 172.21.15.57:59352) 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a0be9640 1 -- 172.21.15.57:0/3908593329 learned_addr learned my addr 172.21.15.57:0/3908593329 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b94d0 0x5585e29ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.306+0000 7f94a1beb640 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.307 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a0be9640 1 -- 172.21.15.57:0/3908593329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 msgr2=0x5585e29c1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a0be9640 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29c1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a0be9640 1 -- 172.21.15.57:0/3908593329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b94d0 msgr2=0x5585e29ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a0be9640 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b94d0 0x5585e29ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a0be9640 1 -- 172.21.15.57:0/3908593329 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e29c0140 con 0x5585e29b9e80 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a13ea640 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b94d0 0x5585e29ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f94a0be9640 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29bad80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x5585e29ba280 tx=0x7f948c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265bf1d623551481 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f9486ffd640 1 -- 172.21.15.57:0/3908593329 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c01d430 con 0x5585e29b9e80 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f9486ffd640 1 -- 172.21.15.57:0/3908593329 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948c023070 con 0x5585e29b9e80 2024-04-21T08:46:01.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.307+0000 7f9486ffd640 1 -- 172.21.15.57:0/3908593329 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c01c3d0 con 0x5585e29b9e80 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 -- 172.21.15.57:0/3908593329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 msgr2=0x5585e29bad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29bad80 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x5585e29ba280 tx=0x7f948c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 -- 172.21.15.57:0/3908593329 shutdown_connections 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29c2a00 0x5585e29c1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b9e80 0x5585e29bad80 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 --2- 172.21.15.57:0/3908593329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29b94d0 0x5585e29ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 -- 172.21.15.57:0/3908593329 >> 172.21.15.57:0/3908593329 conn(0x5585e29b3ba0 msgr2=0x5585e29b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 -- 172.21.15.57:0/3908593329 shutdown_connections 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.308+0000 7f94a6243740 1 -- 172.21.15.57:0/3908593329 wait complete. 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 -- start start 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 Processor -- start 2024-04-21T08:46:01.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29cb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b9e80 0x5585e29c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f948000eb10 con 0x5585e29b94d0 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a13ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f948000ff80 con 0x5585e29c2a00 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f948000f1e0 con 0x5585e29b9e80 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a13ea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59356/0 (socket says 172.21.15.57:59356) 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a13ea640 1 -- 172.21.15.57:0/809051509 learned_addr learned my addr 172.21.15.57:0/809051509 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.309+0000 7f94a6243740 10 client.0 _ll_register_callbacks cb 0x7f949800bc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a0be9640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b9e80 0x5585e29c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a1beb640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a13ea640 1 -- 172.21.15.57:0/809051509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b9e80 msgr2=0x5585e29c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a13ea640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b9e80 0x5585e29c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a13ea640 1 -- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 msgr2=0x5585e29c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a13ea640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a13ea640 1 -- 172.21.15.57:0/809051509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e29c70a0 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f94a13ea640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29cb060 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9498031b40 tx=0x7f9498031f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc5c8eb70fd48d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f9484ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.310+0000 7f9484ff9640 1 -- 172.21.15.57:0/809051509 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498046060 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f9484ff9640 1 -- 172.21.15.57:0/809051509 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9498031470 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948c023070 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f9484ff9640 1 -- 172.21.15.57:0/809051509 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f949804d050 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f94a6243740 10 client.15852 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5585e2a38d50 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f9484ff9640 1 -- 172.21.15.57:0/809051509 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948c023070 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f9484ff9640 1 -- 172.21.15.57:0/809051509 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9498026a10 con 0x5585e29b94d0 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f94a6243740 10 client.15852 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:01.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f94a6243740 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.311+0000 7f94a6243740 20 client.15852 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.312+0000 7f94a6243740 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e2a3b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.312+0000 7f94a6243740 4 client.15852 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.312+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585e28bd930 con 0x5585e29d2ba0 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.312+0000 7f94a0be9640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e2a3b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.312+0000 7f94a0be9640 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e2a3b420 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb3630b7589bcf5c server_cookie=564cb629b3658543 in_seq=0 out_seq=0 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.312+0000 7f9484ff9640 10 client.15852 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f9484ff9640 1 -- 172.21.15.57:0/809051509 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585e28bd930 con 0x5585e29d2ba0 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f9484ff9640 10 client.15852 handle_command_reply: tid=0 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 client.15852 shutdown 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 msgr2=0x5585e29cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29cb060 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9498031b40 tx=0x7f9498031f60 comp rx=0 tx=0).stop 2024-04-21T08:46:01.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 shutdown_connections 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5585e29d2ba0 0x5585e2a3b420 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585e29c2a00 0x5585e29c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5585e29b9e80 0x5585e29c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 --2- 172.21.15.57:0/809051509 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5585e29b94d0 0x5585e29cb060 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 >> 172.21.15.57:0/809051509 conn(0x5585e29b3ba0 msgr2=0x5585e29b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f9484ff9640 0 client.15852 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 shutdown_connections 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 1 -- 172.21.15.57:0/809051509 wait complete. 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.313+0000 7f94a6243740 20 client.15852 trim_cache size 0 max 16384 2024-04-21T08:46:01.317 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f94880a3e30 msgr2=0x7f94880a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f94880a3e30 0x7f94880a62f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9498005d00 tx=0x7f949800b410 comp rx=0 tx=0).stop 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 msgr2=0x7f949c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c19a790 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9494005cc0 tx=0x7f9494035d60 comp rx=0 tx=0).stop 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 shutdown_connections 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f94880a3e30 0x7f94880a62f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f949c10e960 0x7f949c19a790 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f949c1016d0 0x7f949c1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 --2- 172.21.15.57:0/3542444973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f949c100db0 0x7f949c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:01.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 >> 172.21.15.57:0/3542444973 conn(0x7f949c077fb0 msgr2=0x7f949c10fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:01.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 shutdown_connections 2024-04-21T08:46:01.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:01.314+0000 7f94a364e640 1 -- 172.21.15.57:0/3542444973 wait complete. 2024-04-21T08:46:01.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:01 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1829783911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:01.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:01 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1829783911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:01.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:01 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1829783911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:02.325 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:46:02.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:46:02.325 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 -- 172.21.15.57:0/1929490135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a8100d50 msgr2=0x7fd4a8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 --2- 172.21.15.57:0/1929490135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a8100d50 0x7fd4a8101130 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009900 tx=0x7fd4a402bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 -- 172.21.15.57:0/1929490135 shutdown_connections 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 --2- 172.21.15.57:0/1929490135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4a810e900 0x7fd4a8110cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 --2- 172.21.15.57:0/1929490135 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8101670 0x7fd4a810e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 --2- 172.21.15.57:0/1929490135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a8100d50 0x7fd4a8101130 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 -- 172.21.15.57:0/1929490135 >> 172.21.15.57:0/1929490135 conn(0x7fd4a8077f50 msgr2=0x7fd4a8078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 -- 172.21.15.57:0/1929490135 shutdown_connections 2024-04-21T08:46:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.435+0000 7fd4b023f640 1 -- 172.21.15.57:0/1929490135 wait complete. 2024-04-21T08:46:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 Processor -- start 2024-04-21T08:46:02.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 -- start start 2024-04-21T08:46:02.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8100d50 0x7fd4a81a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4a8101670 0x7fd4a81a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 0x7fd4a819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81143d0 con 0x7fd4a810e900 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8114250 con 0x7fd4a8101670 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4b023f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a8114550 con 0x7fd4a8100d50 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4ae7dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 0x7fd4a819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4ae7dc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 0x7fd4a819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59372/0 (socket says 172.21.15.57:59372) 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4ae7dc640 1 -- 172.21.15.57:0/3902309820 learned_addr learned my addr 172.21.15.57:0/3902309820 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8100d50 0x7fd4a81a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.436+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4a8101670 0x7fd4a81a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4ae7dc640 1 -- 172.21.15.57:0/3902309820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8100d50 msgr2=0x7fd4a81a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4ae7dc640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8100d50 0x7fd4a81a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4ae7dc640 1 -- 172.21.15.57:0/3902309820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4a8101670 msgr2=0x7fd4a81a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4ae7dc640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4a8101670 0x7fd4a81a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4ae7dc640 1 -- 172.21.15.57:0/3902309820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a819af80 con 0x7fd4a810e900 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8100d50 0x7fd4a81a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4ae7dc640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 0x7fd4a819a7b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000dac0 tx=0x7fd4a000df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd49effd640 1 -- 172.21.15.57:0/3902309820 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a0017070 con 0x7fd4a810e900 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd49effd640 1 -- 172.21.15.57:0/3902309820 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4a000be30 con 0x7fd4a810e900 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd49effd640 1 -- 172.21.15.57:0/3902309820 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000f820 con 0x7fd4a810e900 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a819b270 con 0x7fd4a810e900 2024-04-21T08:46:02.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.437+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a81a7490 con 0x7fd4a810e900 2024-04-21T08:46:02.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd49effd640 1 -- 172.21.15.57:0/3902309820 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fd4a0010ac0 con 0x7fd4a810e900 2024-04-21T08:46:02.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 Processor -- start 2024-04-21T08:46:02.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 -- start start 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac20cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac29840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56175ab84f00 con 0x56175ac29e10 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56175ab22590 con 0x56175ac21290 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.440+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56175ab80ec0 con 0x56175ac208e0 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59386/0 (socket says 172.21.15.57:59386) 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 -- 172.21.15.57:0/676724063 learned_addr learned my addr 172.21.15.57:0/676724063 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd49effd640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4940a3ec0 0x7fd4940a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac20cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac29840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 -- 172.21.15.57:0/676724063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 msgr2=0x56175ac29840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac29840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 -- 172.21.15.57:0/676724063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 msgr2=0x56175ac2c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 -- 172.21.15.57:0/676724063 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56175a95f7e0 con 0x56175ac208e0 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac29840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd49effd640 1 -- 172.21.15.57:0/3902309820 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd4a00d0ba0 con 0x7fd4a810e900 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ae7dc640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4940a3ec0 0x7fd4940a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.441+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac20cc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4980058c0 tx=0x7fd498005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47fd14eb52715 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.442+0000 7fd493fff640 1 -- 172.21.15.57:0/676724063 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498011d30 con 0x56175ac208e0 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.442+0000 7fd4ae7dc640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4940a3ec0 0x7fd4940a6380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00cde70 tx=0x7fd4a0010db0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.442+0000 7fd493fff640 1 -- 172.21.15.57:0/676724063 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd498005020 con 0x56175ac208e0 2024-04-21T08:46:02.443 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.442+0000 7fd493fff640 1 -- 172.21.15.57:0/676724063 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49800fa20 con 0x56175ac208e0 2024-04-21T08:46:02.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.442+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/676724063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 msgr2=0x56175ac20cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.442+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac20cc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd4980058c0 tx=0x7fd498005f90 comp rx=0 tx=0).stop 2024-04-21T08:46:02.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/676724063 shutdown_connections 2024-04-21T08:46:02.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac29840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/676724063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac20cc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/676724063 >> 172.21.15.57:0/676724063 conn(0x56175ac1afb0 msgr2=0x56175ac1b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/676724063 shutdown_connections 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/676724063 wait complete. 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 -- start start 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.443+0000 7fd4b2e2e740 1 Processor -- start 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac26ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac27410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac27950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56175ac2cd80 con 0x56175ac21290 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56175ab3e580 con 0x56175ac208e0 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56175ac2cf00 con 0x56175ac29e10 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4b2e2e740 10 client.0 _ll_register_callbacks cb 0x56175ab44650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac27410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac26ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac26ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:41450/0 (socket says 172.21.15.57:41450) 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4ad7da640 1 -- 172.21.15.57:0/1798255094 learned_addr learned my addr 172.21.15.57:0/1798255094 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac27410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59402/0 (socket says 172.21.15.57:59402) 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac27950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 -- 172.21.15.57:0/1798255094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 msgr2=0x56175ac27950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac27950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 -- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 msgr2=0x56175ac26ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac26ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 -- 172.21.15.57:0/1798255094 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56175a9bff70 con 0x56175ac21290 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac26ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.444+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac27410 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4000c00 tx=0x7fd4a4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a311b197311a774 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4917fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4917fa640 1 -- 172.21.15.57:0/1798255094 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40035f0 con 0x56175ac21290 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd498005020 con 0x56175ac21290 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4917fa640 1 -- 172.21.15.57:0/1798255094 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd4a4003790 con 0x56175ac21290 2024-04-21T08:46:02.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4917fa640 1 -- 172.21.15.57:0/1798255094 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd4a4003990 con 0x56175ac21290 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 10 client.15864 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56175a857170 con 0x56175ac21290 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4917fa640 1 -- 172.21.15.57:0/1798255094 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd498005020 con 0x56175ac21290 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4917fa640 1 -- 172.21.15.57:0/1798255094 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd4a404ace0 con 0x56175ac21290 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 10 client.15864 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 20 client.15864 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.445+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175ac3a440 0x56175ac3c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.446+0000 7fd4b2e2e740 4 client.15864 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.446+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56175ab92920 con 0x56175ac3a440 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.446+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175ac3a440 0x56175ac3c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.446+0000 7fd4917fa640 10 client.15864 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.446+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175ac3a440 0x56175ac3c830 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4413021b6793b93a server_cookie=e5943c2fb7a971db in_seq=0 out_seq=0 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4917fa640 1 -- 172.21.15.57:0/1798255094 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56175ab92920 con 0x56175ac3a440 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4917fa640 10 client.15864 handle_command_reply: tid=0 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 client.15864 shutdown 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 msgr2=0x56175ac27410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac27410 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4000c00 tx=0x7fd4a4004270 comp rx=0 tx=0).stop 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 shutdown_connections 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175ac3a440 0x56175ac3c830 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4917fa640 0 client.15864 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac27950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac27410 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/1798255094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac26ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.447+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 >> 172.21.15.57:0/1798255094 conn(0x56175ac1afb0 msgr2=0x56175ac1cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.448+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 shutdown_connections 2024-04-21T08:46:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.448+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/1798255094 wait complete. 2024-04-21T08:46:02.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.448+0000 7fd4b2e2e740 20 client.15864 trim_cache size 0 max 16384 2024-04-21T08:46:02.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.467+0000 7fd4b2e2e740 1 Processor -- start 2024-04-21T08:46:02.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.467+0000 7fd4b2e2e740 1 -- start start 2024-04-21T08:46:02.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac29600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac35c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac25e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81af550 con 0x56175ac21290 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81b1db0 con 0x56175ac208e0 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81afba0 con 0x56175ac29e10 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4adfdb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac35c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac29600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4adfdb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac35c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59412/0 (socket says 172.21.15.57:59412) 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4adfdb640 1 -- 172.21.15.57:0/2913383549 learned_addr learned my addr 172.21.15.57:0/2913383549 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac25e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 -- 172.21.15.57:0/2913383549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 msgr2=0x56175ac25e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac25e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 -- 172.21.15.57:0/2913383549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 msgr2=0x56175ac35c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac35c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4ad7da640 1 -- 172.21.15.57:0/2913383549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56175ac21030 con 0x56175ac208e0 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.468+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac35c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.469+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac29600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x56175ac3d060 tx=0x7fd498005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c21fb39c5620dbf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.469+0000 7fd4937fe640 1 -- 172.21.15.57:0/2913383549 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498029040 con 0x56175ac208e0 2024-04-21T08:46:02.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.469+0000 7fd4937fe640 1 -- 172.21.15.57:0/2913383549 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd49800fb90 con 0x56175ac208e0 2024-04-21T08:46:02.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.469+0000 7fd4937fe640 1 -- 172.21.15.57:0/2913383549 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498033040 con 0x56175ac208e0 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/2913383549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 msgr2=0x56175ac29600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac29600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x56175ac3d060 tx=0x7fd498005850 comp rx=0 tx=0).stop 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/2913383549 shutdown_connections 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac29e10 0x56175ac25e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac21290 0x56175ac35c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/2913383549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac208e0 0x56175ac29600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/2913383549 >> 172.21.15.57:0/2913383549 conn(0x56175ac1afb0 msgr2=0x56175ac27720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/2913383549 shutdown_connections 2024-04-21T08:46:02.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/2913383549 wait complete. 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 -- start start 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.470+0000 7fd4b2e2e740 1 Processor -- start 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac32150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac32690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81b0270 con 0x56175ac29e10 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81b16e0 con 0x56175ac21290 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a81b0940 con 0x56175ac208e0 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4b2e2e740 10 client.0 _ll_register_callbacks cb 0x7fd4a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac32690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac32150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac32690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:41470/0 (socket says 172.21.15.57:41470) 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4ad7da640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac32150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:49452/0 (socket says 172.21.15.57:49452) 2024-04-21T08:46:02.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 -- 172.21.15.57:0/4097819343 learned_addr learned my addr 172.21.15.57:0/4097819343 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 -- 172.21.15.57:0/4097819343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 msgr2=0x56175ac32150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac32150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 -- 172.21.15.57:0/4097819343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 msgr2=0x56175ac2f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.471+0000 7fd4adfdb640 1 -- 172.21.15.57:0/4097819343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56175ac2f890 con 0x56175ac21290 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4adfdb640 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac32690 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009850 tx=0x7fd4a40349c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f56960db5d0a1fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4917fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4917fa640 1 -- 172.21.15.57:0/4097819343 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a40037f0 con 0x56175ac21290 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd49800fb90 con 0x56175ac21290 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4917fa640 1 -- 172.21.15.57:0/4097819343 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd4a4034b30 con 0x56175ac21290 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4917fa640 1 -- 172.21.15.57:0/4097819343 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd4a4064050 con 0x56175ac21290 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4b2e2e740 10 client.25412 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56175aca13e0 con 0x56175ac21290 2024-04-21T08:46:02.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.472+0000 7fd4917fa640 1 -- 172.21.15.57:0/4097819343 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd49800fb90 con 0x56175ac21290 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4917fa640 1 -- 172.21.15.57:0/4097819343 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd4a405f030 con 0x56175ac21290 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4b2e2e740 10 client.25412 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4b2e2e740 10 client.25412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4b2e2e740 20 client.25412 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175aca1af0 0x56175aca3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4b2e2e740 4 client.25412 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56175ab8c790 con 0x56175aca1af0 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175aca1af0 0x56175aca3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4ad7da640 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175aca1af0 0x56175aca3ee0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53938305a4b10c6a server_cookie=a0c561c47a1840c1 in_seq=0 out_seq=0 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.473+0000 7fd4917fa640 10 client.25412 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4917fa640 1 -- 172.21.15.57:0/4097819343 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56175ab8c790 con 0x56175aca1af0 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4917fa640 10 client.25412 handle_command_reply: tid=0 2024-04-21T08:46:02.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 client.25412 shutdown 2024-04-21T08:46:02.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 msgr2=0x56175ac32690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac32690 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009850 tx=0x7fd4a40349c0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 shutdown_connections 2024-04-21T08:46:02.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56175aca1af0 0x56175aca3ee0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56175ac29e10 0x56175ac2f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56175ac21290 0x56175ac32690 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 --2- 172.21.15.57:0/4097819343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56175ac208e0 0x56175ac32150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 >> 172.21.15.57:0/4097819343 conn(0x56175ac1afb0 msgr2=0x56175ac19780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4917fa640 0 client.25412 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 shutdown_connections 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 1 -- 172.21.15.57:0/4097819343 wait complete. 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.475+0000 7fd4b2e2e740 20 client.25412 trim_cache size 0 max 16384 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:46:02.479 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "08dc0d58-c4dd-4ffe-b277-9c32e04042c4", 2024-04-21T08:46:02.480 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:46:02.480 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:46:02.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4940a3ec0 msgr2=0x7fd4940a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4940a3ec0 0x7fd4940a6380 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00cde70 tx=0x7fd4a0010db0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 msgr2=0x7fd4a819a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 0x7fd4a819a7b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000dac0 tx=0x7fd4a000df80 comp rx=0 tx=0).stop 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 shutdown_connections 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4940a3ec0 0x7fd4940a6380 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4a810e900 0x7fd4a819a7b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4a8101670 0x7fd4a81a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 --2- 172.21.15.57:0/3902309820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a8100d50 0x7fd4a81a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 >> 172.21.15.57:0/3902309820 conn(0x7fd4a8077f50 msgr2=0x7fd4a8110080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 shutdown_connections 2024-04-21T08:46:02.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.476+0000 7fd4b023f640 1 -- 172.21.15.57:0/3902309820 wait complete. 2024-04-21T08:46:02.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:02.487 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:02.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 -- 172.21.15.57:0/778082599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 msgr2=0x7f196010e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 --2- 172.21.15.57:0/778082599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196010e9a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1958009ca0 tx=0x7f195802c230 comp rx=0 tx=0).stop 2024-04-21T08:46:02.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 -- 172.21.15.57:0/778082599 shutdown_connections 2024-04-21T08:46:02.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 --2- 172.21.15.57:0/778082599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196010e9a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 --2- 172.21.15.57:0/778082599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 0x7f196010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 --2- 172.21.15.57:0/778082599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 0x7f19601035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 -- 172.21.15.57:0/778082599 >> 172.21.15.57:0/778082599 conn(0x7f1960078000 msgr2=0x7f1960078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 -- 172.21.15.57:0/778082599 shutdown_connections 2024-04-21T08:46:02.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.594+0000 7f1965752640 1 -- 172.21.15.57:0/778082599 wait complete. 2024-04-21T08:46:02.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.595+0000 7f1965752640 1 Processor -- start 2024-04-21T08:46:02.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.595+0000 7f1965752640 1 -- start start 2024-04-21T08:46:02.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f1965752640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 0x7f19601986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f1965752640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 0x7f1960198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f1965752640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f1965752640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1960112190 con 0x7f196010c500 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f1965752640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1960112010 con 0x7f19601031c0 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f1965752640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1960112310 con 0x7f1960103ae0 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196019cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59446/0 (socket says 172.21.15.57:59446) 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 -- 172.21.15.57:0/763839792 learned_addr learned my addr 172.21.15.57:0/763839792 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195e7fc640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 0x7f1960198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195effd640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 0x7f19601986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 -- 172.21.15.57:0/763839792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 msgr2=0x7f1960198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 0x7f1960198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 -- 172.21.15.57:0/763839792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 msgr2=0x7f19601986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 0x7f19601986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195f7fe640 1 -- 172.21.15.57:0/763839792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196019d6a0 con 0x7f196010c500 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195e7fc640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 0x7f1960198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.596+0000 7f195effd640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 0x7f19601986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.597+0000 7f195f7fe640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196019cf70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1958005e30 tx=0x7f1958035dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.597+0000 7f1947fff640 1 -- 172.21.15.57:0/763839792 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1958039040 con 0x7f196010c500 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.597+0000 7f1947fff640 1 -- 172.21.15.57:0/763839792 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1958044070 con 0x7f196010c500 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.597+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196019d930 con 0x7f196010c500 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.597+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19601a5220 con 0x7f196010c500 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.597+0000 7f1947fff640 1 -- 172.21.15.57:0/763839792 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580328a0 con 0x7f196010c500 2024-04-21T08:46:02.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.600+0000 7f1947fff640 1 -- 172.21.15.57:0/763839792 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f1958032a40 con 0x7f196010c500 2024-04-21T08:46:02.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.600+0000 7f1947fff640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19480a3f40 0x7f19480a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 Processor -- start 2024-04-21T08:46:02.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f195effd640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19480a3f40 0x7f19480a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 -- start start 2024-04-21T08:46:02.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1947fff640 1 -- 172.21.15.57:0/763839792 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f19580f3000 con 0x7f196010c500 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb31620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb3e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb41280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e67ea961e0 con 0x55e67eb31240 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e67ea33390 con 0x55e67eb31b60 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f1968428740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e67ea921a0 con 0x55e67eb3ee90 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f195e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb3e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f195e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb3e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:41500/0 (socket says 172.21.15.57:41500) 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f195e7fc640 1 -- 172.21.15.57:0/153306770 learned_addr learned my addr 172.21.15.57:0/153306770 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f195effd640 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb31620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.601+0000 7f195f7fe640 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb41280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195effd640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19480a3f40 0x7f19480a6400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1950009fc0 tx=0x7f1950005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195e7fc640 1 -- 172.21.15.57:0/153306770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 msgr2=0x55e67eb41280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195e7fc640 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb41280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195e7fc640 1 -- 172.21.15.57:0/153306770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 msgr2=0x55e67eb31620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195e7fc640 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb31620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195e7fc640 1 -- 172.21.15.57:0/153306770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67e8727d0 con 0x55e67eb31b60 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195f7fe640 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb41280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f195e7fc640 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb3e950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1954011470 tx=0x7f1954011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0cde111d5b1b5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f19457fa640 1 -- 172.21.15.57:0/153306770 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195401e070 con 0x55e67eb31b60 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f19457fa640 1 -- 172.21.15.57:0/153306770 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1954004310 con 0x55e67eb31b60 2024-04-21T08:46:02.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.602+0000 7f19457fa640 1 -- 172.21.15.57:0/153306770 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195400f9a0 con 0x55e67eb31b60 2024-04-21T08:46:02.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 -- 172.21.15.57:0/153306770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 msgr2=0x55e67eb3e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb3e950 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1954011470 tx=0x7f1954011930 comp rx=0 tx=0).stop 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 -- 172.21.15.57:0/153306770 shutdown_connections 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb41280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb3e950 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 --2- 172.21.15.57:0/153306770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb31620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 -- 172.21.15.57:0/153306770 >> 172.21.15.57:0/153306770 conn(0x55e67eb2bcb0 msgr2=0x55e67eb2c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 -- 172.21.15.57:0/153306770 shutdown_connections 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 -- 172.21.15.57:0/153306770 wait complete. 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.603+0000 7f1968428740 1 -- start start 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 Processor -- start 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb38100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb38640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb3c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e67eb32720 con 0x55e67eb31240 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e67ea4f2f0 con 0x55e67eb31b60 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e67eb328a0 con 0x55e67eb3ee90 2024-04-21T08:46:02.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f1968428740 10 client.0 _ll_register_callbacks cb 0x55e67ea5c580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f195effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb38100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f195e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb38640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f195e7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb38640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:41510/0 (socket says 172.21.15.57:41510) 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f195e7fc640 1 -- 172.21.15.57:0/1111667828 learned_addr learned my addr 172.21.15.57:0/1111667828 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.604+0000 7f195f7fe640 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb3c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f195e7fc640 1 -- 172.21.15.57:0/1111667828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 msgr2=0x55e67eb3c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f195e7fc640 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb3c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f195e7fc640 1 -- 172.21.15.57:0/1111667828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 msgr2=0x55e67eb38100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f195e7fc640 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb38100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f195e7fc640 1 -- 172.21.15.57:0/1111667828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e67e7a83f0 con 0x55e67eb31b60 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f195e7fc640 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb38640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1954011440 tx=0x7f1954005630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4794a4f0415fcd65 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1942ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1942ff5640 1 -- 172.21.15.57:0/1111667828 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954027020 con 0x55e67eb31b60 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1954004310 con 0x55e67eb31b60 2024-04-21T08:46:02.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1942ff5640 1 -- 172.21.15.57:0/1111667828 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f195400f760 con 0x55e67eb31b60 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1942ff5640 1 -- 172.21.15.57:0/1111667828 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1954032050 con 0x55e67eb31b60 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1968428740 10 client.25424 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.605+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e67e7680f0 con 0x55e67eb31b60 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1942ff5640 1 -- 172.21.15.57:0/1111667828 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1954004310 con 0x55e67eb31b60 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1942ff5640 1 -- 172.21.15.57:0/1111667828 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1954004a30 con 0x55e67eb31b60 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1968428740 10 client.25424 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1968428740 10 client.25424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1968428740 20 client.25424 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1968428740 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67eb53250 0x55e67eb55640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1968428740 4 client.25424 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.606+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e67eaa3c00 con 0x55e67eb53250 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.607+0000 7f195effd640 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67eb53250 0x55e67eb55640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.607+0000 7f1942ff5640 10 client.25424 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.607+0000 7f195effd640 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67eb53250 0x55e67eb55640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52eb79737738918a server_cookie=8474ff2628648665 in_seq=0 out_seq=0 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1942ff5640 1 -- 172.21.15.57:0/1111667828 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e67eaa3c00 con 0x55e67eb53250 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1942ff5640 10 client.25424 handle_command_reply: tid=0 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 client.25424 shutdown 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 msgr2=0x55e67eb38640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb38640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1954011440 tx=0x7f1954005630 comp rx=0 tx=0).stop 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 shutdown_connections 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67eb53250 0x55e67eb55640 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb3ee90 0x55e67eb3c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb31b60 0x55e67eb38640 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1942ff5640 0 client.25424 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 --2- 172.21.15.57:0/1111667828 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb38100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 >> 172.21.15.57:0/1111667828 conn(0x55e67eb2bcb0 msgr2=0x55e67eb2da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 shutdown_connections 2024-04-21T08:46:02.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 1 -- 172.21.15.57:0/1111667828 wait complete. 2024-04-21T08:46:02.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.608+0000 7f1968428740 20 client.25424 trim_cache size 0 max 16384 2024-04-21T08:46:02.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 Processor -- start 2024-04-21T08:46:02.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 -- start start 2024-04-21T08:46:02.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb37280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb2abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb5ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f19601ab020 con 0x55e67eb31240 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19601ad880 con 0x55e67eb3ee90 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f1968428740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f19601ab670 con 0x55e67eb31b60 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.626+0000 7f195effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb37280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb37280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59466/0 (socket says 172.21.15.57:59466) 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 -- 172.21.15.57:0/1656393380 learned_addr learned my addr 172.21.15.57:0/1656393380 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195e7fc640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb2abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195f7fe640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb5ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 -- 172.21.15.57:0/1656393380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 msgr2=0x55e67eb2abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb2abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 -- 172.21.15.57:0/1656393380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 msgr2=0x55e67eb5ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb5ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 -- 172.21.15.57:0/1656393380 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67eb2dfb0 con 0x55e67eb31240 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195e7fc640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb2abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195f7fe640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb5ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:02.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.627+0000 7f195effd640 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb37280 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1950013010 tx=0x7f1950014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524de61fa747fea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1944ff9640 1 -- 172.21.15.57:0/1656393380 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950031070 con 0x55e67eb31240 2024-04-21T08:46:02.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1944ff9640 1 -- 172.21.15.57:0/1656393380 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195002c810 con 0x55e67eb31240 2024-04-21T08:46:02.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1944ff9640 1 -- 172.21.15.57:0/1656393380 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950031070 con 0x55e67eb31240 2024-04-21T08:46:02.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 -- 172.21.15.57:0/1656393380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 msgr2=0x55e67eb37280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb37280 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1950013010 tx=0x7f1950014000 comp rx=0 tx=0).stop 2024-04-21T08:46:02.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 -- 172.21.15.57:0/1656393380 shutdown_connections 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb5ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb2abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 --2- 172.21.15.57:0/1656393380 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb37280 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 -- 172.21.15.57:0/1656393380 >> 172.21.15.57:0/1656393380 conn(0x55e67eb2bcb0 msgr2=0x55e67eb2a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.628+0000 7f1968428740 1 -- 172.21.15.57:0/1656393380 shutdown_connections 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.629+0000 7f1968428740 1 -- 172.21.15.57:0/1656393380 wait complete. 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.629+0000 7f1968428740 1 -- start start 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.629+0000 7f1968428740 1 Processor -- start 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f1968428740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb4fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f1968428740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb35090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f1968428740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f195effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb4fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f195effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb4fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59470/0 (socket says 172.21.15.57:59470) 2024-04-21T08:46:02.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f195effd640 1 -- 172.21.15.57:0/3871339769 learned_addr learned my addr 172.21.15.57:0/3871339769 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f195e7fc640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb35090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f19601abd40 con 0x55e67eb31240 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.630+0000 7f195f7fe640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19601ad1b0 con 0x55e67eb3ee90 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f19601ac410 con 0x55e67eb31b60 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f1968428740 10 client.0 _ll_register_callbacks cb 0x7f195401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195effd640 1 -- 172.21.15.57:0/3871339769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 msgr2=0x55e67eb35090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195effd640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb35090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195effd640 1 -- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 msgr2=0x55e67eb355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195effd640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195effd640 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e67eb35db0 con 0x55e67eb31240 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195f7fe640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f195effd640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb4fc40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1950004580 tx=0x7f1950003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a2c3d55743e68b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:02.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f1942ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.631+0000 7f1942ff5640 1 -- 172.21.15.57:0/3871339769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950031070 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1942ff5640 1 -- 172.21.15.57:0/3871339769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f195002c4e0 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195002c810 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1942ff5640 1 -- 172.21.15.57:0/3871339769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1950042050 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 10 client.15888 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e67ebb95d0 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1942ff5640 1 -- 172.21.15.57:0/3871339769 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f195002c810 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1942ff5640 1 -- 172.21.15.57:0/3871339769 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f195001a040 con 0x55e67eb31240 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 10 client.15888 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 20 client.15888 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67ebb9c50 0x55e67ebbc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 4 client.15888 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:02.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.632+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e67ea9da70 con 0x55e67ebb9c50 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f195e7fc640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67ebb9c50 0x55e67ebbc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f195e7fc640 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67ebb9c50 0x55e67ebbc070 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=148576c73a5cc2df server_cookie=fd5c7da8110d623b in_seq=0 out_seq=0 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f1942ff5640 10 client.15888 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f1942ff5640 1 -- 172.21.15.57:0/3871339769 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e67ea9da70 con 0x55e67ebb9c50 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f1942ff5640 10 client.15888 handle_command_reply: tid=0 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f1968428740 1 client.15888 shutdown 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 msgr2=0x55e67eb4fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.633+0000 7f1968428740 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb4fc40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1950004580 tx=0x7f1950003f60 comp rx=0 tx=0).stop 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 shutdown_connections 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e67ebb9c50 0x55e67ebbc070 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1942ff5640 0 client.15888 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e67eb3ee90 0x55e67eb355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e67eb31b60 0x55e67eb35090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 --2- 172.21.15.57:0/3871339769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e67eb31240 0x55e67eb4fc40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 >> 172.21.15.57:0/3871339769 conn(0x55e67eb2bcb0 msgr2=0x55e67eb2ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 shutdown_connections 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 1 -- 172.21.15.57:0/3871339769 wait complete. 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1968428740 20 client.15888 trim_cache size 0 max 16384 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stdout: "08dc0d58-c4dd-4ffe-b277-9c32e04042c4": { 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:46:02.638 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "08dc0d58-c4dd-4ffe-b277-9c32e04042c4", 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19480a3f40 msgr2=0x7f19480a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1965752640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19480a3f40 0x7f19480a6400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1950009fc0 tx=0x7f1950005f90 comp rx=0 tx=0).stop 2024-04-21T08:46:02.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 msgr2=0x7f196019cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.634+0000 7f1965752640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196019cf70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1958005e30 tx=0x7f1958035dd0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 shutdown_connections 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19480a3f40 0x7f19480a6400 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f196010c500 0x7f196019cf70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1960103ae0 0x7f1960198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 --2- 172.21.15.57:0/763839792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19601031c0 0x7f19601986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 >> 172.21.15.57:0/763839792 conn(0x7f1960078000 msgr2=0x7f19600fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 shutdown_connections 2024-04-21T08:46:02.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:02.635+0000 7f1965752640 1 -- 172.21.15.57:0/763839792 wait complete. 2024-04-21T08:46:02.645 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:02 smithi057 ceph-mon[27968]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:46:02.645 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:02 smithi057 ceph-mon[27968]: mgrmap e25: x(active, since 36s), standbys: y, z 2024-04-21T08:46:02.647 INFO:tasks.cephfs.filesystem:scrub status for tag:08dc0d58-c4dd-4ffe-b277-9c32e04042c4 - {'path': '/', 'tag': '08dc0d58-c4dd-4ffe-b277-9c32e04042c4', 'options': 'recursive,force'} 2024-04-21T08:46:02.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:02 smithi129 ceph-mon[31272]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:46:02.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:02 smithi129 ceph-mon[31272]: mgrmap e25: x(active, since 36s), standbys: y, z 2024-04-21T08:46:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:02 smithi136 ceph-mon[31791]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-21T08:46:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:02 smithi136 ceph-mon[31791]: mgrmap e25: x(active, since 36s), standbys: y, z 2024-04-21T08:46:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:03 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:46:03.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:03 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:03 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:46:03.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:03 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:03 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:46:03.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:03 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:03 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:46:03.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:03 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:03 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:46:03.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:03 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:03.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:03 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:46:03.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:03 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:04 smithi129 ceph-mon[31272]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-04-21T08:46:04.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:04 smithi057 ceph-mon[27968]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-04-21T08:46:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:04 smithi136 ceph-mon[31791]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-04-21T08:46:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:06 smithi129 ceph-mon[31272]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-21T08:46:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:06 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:46:06.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:06 smithi057 ceph-mon[27968]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-21T08:46:06.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:06 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:46:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:06 smithi136 ceph-mon[31791]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-04-21T08:46:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:06 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:46:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:08 smithi129 ceph-mon[31272]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:08.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:08 smithi057 ceph-mon[27968]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:08 smithi136 ceph-mon[31791]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:10 smithi129 ceph-mon[31272]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-21T08:46:10.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:10 smithi057 ceph-mon[27968]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-21T08:46:10.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:10 smithi136 ceph-mon[31791]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 -- 172.21.15.57:0/1015482269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c1021b0 msgr2=0x7fab2c102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 --2- 172.21.15.57:0/1015482269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c1021b0 0x7fab2c102590 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fab1c009900 tx=0x7fab1c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 -- 172.21.15.57:0/1015482269 shutdown_connections 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 --2- 172.21.15.57:0/1015482269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c107280 0x7fab2c10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 --2- 172.21.15.57:0/1015482269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab2c102b60 0x7fab2c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 --2- 172.21.15.57:0/1015482269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c1021b0 0x7fab2c102590 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 -- 172.21.15.57:0/1015482269 >> 172.21.15.57:0/1015482269 conn(0x7fab2c0fda10 msgr2=0x7fab2c0ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 -- 172.21.15.57:0/1015482269 shutdown_connections 2024-04-21T08:46:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.354+0000 7fab30e5c640 1 -- 172.21.15.57:0/1015482269 wait complete. 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 Processor -- start 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 -- start start 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c1021b0 0x7fab2c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab2c107280 0x7fab2c198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 0x7fab2c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c112240 con 0x7fab2c19cfb0 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1120c0 con 0x7fab2c107280 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab30e5c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1123c0 con 0x7fab2c1021b0 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab2ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 0x7fab2c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab2ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 0x7fab2c19d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55934/0 (socket says 172.21.15.57:55934) 2024-04-21T08:46:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.355+0000 7fab2ad9d640 1 -- 172.21.15.57:0/490097042 learned_addr learned my addr 172.21.15.57:0/490097042 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2a59c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c1021b0 0x7fab2c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab29d9b640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab2c107280 0x7fab2c198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2ad9d640 1 -- 172.21.15.57:0/490097042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c1021b0 msgr2=0x7fab2c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2ad9d640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c1021b0 0x7fab2c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2ad9d640 1 -- 172.21.15.57:0/490097042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab2c107280 msgr2=0x7fab2c198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2ad9d640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab2c107280 0x7fab2c198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2ad9d640 1 -- 172.21.15.57:0/490097042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c19daa0 con 0x7fab2c19cfb0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab2ad9d640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 0x7fab2c19d460 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fab2400eb10 tx=0x7fab2400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab1b7fe640 1 -- 172.21.15.57:0/490097042 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24004b20 con 0x7fab2c19cfb0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab1b7fe640 1 -- 172.21.15.57:0/490097042 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab24004cc0 con 0x7fab2c19cfb0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab1b7fe640 1 -- 172.21.15.57:0/490097042 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24010670 con 0x7fab2c19cfb0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c19dd90 con 0x7fab2c19cfb0 2024-04-21T08:46:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.356+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c1a5530 con 0x7fab2c19cfb0 2024-04-21T08:46:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.359+0000 7fab1b7fe640 1 -- 172.21.15.57:0/490097042 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fab24005020 con 0x7fab2c19cfb0 2024-04-21T08:46:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 Processor -- start 2024-04-21T08:46:11.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 -- start start 2024-04-21T08:46:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 0x5633f3d5b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d5bb60 0x5633f3d5bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d650e0 0x5633f3d674d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5633f3cc08d0 con 0x5633f3d650e0 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5633f3c5de80 con 0x5633f3d5bb60 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab33a50740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5633f3cbc890 con 0x5633f3d5c8d0 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab1b7fe640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fab140a3ee0 0x7fab140a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab2ad9d640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fab140a3ee0 0x7fab140a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab29d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 0x5633f3d5b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab29d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 0x5633f3d5b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42028/0 (socket says 172.21.15.57:42028) 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d5bb60 0x5633f3d5bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.360+0000 7fab29d9b640 1 -- 172.21.15.57:0/2102028487 learned_addr learned my addr 172.21.15.57:0/2102028487 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d650e0 0x5633f3d674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 -- 172.21.15.57:0/2102028487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d5bb60 msgr2=0x5633f3d5bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d5bb60 0x5633f3d5bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 -- 172.21.15.57:0/2102028487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d650e0 msgr2=0x5633f3d674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d650e0 0x5633f3d674d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 -- 172.21.15.57:0/2102028487 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633f3c7f320 con 0x5633f3d5c8d0 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab1b7fe640 1 -- 172.21.15.57:0/490097042 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fab240d1350 con 0x7fab2c19cfb0 2024-04-21T08:46:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab2ad9d640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fab140a3ee0 0x7fab140a63a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab240ce5c0 tx=0x7fab24098850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 0x5633f3d5b620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab200058c0 tx=0x7fab20005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ba815620196505 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.361+0000 7fab18ff9640 1 -- 172.21.15.57:0/2102028487 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20004230 con 0x5633f3d5c8d0 2024-04-21T08:46:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab18ff9640 1 -- 172.21.15.57:0/2102028487 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab200043d0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab18ff9640 1 -- 172.21.15.57:0/2102028487 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20014dc0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 -- 172.21.15.57:0/2102028487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 msgr2=0x5633f3d5b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 0x5633f3d5b620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fab200058c0 tx=0x7fab20005850 comp rx=0 tx=0).stop 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 -- 172.21.15.57:0/2102028487 shutdown_connections 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d650e0 0x5633f3d674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d5bb60 0x5633f3d5bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 --2- 172.21.15.57:0/2102028487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5c8d0 0x5633f3d5b620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 -- 172.21.15.57:0/2102028487 >> 172.21.15.57:0/2102028487 conn(0x5633f3d22c90 msgr2=0x5633f3d56850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.362+0000 7fab33a50740 1 -- 172.21.15.57:0/2102028487 shutdown_connections 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 -- 172.21.15.57:0/2102028487 wait complete. 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 -- start start 2024-04-21T08:46:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 Processor -- start 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d64a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d61df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d62370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5633f3d67e10 con 0x5633f3d5c8d0 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5633f3c48eb0 con 0x5633f3d650e0 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5633f3d67f90 con 0x5633f3d5bb60 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab33a50740 10 client.0 _ll_register_callbacks cb 0x5633f3c75b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d61df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d61df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55954/0 (socket says 172.21.15.57:55954) 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.363+0000 7fab2a59c640 1 -- 172.21.15.57:0/1345870943 learned_addr learned my addr 172.21.15.57:0/1345870943 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab29d9b640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d62370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab2a59c640 1 -- 172.21.15.57:0/1345870943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 msgr2=0x5633f3d64a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab2a59c640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d64a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab2a59c640 1 -- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 msgr2=0x5633f3d62370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab2a59c640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d62370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab2a59c640 1 -- 172.21.15.57:0/1345870943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633f39d3210 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab29d9b640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d62370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab2a59c640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d61df0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fab1c006010 tx=0x7fab1c02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7db7c607654556 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab11ffb640 1 -- 172.21.15.57:0/1345870943 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c02c5d0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab11ffb640 1 -- 172.21.15.57:0/1345870943 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fab1c005030 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.364+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab200043d0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab11ffb640 1 -- 172.21.15.57:0/1345870943 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fab1c052050 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 10 client.15900 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5633f3a9f620 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab11ffb640 1 -- 172.21.15.57:0/1345870943 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab1c0046c0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab11ffb640 1 -- 172.21.15.57:0/1345870943 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fab1c0035f0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 10 client.15900 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 10 client.15900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 20 client.15900 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3d75120 0x5633f3d77590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 4 client.15900 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fab200043d0 con 0x5633f3d75120 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.365+0000 7fab29d9b640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3d75120 0x5633f3d77590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.366+0000 7fab29d9b640 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3d75120 0x5633f3d77590 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41ac3bcc1e63f6ee server_cookie=3eaa5f1dcbec9c40 in_seq=0 out_seq=0 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.366+0000 7fab11ffb640 10 client.15900 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab11ffb640 1 -- 172.21.15.57:0/1345870943 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fab200043d0 con 0x5633f3d75120 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab11ffb640 10 client.15900 handle_command_reply: tid=0 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 client.15900 shutdown 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 msgr2=0x5633f3d61df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d61df0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fab1c006010 tx=0x7fab1c02c9f0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 shutdown_connections 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3d75120 0x5633f3d77590 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d62370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d61df0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 --2- 172.21.15.57:0/1345870943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d64a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 >> 172.21.15.57:0/1345870943 conn(0x5633f3d22c90 msgr2=0x5633f3d585a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab11ffb640 0 client.15900 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 shutdown_connections 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 1 -- 172.21.15.57:0/1345870943 wait complete. 2024-04-21T08:46:11.371 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.367+0000 7fab33a50740 20 client.15900 trim_cache size 0 max 16384 2024-04-21T08:46:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 Processor -- start 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 -- start start 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d70d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d77130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d64520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1aaf50 con 0x5633f3d5c8d0 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1ad7b0 con 0x5633f3d650e0 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d77130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d77130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55960/0 (socket says 172.21.15.57:55960) 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab2a59c640 1 -- 172.21.15.57:0/4214211130 learned_addr learned my addr 172.21.15.57:0/4214211130 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab33a50740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1ab5a0 con 0x5633f3d5bb60 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.382+0000 7fab29d9b640 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d70d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab29d9b640 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d64520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab2a59c640 1 -- 172.21.15.57:0/4214211130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 msgr2=0x5633f3d70d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab2a59c640 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d70d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab2a59c640 1 -- 172.21.15.57:0/4214211130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 msgr2=0x5633f3d64520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab2a59c640 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d64520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab2a59c640 1 -- 172.21.15.57:0/4214211130 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633f3d77900 con 0x5633f3d5c8d0 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab29d9b640 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d70d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab2a59c640 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d77130 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02ce10 tx=0x7fab1c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f67d70bd7281c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab13fff640 1 -- 172.21.15.57:0/4214211130 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c005850 con 0x5633f3d5c8d0 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab13fff640 1 -- 172.21.15.57:0/4214211130 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab1c02c5d0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.383+0000 7fab13fff640 1 -- 172.21.15.57:0/4214211130 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c0046c0 con 0x5633f3d5c8d0 2024-04-21T08:46:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 -- 172.21.15.57:0/4214211130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 msgr2=0x5633f3d77130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d77130 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fab1c02ce10 tx=0x7fab1c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 -- 172.21.15.57:0/4214211130 shutdown_connections 2024-04-21T08:46:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d64520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d77130 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 --2- 172.21.15.57:0/4214211130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d70d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 -- 172.21.15.57:0/4214211130 >> 172.21.15.57:0/4214211130 conn(0x5633f3d22c90 msgr2=0x5633f3d606c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 -- 172.21.15.57:0/4214211130 shutdown_connections 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 -- 172.21.15.57:0/4214211130 wait complete. 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 -- start start 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.384+0000 7fab33a50740 1 Processor -- start 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d6d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d688b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d688b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab2a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d688b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55972/0 (socket says 172.21.15.57:55972) 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d68df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab2a59c640 1 -- 172.21.15.57:0/2989958246 learned_addr learned my addr 172.21.15.57:0/2989958246 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1abc70 con 0x5633f3d5c8d0 2024-04-21T08:46:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1ad0e0 con 0x5633f3d650e0 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d6d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c1ac340 con 0x5633f3d5bb60 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d68df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab33a50740 10 client.0 _ll_register_callbacks cb 0x7fab1c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 -- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 msgr2=0x5633f3d68df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d68df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 -- 172.21.15.57:0/2989958246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 msgr2=0x5633f3d688b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d688b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab29d9b640 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633f3d695f0 con 0x5633f3d5bb60 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.385+0000 7fab2a59c640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d688b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab29d9b640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d6d540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5633f3d650b0 tx=0x7fab2002d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e218b0037d386591 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:46:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab11ffb640 1 -- 172.21.15.57:0/2989958246 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20032dc0 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab1c02c5d0 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab11ffb640 1 -- 172.21.15.57:0/2989958246 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fab20005040 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab11ffb640 1 -- 172.21.15.57:0/2989958246 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fab20005260 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab33a50740 10 client.25420 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5633f3ddb650 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.386+0000 7fab11ffb640 1 -- 172.21.15.57:0/2989958246 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab1c02c5d0 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab11ffb640 1 -- 172.21.15.57:0/2989958246 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fab20032230 con 0x5633f3d5bb60 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab33a50740 10 client.25420 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab33a50740 10 client.25420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab33a50740 20 client.25420 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab33a50740 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3ddbbe0 0x5633f3dddfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab33a50740 4 client.25420 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5633f3cd4880 con 0x5633f3ddbbe0 2024-04-21T08:46:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab2a59c640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3ddbbe0 0x5633f3dddfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab2a59c640 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3ddbbe0 0x5633f3dddfd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b807341461be5562 server_cookie=64d628cebccb26de in_seq=0 out_seq=0 2024-04-21T08:46:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.387+0000 7fab11ffb640 10 client.25420 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab11ffb640 1 -- 172.21.15.57:0/2989958246 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5633f3cd4880 con 0x5633f3ddbbe0 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab11ffb640 10 client.25420 handle_command_reply: tid=0 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 client.25420 shutdown 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 msgr2=0x5633f3d6d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d6d540 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x5633f3d650b0 tx=0x7fab2002d040 comp rx=0 tx=0).stop 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 shutdown_connections 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5633f3ddbbe0 0x5633f3dddfd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5633f3d650e0 0x5633f3d68df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab11ffb640 0 client.25420 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5633f3d5c8d0 0x5633f3d688b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 --2- 172.21.15.57:0/2989958246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5633f3d5bb60 0x5633f3d6d540 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 >> 172.21.15.57:0/2989958246 conn(0x5633f3d22c90 msgr2=0x5633f3d55040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 shutdown_connections 2024-04-21T08:46:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 1 -- 172.21.15.57:0/2989958246 wait complete. 2024-04-21T08:46:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.396+0000 7fab33a50740 20 client.25420 trim_cache size 0 max 16384 2024-04-21T08:46:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:46:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:46:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:46:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fab140a3ee0 msgr2=0x7fab140a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fab140a3ee0 0x7fab140a63a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fab240ce5c0 tx=0x7fab24098850 comp rx=0 tx=0).stop 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 msgr2=0x7fab2c19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 0x7fab2c19d460 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fab2400eb10 tx=0x7fab2400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 shutdown_connections 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fab140a3ee0 0x7fab140a63a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab2c19cfb0 0x7fab2c19d460 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab2c107280 0x7fab2c198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 --2- 172.21.15.57:0/490097042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab2c1021b0 0x7fab2c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 >> 172.21.15.57:0/490097042 conn(0x7fab2c0fda10 msgr2=0x7fab2c0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 shutdown_connections 2024-04-21T08:46:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:11.397+0000 7fab30e5c640 1 -- 172.21.15.57:0/490097042 wait complete. 2024-04-21T08:46:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:12 smithi129 ceph-mon[31272]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:12.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:12 smithi057 ceph-mon[27968]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:12 smithi136 ceph-mon[31791]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:14 smithi129 ceph-mon[31272]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:14.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:14 smithi057 ceph-mon[27968]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:14 smithi136 ceph-mon[31791]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 19 op/s 2024-04-21T08:46:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:16 smithi129 ceph-mon[31272]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-21T08:46:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:16 smithi057 ceph-mon[27968]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-21T08:46:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:16 smithi136 ceph-mon[31791]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-04-21T08:46:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:17 smithi129 ceph-mon[31272]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-21T08:46:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:17 smithi057 ceph-mon[27968]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-21T08:46:17.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:17 smithi136 ceph-mon[31791]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2024-04-21T08:46:19.972 INFO:tasks.workunit.client.0.smithi057.stderr:Note: switching to '4f6bc5273643aaf5660909e85400647a8951c10b'. 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr:state without impacting any branches by switching back to a branch. 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr: git switch -c 2024-04-21T08:46:19.973 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr:Or undo this operation with: 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr: git switch - 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:19.974 INFO:tasks.workunit.client.0.smithi057.stderr:HEAD is now at 4f6bc527364 use ubuntu for fs/workload tests 2024-04-21T08:46:19.977 DEBUG:teuthology.orchestra.run.smithi057:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-21T08:46:20.006 INFO:tasks.workunit.client.0.smithi057.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-21T08:46:20.008 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:46:20.008 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-21T08:46:20.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:19 smithi129 ceph-mon[31272]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-21T08:46:20.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:19 smithi057 ceph-mon[27968]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-21T08:46:20.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:19 smithi136 ceph-mon[31791]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-21T08:46:20.635 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-21T08:46:20.690 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-21T08:46:20.732 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-21T08:46:20.733 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:46:20.733 INFO:tasks.workunit.client.0.smithi057.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-21T08:46:20.773 INFO:tasks.workunit.client.0.smithi057.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-21T08:46:20.776 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:46:20.776 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-21T08:46:20.832 INFO:tasks.workunit:Running workunits matching kernel_untar_build.sh on client.0... 2024-04-21T08:46:20.834 INFO:tasks.workunit:Running workunit kernel_untar_build.sh... 2024-04-21T08:46:20.834 DEBUG:teuthology.orchestra.run.smithi057:workunit test kernel_untar_build.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh 2024-04-21T08:46:20.894 INFO:tasks.workunit.client.0.smithi057.stderr:+ wget -O linux.tar.xz http://download.ceph.com/qa/linux-6.5.11.tar.xz 2024-04-21T08:46:20.914 INFO:tasks.workunit.client.0.smithi057.stderr:--2024-04-21 08:46:20-- http://download.ceph.com/qa/linux-6.5.11.tar.xz 2024-04-21T08:46:20.915 INFO:tasks.workunit.client.0.smithi057.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-21T08:46:20.936 INFO:tasks.workunit.client.0.smithi057.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-21T08:46:20.957 INFO:tasks.workunit.client.0.smithi057.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-21T08:46:20.957 INFO:tasks.workunit.client.0.smithi057.stderr:Length: 139010916 (133M) [application/octet-stream] 2024-04-21T08:46:20.957 INFO:tasks.workunit.client.0.smithi057.stderr:Saving to: ‘linux.tar.xz’ 2024-04-21T08:46:20.960 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:20.998 INFO:tasks.workunit.client.0.smithi057.stderr: 0K .......... .......... .......... .......... .......... 0% 1.19M 1m51s 2024-04-21T08:46:21.019 INFO:tasks.workunit.client.0.smithi057.stderr: 50K .......... .......... .......... .......... .......... 0% 2.40M 83s 2024-04-21T08:46:21.020 INFO:tasks.workunit.client.0.smithi057.stderr: 100K .......... .......... .......... .......... .......... 0% 208M 56s 2024-04-21T08:46:21.020 INFO:tasks.workunit.client.0.smithi057.stderr: 150K .......... .......... .......... .......... .......... 0% 301M 42s 2024-04-21T08:46:21.041 INFO:tasks.workunit.client.0.smithi057.stderr: 200K .......... .......... .......... .......... .......... 0% 2.41M 44s 2024-04-21T08:46:21.041 INFO:tasks.workunit.client.0.smithi057.stderr: 250K .......... .......... .......... .......... .......... 0% 300M 37s 2024-04-21T08:46:21.041 INFO:tasks.workunit.client.0.smithi057.stderr: 300K .......... .......... .......... .......... .......... 0% 315M 32s 2024-04-21T08:46:21.041 INFO:tasks.workunit.client.0.smithi057.stderr: 350K .......... .......... .......... .......... .......... 0% 358M 28s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 400K .......... .......... .......... .......... .......... 0% 2.41M 31s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 450K .......... .......... .......... .......... .......... 0% 284M 28s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 500K .......... .......... .......... .......... .......... 0% 338M 25s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 550K .......... .......... .......... .......... .......... 0% 361M 23s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 600K .......... .......... .......... .......... .......... 0% 334M 21s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 650K .......... .......... .......... .......... .......... 0% 320M 20s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 700K .......... .......... .......... .......... .......... 0% 49.3M 19s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 750K .......... .......... .......... .......... .......... 0% 24.9M 18s 2024-04-21T08:46:21.065 INFO:tasks.workunit.client.0.smithi057.stderr: 800K .......... .......... .......... .......... .......... 0% 126M 17s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 850K .......... .......... .......... .......... .......... 0% 3.03M 18s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 900K .......... .......... .......... .......... .......... 0% 194M 17s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 950K .......... .......... .......... .......... .......... 0% 302M 17s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 1000K .......... .......... .......... .......... .......... 0% 353M 16s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 1050K .......... .......... .......... .......... .......... 0% 357M 15s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 1100K .......... .......... .......... .......... .......... 0% 307M 14s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 1150K .......... .......... .......... .......... .......... 0% 25.6M 14s 2024-04-21T08:46:21.086 INFO:tasks.workunit.client.0.smithi057.stderr: 1200K .......... .......... .......... .......... .......... 0% 23.9M 14s 2024-04-21T08:46:21.087 INFO:tasks.workunit.client.0.smithi057.stderr: 1250K .......... .......... .......... .......... .......... 0% 25.6M 13s 2024-04-21T08:46:21.088 INFO:tasks.workunit.client.0.smithi057.stderr: 1300K .......... .......... .......... .......... .......... 0% 308M 13s 2024-04-21T08:46:21.089 INFO:tasks.workunit.client.0.smithi057.stderr: 1350K .......... .......... .......... .......... .......... 1% 23.2M 13s 2024-04-21T08:46:21.091 INFO:tasks.workunit.client.0.smithi057.stderr: 1400K .......... .......... .......... .......... .......... 1% 21.6M 12s 2024-04-21T08:46:21.093 INFO:tasks.workunit.client.0.smithi057.stderr: 1450K .......... .......... .......... .......... .......... 1% 34.8M 12s 2024-04-21T08:46:21.095 INFO:tasks.workunit.client.0.smithi057.stderr: 1500K .......... .......... .......... .......... .......... 1% 19.0M 12s 2024-04-21T08:46:21.095 INFO:tasks.workunit.client.0.smithi057.stderr: 1550K .......... .......... .......... .......... .......... 1% 135M 12s 2024-04-21T08:46:21.097 INFO:tasks.workunit.client.0.smithi057.stderr: 1600K .......... .......... .......... .......... .......... 1% 24.2M 11s 2024-04-21T08:46:21.099 INFO:tasks.workunit.client.0.smithi057.stderr: 1650K .......... .......... .......... .......... .......... 1% 39.8M 11s 2024-04-21T08:46:21.102 INFO:tasks.workunit.client.0.smithi057.stderr: 1700K .......... .......... .......... .......... .......... 1% 15.9M 11s 2024-04-21T08:46:21.104 INFO:tasks.workunit.client.0.smithi057.stderr: 1750K .......... .......... .......... .......... .......... 1% 23.2M 11s 2024-04-21T08:46:21.104 INFO:tasks.workunit.client.0.smithi057.stderr: 1800K .......... .......... .......... .......... .......... 1% 136M 11s 2024-04-21T08:46:21.106 INFO:tasks.workunit.client.0.smithi057.stderr: 1850K .......... .......... .......... .......... .......... 1% 25.0M 11s 2024-04-21T08:46:21.108 INFO:tasks.workunit.client.0.smithi057.stderr: 1900K .......... .......... .......... .......... .......... 1% 21.7M 10s 2024-04-21T08:46:21.111 INFO:tasks.workunit.client.0.smithi057.stderr: 1950K .......... .......... .......... .......... .......... 1% 22.5M 10s 2024-04-21T08:46:21.113 INFO:tasks.workunit.client.0.smithi057.stderr: 2000K .......... .......... .......... .......... .......... 1% 23.4M 10s 2024-04-21T08:46:21.113 INFO:tasks.workunit.client.0.smithi057.stderr: 2050K .......... .......... .......... .......... .......... 1% 268M 10s 2024-04-21T08:46:21.115 INFO:tasks.workunit.client.0.smithi057.stderr: 2100K .......... .......... .......... .......... .......... 1% 21.5M 10s 2024-04-21T08:46:21.118 INFO:tasks.workunit.client.0.smithi057.stderr: 2150K .......... .......... .......... .......... .......... 1% 26.8M 10s 2024-04-21T08:46:21.119 INFO:tasks.workunit.client.0.smithi057.stderr: 2200K .......... .......... .......... .......... .......... 1% 22.6M 10s 2024-04-21T08:46:21.121 INFO:tasks.workunit.client.0.smithi057.stderr: 2250K .......... .......... .......... .......... .......... 1% 24.3M 10s 2024-04-21T08:46:21.121 INFO:tasks.workunit.client.0.smithi057.stderr: 2300K .......... .......... .......... .......... .......... 1% 157M 9s 2024-04-21T08:46:21.123 INFO:tasks.workunit.client.0.smithi057.stderr: 2350K .......... .......... .......... .......... .......... 1% 23.2M 9s 2024-04-21T08:46:21.125 INFO:tasks.workunit.client.0.smithi057.stderr: 2400K .......... .......... .......... .......... .......... 1% 24.9M 9s 2024-04-21T08:46:21.127 INFO:tasks.workunit.client.0.smithi057.stderr: 2450K .......... .......... .......... .......... .......... 1% 26.3M 9s 2024-04-21T08:46:21.131 INFO:tasks.workunit.client.0.smithi057.stderr: 2500K .......... .......... .......... .......... .......... 1% 20.9M 9s 2024-04-21T08:46:21.131 INFO:tasks.workunit.client.0.smithi057.stderr: 2550K .......... .......... .......... .......... .......... 1% 120M 9s 2024-04-21T08:46:21.132 INFO:tasks.workunit.client.0.smithi057.stderr: 2600K .......... .......... .......... .......... .......... 1% 30.4M 9s 2024-04-21T08:46:21.134 INFO:tasks.workunit.client.0.smithi057.stderr: 2650K .......... .......... .......... .......... .......... 1% 21.0M 9s 2024-04-21T08:46:21.136 INFO:tasks.workunit.client.0.smithi057.stderr: 2700K .......... .......... .......... .......... .......... 2% 26.0M 9s 2024-04-21T08:46:21.138 INFO:tasks.workunit.client.0.smithi057.stderr: 2750K .......... .......... .......... .......... .......... 2% 23.5M 9s 2024-04-21T08:46:21.138 INFO:tasks.workunit.client.0.smithi057.stderr: 2800K .......... .......... .......... .......... .......... 2% 296M 8s 2024-04-21T08:46:21.140 INFO:tasks.workunit.client.0.smithi057.stderr: 2850K .......... .......... .......... .......... .......... 2% 29.6M 8s 2024-04-21T08:46:21.142 INFO:tasks.workunit.client.0.smithi057.stderr: 2900K .......... .......... .......... .......... .......... 2% 22.4M 8s 2024-04-21T08:46:21.145 INFO:tasks.workunit.client.0.smithi057.stderr: 2950K .......... .......... .......... .......... .......... 2% 18.1M 8s 2024-04-21T08:46:21.146 INFO:tasks.workunit.client.0.smithi057.stderr: 3000K .......... .......... .......... .......... .......... 2% 34.0M 8s 2024-04-21T08:46:21.147 INFO:tasks.workunit.client.0.smithi057.stderr: 3050K .......... .......... .......... .......... .......... 2% 59.6M 8s 2024-04-21T08:46:21.149 INFO:tasks.workunit.client.0.smithi057.stderr: 3100K .......... .......... .......... .......... .......... 2% 24.6M 8s 2024-04-21T08:46:21.151 INFO:tasks.workunit.client.0.smithi057.stderr: 3150K .......... .......... .......... .......... .......... 2% 27.7M 8s 2024-04-21T08:46:21.153 INFO:tasks.workunit.client.0.smithi057.stderr: 3200K .......... .......... .......... .......... .......... 2% 18.5M 8s 2024-04-21T08:46:21.155 INFO:tasks.workunit.client.0.smithi057.stderr: 3250K .......... .......... .......... .......... .......... 2% 25.3M 8s 2024-04-21T08:46:21.156 INFO:tasks.workunit.client.0.smithi057.stderr: 3300K .......... .......... .......... .......... .......... 2% 171M 8s 2024-04-21T08:46:21.157 INFO:tasks.workunit.client.0.smithi057.stderr: 3350K .......... .......... .......... .......... .......... 2% 24.2M 8s 2024-04-21T08:46:21.158 INFO:tasks.workunit.client.0.smithi057.stderr: 3400K .......... .......... .......... .......... .......... 2% 51.8M 8s 2024-04-21T08:46:21.161 INFO:tasks.workunit.client.0.smithi057.stderr: 3450K .......... .......... .......... .......... .......... 2% 18.8M 8s 2024-04-21T08:46:21.164 INFO:tasks.workunit.client.0.smithi057.stderr: 3500K .......... .......... .......... .......... .......... 2% 22.2M 8s 2024-04-21T08:46:21.164 INFO:tasks.workunit.client.0.smithi057.stderr: 3550K .......... .......... .......... .......... .......... 2% 229M 8s 2024-04-21T08:46:21.165 INFO:tasks.workunit.client.0.smithi057.stderr: 3600K .......... .......... .......... .......... .......... 2% 22.9M 8s 2024-04-21T08:46:21.168 INFO:tasks.workunit.client.0.smithi057.stderr: 3650K .......... .......... .......... .......... .......... 2% 25.3M 8s 2024-04-21T08:46:21.176 INFO:tasks.workunit.client.0.smithi057.stderr: 3700K .......... .......... .......... .......... .......... 2% 10.3M 8s 2024-04-21T08:46:21.176 INFO:tasks.workunit.client.0.smithi057.stderr: 3750K .......... .......... .......... .......... .......... 2% 123M 7s 2024-04-21T08:46:21.176 INFO:tasks.workunit.client.0.smithi057.stderr: 3800K .......... .......... .......... .......... .......... 2% 288M 7s 2024-04-21T08:46:21.176 INFO:tasks.workunit.client.0.smithi057.stderr: 3850K .......... .......... .......... .......... .......... 2% 44.5M 7s 2024-04-21T08:46:21.176 INFO:tasks.workunit.client.0.smithi057.stderr: 3900K .......... .......... .......... .......... .......... 2% 27.5M 7s 2024-04-21T08:46:21.178 INFO:tasks.workunit.client.0.smithi057.stderr: 3950K .......... .......... .......... .......... .......... 2% 27.6M 7s 2024-04-21T08:46:21.180 INFO:tasks.workunit.client.0.smithi057.stderr: 4000K .......... .......... .......... .......... .......... 2% 22.1M 7s 2024-04-21T08:46:21.180 INFO:tasks.workunit.client.0.smithi057.stderr: 4050K .......... .......... .......... .......... .......... 3% 68.3M 7s 2024-04-21T08:46:21.182 INFO:tasks.workunit.client.0.smithi057.stderr: 4100K .......... .......... .......... .......... .......... 3% 24.4M 7s 2024-04-21T08:46:21.184 INFO:tasks.workunit.client.0.smithi057.stderr: 4150K .......... .......... .......... .......... .......... 3% 23.3M 7s 2024-04-21T08:46:21.186 INFO:tasks.workunit.client.0.smithi057.stderr: 4200K .......... .......... .......... .......... .......... 3% 36.7M 7s 2024-04-21T08:46:21.188 INFO:tasks.workunit.client.0.smithi057.stderr: 4250K .......... .......... .......... .......... .......... 3% 25.8M 7s 2024-04-21T08:46:21.188 INFO:tasks.workunit.client.0.smithi057.stderr: 4300K .......... .......... .......... .......... .......... 3% 47.2M 7s 2024-04-21T08:46:21.190 INFO:tasks.workunit.client.0.smithi057.stderr: 4350K .......... .......... .......... .......... .......... 3% 24.2M 7s 2024-04-21T08:46:21.192 INFO:tasks.workunit.client.0.smithi057.stderr: 4400K .......... .......... .......... .......... .......... 3% 33.3M 7s 2024-04-21T08:46:21.194 INFO:tasks.workunit.client.0.smithi057.stderr: 4450K .......... .......... .......... .......... .......... 3% 26.6M 7s 2024-04-21T08:46:21.197 INFO:tasks.workunit.client.0.smithi057.stderr: 4500K .......... .......... .......... .......... .......... 3% 21.9M 7s 2024-04-21T08:46:21.197 INFO:tasks.workunit.client.0.smithi057.stderr: 4550K .......... .......... .......... .......... .......... 3% 64.6M 7s 2024-04-21T08:46:21.199 INFO:tasks.workunit.client.0.smithi057.stderr: 4600K .......... .......... .......... .......... .......... 3% 22.5M 7s 2024-04-21T08:46:21.202 INFO:tasks.workunit.client.0.smithi057.stderr: 4650K .......... .......... .......... .......... .......... 3% 38.1M 7s 2024-04-21T08:46:21.204 INFO:tasks.workunit.client.0.smithi057.stderr: 4700K .......... .......... .......... .......... .......... 3% 14.6M 7s 2024-04-21T08:46:21.206 INFO:tasks.workunit.client.0.smithi057.stderr: 4750K .......... .......... .......... .......... .......... 3% 22.2M 7s 2024-04-21T08:46:21.207 INFO:tasks.workunit.client.0.smithi057.stderr: 4800K .......... .......... .......... .......... .......... 3% 62.6M 7s 2024-04-21T08:46:21.209 INFO:tasks.workunit.client.0.smithi057.stderr: 4850K .......... .......... .......... .......... .......... 3% 22.2M 7s 2024-04-21T08:46:21.210 INFO:tasks.workunit.client.0.smithi057.stderr: 4900K .......... .......... .......... .......... .......... 3% 41.4M 7s 2024-04-21T08:46:21.212 INFO:tasks.workunit.client.0.smithi057.stderr: 4950K .......... .......... .......... .......... .......... 3% 21.9M 7s 2024-04-21T08:46:21.214 INFO:tasks.workunit.client.0.smithi057.stderr: 5000K .......... .......... .......... .......... .......... 3% 24.3M 7s 2024-04-21T08:46:21.215 INFO:tasks.workunit.client.0.smithi057.stderr: 5050K .......... .......... .......... .......... .......... 3% 57.2M 7s 2024-04-21T08:46:21.217 INFO:tasks.workunit.client.0.smithi057.stderr: 5100K .......... .......... .......... .......... .......... 3% 23.5M 7s 2024-04-21T08:46:21.219 INFO:tasks.workunit.client.0.smithi057.stderr: 5150K .......... .......... .......... .......... .......... 3% 32.7M 7s 2024-04-21T08:46:21.221 INFO:tasks.workunit.client.0.smithi057.stderr: 5200K .......... .......... .......... .......... .......... 3% 25.4M 7s 2024-04-21T08:46:21.223 INFO:tasks.workunit.client.0.smithi057.stderr: 5250K .......... .......... .......... .......... .......... 3% 22.5M 7s 2024-04-21T08:46:21.224 INFO:tasks.workunit.client.0.smithi057.stderr: 5300K .......... .......... .......... .......... .......... 3% 53.5M 7s 2024-04-21T08:46:21.226 INFO:tasks.workunit.client.0.smithi057.stderr: 5350K .......... .......... .......... .......... .......... 3% 24.2M 6s 2024-04-21T08:46:21.228 INFO:tasks.workunit.client.0.smithi057.stderr: 5400K .......... .......... .......... .......... .......... 4% 24.4M 6s 2024-04-21T08:46:21.229 INFO:tasks.workunit.client.0.smithi057.stderr: 5450K .......... .......... .......... .......... .......... 4% 35.2M 6s 2024-04-21T08:46:21.230 INFO:tasks.workunit.client.0.smithi057.stderr: 5500K .......... .......... .......... .......... .......... 4% 35.7M 6s 2024-04-21T08:46:21.233 INFO:tasks.workunit.client.0.smithi057.stderr: 5550K .......... .......... .......... .......... .......... 4% 33.7M 6s 2024-04-21T08:46:21.234 INFO:tasks.workunit.client.0.smithi057.stderr: 5600K .......... .......... .......... .......... .......... 4% 22.8M 6s 2024-04-21T08:46:21.236 INFO:tasks.workunit.client.0.smithi057.stderr: 5650K .......... .......... .......... .......... .......... 4% 25.0M 6s 2024-04-21T08:46:21.237 INFO:tasks.workunit.client.0.smithi057.stderr: 5700K .......... .......... .......... .......... .......... 4% 33.3M 6s 2024-04-21T08:46:21.239 INFO:tasks.workunit.client.0.smithi057.stderr: 5750K .......... .......... .......... .......... .......... 4% 25.4M 6s 2024-04-21T08:46:21.241 INFO:tasks.workunit.client.0.smithi057.stderr: 5800K .......... .......... .......... .......... .......... 4% 24.3M 6s 2024-04-21T08:46:21.242 INFO:tasks.workunit.client.0.smithi057.stderr: 5850K .......... .......... .......... .......... .......... 4% 45.3M 6s 2024-04-21T08:46:21.244 INFO:tasks.workunit.client.0.smithi057.stderr: 5900K .......... .......... .......... .......... .......... 4% 25.1M 6s 2024-04-21T08:46:21.246 INFO:tasks.workunit.client.0.smithi057.stderr: 5950K .......... .......... .......... .......... .......... 4% 32.5M 6s 2024-04-21T08:46:21.248 INFO:tasks.workunit.client.0.smithi057.stderr: 6000K .......... .......... .......... .......... .......... 4% 27.4M 6s 2024-04-21T08:46:21.250 INFO:tasks.workunit.client.0.smithi057.stderr: 6050K .......... .......... .......... .......... .......... 4% 22.6M 6s 2024-04-21T08:46:21.251 INFO:tasks.workunit.client.0.smithi057.stderr: 6100K .......... .......... .......... .......... .......... 4% 56.6M 6s 2024-04-21T08:46:21.253 INFO:tasks.workunit.client.0.smithi057.stderr: 6150K .......... .......... .......... .......... .......... 4% 23.4M 6s 2024-04-21T08:46:21.254 INFO:tasks.workunit.client.0.smithi057.stderr: 6200K .......... .......... .......... .......... .......... 4% 37.3M 6s 2024-04-21T08:46:21.256 INFO:tasks.workunit.client.0.smithi057.stderr: 6250K .......... .......... .......... .......... .......... 4% 23.1M 6s 2024-04-21T08:46:21.258 INFO:tasks.workunit.client.0.smithi057.stderr: 6300K .......... .......... .......... .......... .......... 4% 23.2M 6s 2024-04-21T08:46:21.260 INFO:tasks.workunit.client.0.smithi057.stderr: 6350K .......... .......... .......... .......... .......... 4% 32.4M 6s 2024-04-21T08:46:21.262 INFO:tasks.workunit.client.0.smithi057.stderr: 6400K .......... .......... .......... .......... .......... 4% 24.1M 6s 2024-04-21T08:46:21.263 INFO:tasks.workunit.client.0.smithi057.stderr: 6450K .......... .......... .......... .......... .......... 4% 36.3M 6s 2024-04-21T08:46:21.265 INFO:tasks.workunit.client.0.smithi057.stderr: 6500K .......... .......... .......... .......... .......... 4% 23.5M 6s 2024-04-21T08:46:21.267 INFO:tasks.workunit.client.0.smithi057.stderr: 6550K .......... .......... .......... .......... .......... 4% 23.9M 6s 2024-04-21T08:46:21.268 INFO:tasks.workunit.client.0.smithi057.stderr: 6600K .......... .......... .......... .......... .......... 4% 43.6M 6s 2024-04-21T08:46:21.271 INFO:tasks.workunit.client.0.smithi057.stderr: 6650K .......... .......... .......... .......... .......... 4% 19.9M 6s 2024-04-21T08:46:21.272 INFO:tasks.workunit.client.0.smithi057.stderr: 6700K .......... .......... .......... .......... .......... 4% 42.8M 6s 2024-04-21T08:46:21.274 INFO:tasks.workunit.client.0.smithi057.stderr: 6750K .......... .......... .......... .......... .......... 5% 22.4M 6s 2024-04-21T08:46:21.275 INFO:tasks.workunit.client.0.smithi057.stderr: 6800K .......... .......... .......... .......... .......... 5% 40.0M 6s 2024-04-21T08:46:21.277 INFO:tasks.workunit.client.0.smithi057.stderr: 6850K .......... .......... .......... .......... .......... 5% 27.8M 6s 2024-04-21T08:46:21.279 INFO:tasks.workunit.client.0.smithi057.stderr: 6900K .......... .......... .......... .......... .......... 5% 21.6M 6s 2024-04-21T08:46:21.281 INFO:tasks.workunit.client.0.smithi057.stderr: 6950K .......... .......... .......... .......... .......... 5% 35.3M 6s 2024-04-21T08:46:21.283 INFO:tasks.workunit.client.0.smithi057.stderr: 7000K .......... .......... .......... .......... .......... 5% 25.0M 6s 2024-04-21T08:46:21.285 INFO:tasks.workunit.client.0.smithi057.stderr: 7050K .......... .......... .......... .......... .......... 5% 23.2M 6s 2024-04-21T08:46:21.286 INFO:tasks.workunit.client.0.smithi057.stderr: 7100K .......... .......... .......... .......... .......... 5% 58.0M 6s 2024-04-21T08:46:21.288 INFO:tasks.workunit.client.0.smithi057.stderr: 7150K .......... .......... .......... .......... .......... 5% 24.2M 6s 2024-04-21T08:46:21.289 INFO:tasks.workunit.client.0.smithi057.stderr: 7200K .......... .......... .......... .......... .......... 5% 34.4M 6s 2024-04-21T08:46:21.291 INFO:tasks.workunit.client.0.smithi057.stderr: 7250K .......... .......... .......... .......... .......... 5% 24.3M 6s 2024-04-21T08:46:21.293 INFO:tasks.workunit.client.0.smithi057.stderr: 7300K .......... .......... .......... .......... .......... 5% 23.9M 6s 2024-04-21T08:46:21.294 INFO:tasks.workunit.client.0.smithi057.stderr: 7350K .......... .......... .......... .......... .......... 5% 59.7M 6s 2024-04-21T08:46:21.296 INFO:tasks.workunit.client.0.smithi057.stderr: 7400K .......... .......... .......... .......... .......... 5% 23.8M 6s 2024-04-21T08:46:21.297 INFO:tasks.workunit.client.0.smithi057.stderr: 7450K .......... .......... .......... .......... .......... 5% 35.1M 6s 2024-04-21T08:46:21.300 INFO:tasks.workunit.client.0.smithi057.stderr: 7500K .......... .......... .......... .......... .......... 5% 23.6M 6s 2024-04-21T08:46:21.301 INFO:tasks.workunit.client.0.smithi057.stderr: 7550K .......... .......... .......... .......... .......... 5% 25.6M 6s 2024-04-21T08:46:21.302 INFO:tasks.workunit.client.0.smithi057.stderr: 7600K .......... .......... .......... .......... .......... 5% 45.5M 6s 2024-04-21T08:46:21.304 INFO:tasks.workunit.client.0.smithi057.stderr: 7650K .......... .......... .......... .......... .......... 5% 24.6M 6s 2024-04-21T08:46:21.306 INFO:tasks.workunit.client.0.smithi057.stderr: 7700K .......... .......... .......... .......... .......... 5% 33.5M 6s 2024-04-21T08:46:21.308 INFO:tasks.workunit.client.0.smithi057.stderr: 7750K .......... .......... .......... .......... .......... 5% 23.8M 6s 2024-04-21T08:46:21.310 INFO:tasks.workunit.client.0.smithi057.stderr: 7800K .......... .......... .......... .......... .......... 5% 24.6M 6s 2024-04-21T08:46:21.311 INFO:tasks.workunit.client.0.smithi057.stderr: 7850K .......... .......... .......... .......... .......... 5% 55.9M 6s 2024-04-21T08:46:21.313 INFO:tasks.workunit.client.0.smithi057.stderr: 7900K .......... .......... .......... .......... .......... 5% 22.6M 6s 2024-04-21T08:46:21.314 INFO:tasks.workunit.client.0.smithi057.stderr: 7950K .......... .......... .......... .......... .......... 5% 39.2M 6s 2024-04-21T08:46:21.317 INFO:tasks.workunit.client.0.smithi057.stderr: 8000K .......... .......... .......... .......... .......... 5% 20.7M 6s 2024-04-21T08:46:21.318 INFO:tasks.workunit.client.0.smithi057.stderr: 8050K .......... .......... .......... .......... .......... 5% 46.2M 6s 2024-04-21T08:46:21.320 INFO:tasks.workunit.client.0.smithi057.stderr: 8100K .......... .......... .......... .......... .......... 6% 25.2M 6s 2024-04-21T08:46:21.321 INFO:tasks.workunit.client.0.smithi057.stderr: 8150K .......... .......... .......... .......... .......... 6% 32.5M 6s 2024-04-21T08:46:21.323 INFO:tasks.workunit.client.0.smithi057.stderr: 8200K .......... .......... .......... .......... .......... 6% 27.8M 6s 2024-04-21T08:46:21.325 INFO:tasks.workunit.client.0.smithi057.stderr: 8250K .......... .......... .......... .......... .......... 6% 23.2M 6s 2024-04-21T08:46:21.327 INFO:tasks.workunit.client.0.smithi057.stderr: 8300K .......... .......... .......... .......... .......... 6% 23.6M 6s 2024-04-21T08:46:21.328 INFO:tasks.workunit.client.0.smithi057.stderr: 8350K .......... .......... .......... .......... .......... 6% 53.8M 6s 2024-04-21T08:46:21.330 INFO:tasks.workunit.client.0.smithi057.stderr: 8400K .......... .......... .......... .......... .......... 6% 22.6M 6s 2024-04-21T08:46:21.332 INFO:tasks.workunit.client.0.smithi057.stderr: 8450K .......... .......... .......... .......... .......... 6% 23.9M 6s 2024-04-21T08:46:21.333 INFO:tasks.workunit.client.0.smithi057.stderr: 8500K .......... .......... .......... .......... .......... 6% 42.7M 6s 2024-04-21T08:46:21.335 INFO:tasks.workunit.client.0.smithi057.stderr: 8550K .......... .......... .......... .......... .......... 6% 23.4M 6s 2024-04-21T08:46:21.336 INFO:tasks.workunit.client.0.smithi057.stderr: 8600K .......... .......... .......... .......... .......... 6% 46.7M 6s 2024-04-21T08:46:21.339 INFO:tasks.workunit.client.0.smithi057.stderr: 8650K .......... .......... .......... .......... .......... 6% 23.4M 6s 2024-04-21T08:46:21.339 INFO:tasks.workunit.client.0.smithi057.stderr: 8700K .......... .......... .......... .......... .......... 6% 44.6M 6s 2024-04-21T08:46:21.342 INFO:tasks.workunit.client.0.smithi057.stderr: 8750K .......... .......... .......... .......... .......... 6% 22.2M 6s 2024-04-21T08:46:21.344 INFO:tasks.workunit.client.0.smithi057.stderr: 8800K .......... .......... .......... .......... .......... 6% 23.7M 6s 2024-04-21T08:46:21.345 INFO:tasks.workunit.client.0.smithi057.stderr: 8850K .......... .......... .......... .......... .......... 6% 54.2M 6s 2024-04-21T08:46:21.347 INFO:tasks.workunit.client.0.smithi057.stderr: 8900K .......... .......... .......... .......... .......... 6% 23.8M 6s 2024-04-21T08:46:21.348 INFO:tasks.workunit.client.0.smithi057.stderr: 8950K .......... .......... .......... .......... .......... 6% 37.2M 6s 2024-04-21T08:46:21.350 INFO:tasks.workunit.client.0.smithi057.stderr: 9000K .......... .......... .......... .......... .......... 6% 24.7M 6s 2024-04-21T08:46:21.352 INFO:tasks.workunit.client.0.smithi057.stderr: 9050K .......... .......... .......... .......... .......... 6% 24.1M 6s 2024-04-21T08:46:21.353 INFO:tasks.workunit.client.0.smithi057.stderr: 9100K .......... .......... .......... .......... .......... 6% 54.3M 5s 2024-04-21T08:46:21.355 INFO:tasks.workunit.client.0.smithi057.stderr: 9150K .......... .......... .......... .......... .......... 6% 22.3M 5s 2024-04-21T08:46:21.357 INFO:tasks.workunit.client.0.smithi057.stderr: 9200K .......... .......... .......... .......... .......... 6% 34.3M 5s 2024-04-21T08:46:21.359 INFO:tasks.workunit.client.0.smithi057.stderr: 9250K .......... .......... .......... .......... .......... 6% 24.5M 5s 2024-04-21T08:46:21.361 INFO:tasks.workunit.client.0.smithi057.stderr: 9300K .......... .......... .......... .......... .......... 6% 23.8M 5s 2024-04-21T08:46:21.361 INFO:tasks.workunit.client.0.smithi057.stderr: 9350K .......... .......... .......... .......... .......... 6% 57.4M 5s 2024-04-21T08:46:21.364 INFO:tasks.workunit.client.0.smithi057.stderr: 9400K .......... .......... .......... .......... .......... 6% 28.0M 5s 2024-04-21T08:46:21.365 INFO:tasks.workunit.client.0.smithi057.stderr: 9450K .......... .......... .......... .......... .......... 6% 26.2M 5s 2024-04-21T08:46:21.367 INFO:tasks.workunit.client.0.smithi057.stderr: 9500K .......... .......... .......... .......... .......... 7% 25.8M 5s 2024-04-21T08:46:21.370 INFO:tasks.workunit.client.0.smithi057.stderr: 9550K .......... .......... .......... .......... .......... 7% 23.8M 5s 2024-04-21T08:46:21.370 INFO:tasks.workunit.client.0.smithi057.stderr: 9600K .......... .......... .......... .......... .......... 7% 64.5M 5s 2024-04-21T08:46:21.372 INFO:tasks.workunit.client.0.smithi057.stderr: 9650K .......... .......... .......... .......... .......... 7% 23.5M 5s 2024-04-21T08:46:21.374 INFO:tasks.workunit.client.0.smithi057.stderr: 9700K .......... .......... .......... .......... .......... 7% 40.6M 5s 2024-04-21T08:46:21.375 INFO:tasks.workunit.client.0.smithi057.stderr: 9750K .......... .......... .......... .......... .......... 7% 23.6M 5s 2024-04-21T08:46:21.377 INFO:tasks.workunit.client.0.smithi057.stderr: 9800K .......... .......... .......... .......... .......... 7% 22.7M 5s 2024-04-21T08:46:21.378 INFO:tasks.workunit.client.0.smithi057.stderr: 9850K .......... .......... .......... .......... .......... 7% 56.7M 5s 2024-04-21T08:46:21.380 INFO:tasks.workunit.client.0.smithi057.stderr: 9900K .......... .......... .......... .......... .......... 7% 22.3M 5s 2024-04-21T08:46:21.383 INFO:tasks.workunit.client.0.smithi057.stderr: 9950K .......... .......... .......... .......... .......... 7% 39.2M 5s 2024-04-21T08:46:21.385 INFO:tasks.workunit.client.0.smithi057.stderr: 10000K .......... .......... .......... .......... .......... 7% 15.9M 5s 2024-04-21T08:46:21.387 INFO:tasks.workunit.client.0.smithi057.stderr: 10050K .......... .......... .......... .......... .......... 7% 22.3M 5s 2024-04-21T08:46:21.388 INFO:tasks.workunit.client.0.smithi057.stderr: 10100K .......... .......... .......... .......... .......... 7% 51.9M 5s 2024-04-21T08:46:21.389 INFO:tasks.workunit.client.0.smithi057.stderr: 10150K .......... .......... .......... .......... .......... 7% 29.9M 5s 2024-04-21T08:46:21.392 INFO:tasks.workunit.client.0.smithi057.stderr: 10200K .......... .......... .......... .......... .......... 7% 27.9M 5s 2024-04-21T08:46:21.394 INFO:tasks.workunit.client.0.smithi057.stderr: 10250K .......... .......... .......... .......... .......... 7% 24.4M 5s 2024-04-21T08:46:21.395 INFO:tasks.workunit.client.0.smithi057.stderr: 10300K .......... .......... .......... .......... .......... 7% 23.6M 5s 2024-04-21T08:46:21.396 INFO:tasks.workunit.client.0.smithi057.stderr: 10350K .......... .......... .......... .......... .......... 7% 56.0M 5s 2024-04-21T08:46:21.398 INFO:tasks.workunit.client.0.smithi057.stderr: 10400K .......... .......... .......... .......... .......... 7% 25.4M 5s 2024-04-21T08:46:21.400 INFO:tasks.workunit.client.0.smithi057.stderr: 10450K .......... .......... .......... .......... .......... 7% 33.2M 5s 2024-04-21T08:46:21.402 INFO:tasks.workunit.client.0.smithi057.stderr: 10500K .......... .......... .......... .......... .......... 7% 21.9M 5s 2024-04-21T08:46:21.404 INFO:tasks.workunit.client.0.smithi057.stderr: 10550K .......... .......... .......... .......... .......... 7% 24.6M 5s 2024-04-21T08:46:21.405 INFO:tasks.workunit.client.0.smithi057.stderr: 10600K .......... .......... .......... .......... .......... 7% 60.5M 5s 2024-04-21T08:46:21.407 INFO:tasks.workunit.client.0.smithi057.stderr: 10650K .......... .......... .......... .......... .......... 7% 23.6M 5s 2024-04-21T08:46:21.409 INFO:tasks.workunit.client.0.smithi057.stderr: 10700K .......... .......... .......... .......... .......... 7% 23.7M 5s 2024-04-21T08:46:21.410 INFO:tasks.workunit.client.0.smithi057.stderr: 10750K .......... .......... .......... .......... .......... 7% 35.2M 5s 2024-04-21T08:46:21.412 INFO:tasks.workunit.client.0.smithi057.stderr: 10800K .......... .......... .......... .......... .......... 7% 24.8M 5s 2024-04-21T08:46:21.414 INFO:tasks.workunit.client.0.smithi057.stderr: 10850K .......... .......... .......... .......... .......... 8% 46.8M 5s 2024-04-21T08:46:21.415 INFO:tasks.workunit.client.0.smithi057.stderr: 10900K .......... .......... .......... .......... .......... 8% 24.4M 5s 2024-04-21T08:46:21.417 INFO:tasks.workunit.client.0.smithi057.stderr: 10950K .......... .......... .......... .......... .......... 8% 24.9M 5s 2024-04-21T08:46:21.418 INFO:tasks.workunit.client.0.smithi057.stderr: 11000K .......... .......... .......... .......... .......... 8% 38.1M 5s 2024-04-21T08:46:21.420 INFO:tasks.workunit.client.0.smithi057.stderr: 11050K .......... .......... .......... .......... .......... 8% 23.0M 5s 2024-04-21T08:46:21.423 INFO:tasks.workunit.client.0.smithi057.stderr: 11100K .......... .......... .......... .......... .......... 8% 24.8M 5s 2024-04-21T08:46:21.423 INFO:tasks.workunit.client.0.smithi057.stderr: 11150K .......... .......... .......... .......... .......... 8% 47.4M 5s 2024-04-21T08:46:21.425 INFO:tasks.workunit.client.0.smithi057.stderr: 11200K .......... .......... .......... .......... .......... 8% 24.8M 5s 2024-04-21T08:46:21.427 INFO:tasks.workunit.client.0.smithi057.stderr: 11250K .......... .......... .......... .......... .......... 8% 32.8M 5s 2024-04-21T08:46:21.429 INFO:tasks.workunit.client.0.smithi057.stderr: 11300K .......... .......... .......... .......... .......... 8% 22.2M 5s 2024-04-21T08:46:21.432 INFO:tasks.workunit.client.0.smithi057.stderr: 11350K .......... .......... .......... .......... .......... 8% 21.2M 5s 2024-04-21T08:46:21.433 INFO:tasks.workunit.client.0.smithi057.stderr: 11400K .......... .......... .......... .......... .......... 8% 53.4M 5s 2024-04-21T08:46:21.434 INFO:tasks.workunit.client.0.smithi057.stderr: 11450K .......... .......... .......... .......... .......... 8% 38.2M 5s 2024-04-21T08:46:21.436 INFO:tasks.workunit.client.0.smithi057.stderr: 11500K .......... .......... .......... .......... .......... 8% 23.7M 5s 2024-04-21T08:46:21.438 INFO:tasks.workunit.client.0.smithi057.stderr: 11550K .......... .......... .......... .......... .......... 8% 25.3M 5s 2024-04-21T08:46:21.440 INFO:tasks.workunit.client.0.smithi057.stderr: 11600K .......... .......... .......... .......... .......... 8% 22.1M 5s 2024-04-21T08:46:21.441 INFO:tasks.workunit.client.0.smithi057.stderr: 11650K .......... .......... .......... .......... .......... 8% 50.6M 5s 2024-04-21T08:46:21.443 INFO:tasks.workunit.client.0.smithi057.stderr: 11700K .......... .......... .......... .......... .......... 8% 25.4M 5s 2024-04-21T08:46:21.444 INFO:tasks.workunit.client.0.smithi057.stderr: 11750K .......... .......... .......... .......... .......... 8% 37.7M 5s 2024-04-21T08:46:21.446 INFO:tasks.workunit.client.0.smithi057.stderr: 11800K .......... .......... .......... .......... .......... 8% 24.0M 5s 2024-04-21T08:46:21.448 INFO:tasks.workunit.client.0.smithi057.stderr: 11850K .......... .......... .......... .......... .......... 8% 23.8M 5s 2024-04-21T08:46:21.449 INFO:tasks.workunit.client.0.smithi057.stderr: 11900K .......... .......... .......... .......... .......... 8% 49.8M 5s 2024-04-21T08:46:21.451 INFO:tasks.workunit.client.0.smithi057.stderr: 11950K .......... .......... .......... .......... .......... 8% 24.4M 5s 2024-04-21T08:46:21.452 INFO:tasks.workunit.client.0.smithi057.stderr: 12000K .......... .......... .......... .......... .......... 8% 37.5M 5s 2024-04-21T08:46:21.454 INFO:tasks.workunit.client.0.smithi057.stderr: 12050K .......... .......... .......... .......... .......... 8% 24.6M 5s 2024-04-21T08:46:21.457 INFO:tasks.workunit.client.0.smithi057.stderr: 12100K .......... .......... .......... .......... .......... 8% 22.4M 5s 2024-04-21T08:46:21.457 INFO:tasks.workunit.client.0.smithi057.stderr: 12150K .......... .......... .......... .......... .......... 8% 57.4M 5s 2024-04-21T08:46:21.460 INFO:tasks.workunit.client.0.smithi057.stderr: 12200K .......... .......... .......... .......... .......... 9% 23.5M 5s 2024-04-21T08:46:21.461 INFO:tasks.workunit.client.0.smithi057.stderr: 12250K .......... .......... .......... .......... .......... 9% 36.1M 5s 2024-04-21T08:46:21.463 INFO:tasks.workunit.client.0.smithi057.stderr: 12300K .......... .......... .......... .......... .......... 9% 24.9M 5s 2024-04-21T08:46:21.465 INFO:tasks.workunit.client.0.smithi057.stderr: 12350K .......... .......... .......... .......... .......... 9% 23.0M 5s 2024-04-21T08:46:21.466 INFO:tasks.workunit.client.0.smithi057.stderr: 12400K .......... .......... .......... .......... .......... 9% 48.4M 5s 2024-04-21T08:46:21.468 INFO:tasks.workunit.client.0.smithi057.stderr: 12450K .......... .......... .......... .......... .......... 9% 24.8M 5s 2024-04-21T08:46:21.469 INFO:tasks.workunit.client.0.smithi057.stderr: 12500K .......... .......... .......... .......... .......... 9% 42.1M 5s 2024-04-21T08:46:21.471 INFO:tasks.workunit.client.0.smithi057.stderr: 12550K .......... .......... .......... .......... .......... 9% 22.8M 5s 2024-04-21T08:46:21.473 INFO:tasks.workunit.client.0.smithi057.stderr: 12600K .......... .......... .......... .......... .......... 9% 23.9M 5s 2024-04-21T08:46:21.474 INFO:tasks.workunit.client.0.smithi057.stderr: 12650K .......... .......... .......... .......... .......... 9% 49.6M 5s 2024-04-21T08:46:21.476 INFO:tasks.workunit.client.0.smithi057.stderr: 12700K .......... .......... .......... .......... .......... 9% 24.6M 5s 2024-04-21T08:46:21.478 INFO:tasks.workunit.client.0.smithi057.stderr: 12750K .......... .......... .......... .......... .......... 9% 36.9M 5s 2024-04-21T08:46:21.480 INFO:tasks.workunit.client.0.smithi057.stderr: 12800K .......... .......... .......... .......... .......... 9% 23.3M 5s 2024-04-21T08:46:21.482 INFO:tasks.workunit.client.0.smithi057.stderr: 12850K .......... .......... .......... .......... .......... 9% 24.6M 5s 2024-04-21T08:46:21.483 INFO:tasks.workunit.client.0.smithi057.stderr: 12900K .......... .......... .......... .......... .......... 9% 48.8M 5s 2024-04-21T08:46:21.485 INFO:tasks.workunit.client.0.smithi057.stderr: 12950K .......... .......... .......... .......... .......... 9% 24.4M 5s 2024-04-21T08:46:21.486 INFO:tasks.workunit.client.0.smithi057.stderr: 13000K .......... .......... .......... .......... .......... 9% 40.7M 5s 2024-04-21T08:46:21.488 INFO:tasks.workunit.client.0.smithi057.stderr: 13050K .......... .......... .......... .......... .......... 9% 23.7M 5s 2024-04-21T08:46:21.491 INFO:tasks.workunit.client.0.smithi057.stderr: 13100K .......... .......... .......... .......... .......... 9% 16.6M 5s 2024-04-21T08:46:21.492 INFO:tasks.workunit.client.0.smithi057.stderr: 13150K .......... .......... .......... .......... .......... 9% 31.3M 5s 2024-04-21T08:46:21.494 INFO:tasks.workunit.client.0.smithi057.stderr: 13200K .......... .......... .......... .......... .......... 9% 23.4M 5s 2024-04-21T08:46:21.496 INFO:tasks.workunit.client.0.smithi057.stderr: 13250K .......... .......... .......... .......... .......... 9% 35.2M 5s 2024-04-21T08:46:21.498 INFO:tasks.workunit.client.0.smithi057.stderr: 13300K .......... .......... .......... .......... .......... 9% 23.4M 5s 2024-04-21T08:46:21.500 INFO:tasks.workunit.client.0.smithi057.stderr: 13350K .......... .......... .......... .......... .......... 9% 23.8M 5s 2024-04-21T08:46:21.501 INFO:tasks.workunit.client.0.smithi057.stderr: 13400K .......... .......... .......... .......... .......... 9% 52.7M 5s 2024-04-21T08:46:21.503 INFO:tasks.workunit.client.0.smithi057.stderr: 13450K .......... .......... .......... .......... .......... 9% 23.8M 5s 2024-04-21T08:46:21.504 INFO:tasks.workunit.client.0.smithi057.stderr: 13500K .......... .......... .......... .......... .......... 9% 37.5M 5s 2024-04-21T08:46:21.506 INFO:tasks.workunit.client.0.smithi057.stderr: 13550K .......... .......... .......... .......... .......... 10% 23.4M 5s 2024-04-21T08:46:21.508 INFO:tasks.workunit.client.0.smithi057.stderr: 13600K .......... .......... .......... .......... .......... 10% 25.8M 5s 2024-04-21T08:46:21.509 INFO:tasks.workunit.client.0.smithi057.stderr: 13650K .......... .......... .......... .......... .......... 10% 51.5M 5s 2024-04-21T08:46:21.511 INFO:tasks.workunit.client.0.smithi057.stderr: 13700K .......... .......... .......... .......... .......... 10% 22.3M 5s 2024-04-21T08:46:21.513 INFO:tasks.workunit.client.0.smithi057.stderr: 13750K .......... .......... .......... .......... .......... 10% 37.5M 5s 2024-04-21T08:46:21.515 INFO:tasks.workunit.client.0.smithi057.stderr: 13800K .......... .......... .......... .......... .......... 10% 22.8M 5s 2024-04-21T08:46:21.517 INFO:tasks.workunit.client.0.smithi057.stderr: 13850K .......... .......... .......... .......... .......... 10% 24.7M 5s 2024-04-21T08:46:21.518 INFO:tasks.workunit.client.0.smithi057.stderr: 13900K .......... .......... .......... .......... .......... 10% 55.8M 5s 2024-04-21T08:46:21.520 INFO:tasks.workunit.client.0.smithi057.stderr: 13950K .......... .......... .......... .......... .......... 10% 23.6M 5s 2024-04-21T08:46:21.521 INFO:tasks.workunit.client.0.smithi057.stderr: 14000K .......... .......... .......... .......... .......... 10% 38.1M 5s 2024-04-21T08:46:21.523 INFO:tasks.workunit.client.0.smithi057.stderr: 14050K .......... .......... .......... .......... .......... 10% 24.5M 5s 2024-04-21T08:46:21.525 INFO:tasks.workunit.client.0.smithi057.stderr: 14100K .......... .......... .......... .......... .......... 10% 23.0M 5s 2024-04-21T08:46:21.526 INFO:tasks.workunit.client.0.smithi057.stderr: 14150K .......... .......... .......... .......... .......... 10% 49.4M 5s 2024-04-21T08:46:21.528 INFO:tasks.workunit.client.0.smithi057.stderr: 14200K .......... .......... .......... .......... .......... 10% 23.4M 5s 2024-04-21T08:46:21.529 INFO:tasks.workunit.client.0.smithi057.stderr: 14250K .......... .......... .......... .......... .......... 10% 42.2M 5s 2024-04-21T08:46:21.532 INFO:tasks.workunit.client.0.smithi057.stderr: 14300K .......... .......... .......... .......... .......... 10% 22.3M 5s 2024-04-21T08:46:21.534 INFO:tasks.workunit.client.0.smithi057.stderr: 14350K .......... .......... .......... .......... .......... 10% 25.5M 5s 2024-04-21T08:46:21.534 INFO:tasks.workunit.client.0.smithi057.stderr: 14400K .......... .......... .......... .......... .......... 10% 47.7M 5s 2024-04-21T08:46:21.536 INFO:tasks.workunit.client.0.smithi057.stderr: 14450K .......... .......... .......... .......... .......... 10% 23.8M 5s 2024-04-21T08:46:21.538 INFO:tasks.workunit.client.0.smithi057.stderr: 14500K .......... .......... .......... .......... .......... 10% 36.2M 5s 2024-04-21T08:46:21.540 INFO:tasks.workunit.client.0.smithi057.stderr: 14550K .......... .......... .......... .......... .......... 10% 23.0M 5s 2024-04-21T08:46:21.542 INFO:tasks.workunit.client.0.smithi057.stderr: 14600K .......... .......... .......... .......... .......... 10% 28.7M 5s 2024-04-21T08:46:21.543 INFO:tasks.workunit.client.0.smithi057.stderr: 14650K .......... .......... .......... .......... .......... 10% 43.9M 5s 2024-04-21T08:46:21.545 INFO:tasks.workunit.client.0.smithi057.stderr: 14700K .......... .......... .......... .......... .......... 10% 28.0M 5s 2024-04-21T08:46:21.546 INFO:tasks.workunit.client.0.smithi057.stderr: 14750K .......... .......... .......... .......... .......... 10% 29.1M 5s 2024-04-21T08:46:21.548 INFO:tasks.workunit.client.0.smithi057.stderr: 14800K .......... .......... .......... .......... .......... 10% 24.8M 5s 2024-04-21T08:46:21.550 INFO:tasks.workunit.client.0.smithi057.stderr: 14850K .......... .......... .......... .......... .......... 10% 23.2M 5s 2024-04-21T08:46:21.551 INFO:tasks.workunit.client.0.smithi057.stderr: 14900K .......... .......... .......... .......... .......... 11% 50.2M 5s 2024-04-21T08:46:21.554 INFO:tasks.workunit.client.0.smithi057.stderr: 14950K .......... .......... .......... .......... .......... 11% 19.1M 5s 2024-04-21T08:46:21.558 INFO:tasks.workunit.client.0.smithi057.stderr: 15000K .......... .......... .......... .......... .......... 11% 32.9M 5s 2024-04-21T08:46:21.558 INFO:tasks.workunit.client.0.smithi057.stderr: 15050K .......... .......... .......... .......... .......... 11% 24.0M 5s 2024-04-21T08:46:21.560 INFO:tasks.workunit.client.0.smithi057.stderr: 15100K .......... .......... .......... .......... .......... 11% 24.1M 5s 2024-04-21T08:46:21.561 INFO:tasks.workunit.client.0.smithi057.stderr: 15150K .......... .......... .......... .......... .......... 11% 57.9M 5s 2024-04-21T08:46:21.564 INFO:tasks.workunit.client.0.smithi057.stderr: 15200K .......... .......... .......... .......... .......... 11% 20.0M 5s 2024-04-21T08:46:21.565 INFO:tasks.workunit.client.0.smithi057.stderr: 15250K .......... .......... .......... .......... .......... 11% 35.9M 5s 2024-04-21T08:46:21.567 INFO:tasks.workunit.client.0.smithi057.stderr: 15300K .......... .......... .......... .......... .......... 11% 22.6M 5s 2024-04-21T08:46:21.569 INFO:tasks.workunit.client.0.smithi057.stderr: 15350K .......... .......... .......... .......... .......... 11% 25.0M 5s 2024-04-21T08:46:21.571 INFO:tasks.workunit.client.0.smithi057.stderr: 15400K .......... .......... .......... .......... .......... 11% 50.6M 5s 2024-04-21T08:46:21.571 INFO:tasks.workunit.client.0.smithi057.stderr: 15450K .......... .......... .......... .......... .......... 11% 29.4M 5s 2024-04-21T08:46:21.573 INFO:tasks.workunit.client.0.smithi057.stderr: 15500K .......... .......... .......... .......... .......... 11% 23.3M 5s 2024-04-21T08:46:21.575 INFO:tasks.workunit.client.0.smithi057.stderr: 15550K .......... .......... .......... .......... .......... 11% 27.5M 5s 2024-04-21T08:46:21.577 INFO:tasks.workunit.client.0.smithi057.stderr: 15600K .......... .......... .......... .......... .......... 11% 24.0M 5s 2024-04-21T08:46:21.577 INFO:tasks.workunit.client.0.smithi057.stderr: 15650K .......... .......... .......... .......... .......... 11% 56.6M 5s 2024-04-21T08:46:21.580 INFO:tasks.workunit.client.0.smithi057.stderr: 15700K .......... .......... .......... .......... .......... 11% 23.3M 5s 2024-04-21T08:46:21.582 INFO:tasks.workunit.client.0.smithi057.stderr: 15750K .......... .......... .......... .......... .......... 11% 24.6M 5s 2024-04-21T08:46:21.584 INFO:tasks.workunit.client.0.smithi057.stderr: 15800K .......... .......... .......... .......... .......... 11% 34.4M 5s 2024-04-21T08:46:21.585 INFO:tasks.workunit.client.0.smithi057.stderr: 15850K .......... .......... .......... .......... .......... 11% 24.2M 5s 2024-04-21T08:46:21.587 INFO:tasks.workunit.client.0.smithi057.stderr: 15900K .......... .......... .......... .......... .......... 11% 58.7M 5s 2024-04-21T08:46:21.588 INFO:tasks.workunit.client.0.smithi057.stderr: 15950K .......... .......... .......... .......... .......... 11% 22.9M 5s 2024-04-21T08:46:21.590 INFO:tasks.workunit.client.0.smithi057.stderr: 16000K .......... .......... .......... .......... .......... 11% 24.8M 5s 2024-04-21T08:46:21.591 INFO:tasks.workunit.client.0.smithi057.stderr: 16050K .......... .......... .......... .......... .......... 11% 33.6M 5s 2024-04-21T08:46:21.593 INFO:tasks.workunit.client.0.smithi057.stderr: 16100K .......... .......... .......... .......... .......... 11% 39.1M 5s 2024-04-21T08:46:21.596 INFO:tasks.workunit.client.0.smithi057.stderr: 16150K .......... .......... .......... .......... .......... 11% 32.8M 5s 2024-04-21T08:46:21.596 INFO:tasks.workunit.client.0.smithi057.stderr: 16200K .......... .......... .......... .......... .......... 11% 22.3M 5s 2024-04-21T08:46:21.598 INFO:tasks.workunit.client.0.smithi057.stderr: 16250K .......... .......... .......... .......... .......... 12% 25.2M 5s 2024-04-21T08:46:21.600 INFO:tasks.workunit.client.0.smithi057.stderr: 16300K .......... .......... .......... .......... .......... 12% 32.8M 5s 2024-04-21T08:46:21.602 INFO:tasks.workunit.client.0.smithi057.stderr: 16350K .......... .......... .......... .......... .......... 12% 27.0M 5s 2024-04-21T08:46:21.604 INFO:tasks.workunit.client.0.smithi057.stderr: 16400K .......... .......... .......... .......... .......... 12% 21.0M 5s 2024-04-21T08:46:21.605 INFO:tasks.workunit.client.0.smithi057.stderr: 16450K .......... .......... .......... .......... .......... 12% 54.2M 5s 2024-04-21T08:46:21.607 INFO:tasks.workunit.client.0.smithi057.stderr: 16500K .......... .......... .......... .......... .......... 12% 24.0M 5s 2024-04-21T08:46:21.608 INFO:tasks.workunit.client.0.smithi057.stderr: 16550K .......... .......... .......... .......... .......... 12% 37.3M 5s 2024-04-21T08:46:21.610 INFO:tasks.workunit.client.0.smithi057.stderr: 16600K .......... .......... .......... .......... .......... 12% 23.9M 5s 2024-04-21T08:46:21.613 INFO:tasks.workunit.client.0.smithi057.stderr: 16650K .......... .......... .......... .......... .......... 12% 19.4M 5s 2024-04-21T08:46:21.613 INFO:tasks.workunit.client.0.smithi057.stderr: 16700K .......... .......... .......... .......... .......... 12% 58.2M 5s 2024-04-21T08:46:21.616 INFO:tasks.workunit.client.0.smithi057.stderr: 16750K .......... .......... .......... .......... .......... 12% 23.8M 5s 2024-04-21T08:46:21.617 INFO:tasks.workunit.client.0.smithi057.stderr: 16800K .......... .......... .......... .......... .......... 12% 39.1M 5s 2024-04-21T08:46:21.620 INFO:tasks.workunit.client.0.smithi057.stderr: 16850K .......... .......... .......... .......... .......... 12% 23.5M 5s 2024-04-21T08:46:21.623 INFO:tasks.workunit.client.0.smithi057.stderr: 16900K .......... .......... .......... .......... .......... 12% 23.4M 5s 2024-04-21T08:46:21.623 INFO:tasks.workunit.client.0.smithi057.stderr: 16950K .......... .......... .......... .......... .......... 12% 52.7M 5s 2024-04-21T08:46:21.630 INFO:tasks.workunit.client.0.smithi057.stderr: 17000K .......... .......... .......... .......... .......... 12% 22.9M 5s 2024-04-21T08:46:21.630 INFO:tasks.workunit.client.0.smithi057.stderr: 17050K .......... .......... .......... .......... .......... 12% 42.8M 5s 2024-04-21T08:46:21.630 INFO:tasks.workunit.client.0.smithi057.stderr: 17100K .......... .......... .......... .......... .......... 12% 23.1M 5s 2024-04-21T08:46:21.630 INFO:tasks.workunit.client.0.smithi057.stderr: 17150K .......... .......... .......... .......... .......... 12% 24.3M 5s 2024-04-21T08:46:21.630 INFO:tasks.workunit.client.0.smithi057.stderr: 17200K .......... .......... .......... .......... .......... 12% 48.9M 5s 2024-04-21T08:46:21.633 INFO:tasks.workunit.client.0.smithi057.stderr: 17250K .......... .......... .......... .......... .......... 12% 23.1M 5s 2024-04-21T08:46:21.634 INFO:tasks.workunit.client.0.smithi057.stderr: 17300K .......... .......... .......... .......... .......... 12% 38.8M 5s 2024-04-21T08:46:21.637 INFO:tasks.workunit.client.0.smithi057.stderr: 17350K .......... .......... .......... .......... .......... 12% 20.6M 5s 2024-04-21T08:46:21.639 INFO:tasks.workunit.client.0.smithi057.stderr: 17400K .......... .......... .......... .......... .......... 12% 26.8M 5s 2024-04-21T08:46:21.639 INFO:tasks.workunit.client.0.smithi057.stderr: 17450K .......... .......... .......... .......... .......... 12% 52.5M 5s 2024-04-21T08:46:21.641 INFO:tasks.workunit.client.0.smithi057.stderr: 17500K .......... .......... .......... .......... .......... 12% 24.6M 5s 2024-04-21T08:46:21.642 INFO:tasks.workunit.client.0.smithi057.stderr: 17550K .......... .......... .......... .......... .......... 12% 32.7M 5s 2024-04-21T08:46:21.644 INFO:tasks.workunit.client.0.smithi057.stderr: 17600K .......... .......... .......... .......... .......... 13% 25.0M 5s 2024-04-21T08:46:21.647 INFO:tasks.workunit.client.0.smithi057.stderr: 17650K .......... .......... .......... .......... .......... 13% 22.8M 5s 2024-04-21T08:46:21.647 INFO:tasks.workunit.client.0.smithi057.stderr: 17700K .......... .......... .......... .......... .......... 13% 49.1M 5s 2024-04-21T08:46:21.649 INFO:tasks.workunit.client.0.smithi057.stderr: 17750K .......... .......... .......... .......... .......... 13% 25.3M 5s 2024-04-21T08:46:21.651 INFO:tasks.workunit.client.0.smithi057.stderr: 17800K .......... .......... .......... .......... .......... 13% 38.5M 5s 2024-04-21T08:46:21.653 INFO:tasks.workunit.client.0.smithi057.stderr: 17850K .......... .......... .......... .......... .......... 13% 23.8M 5s 2024-04-21T08:46:21.655 INFO:tasks.workunit.client.0.smithi057.stderr: 17900K .......... .......... .......... .......... .......... 13% 24.4M 5s 2024-04-21T08:46:21.656 INFO:tasks.workunit.client.0.smithi057.stderr: 17950K .......... .......... .......... .......... .......... 13% 50.0M 5s 2024-04-21T08:46:21.658 INFO:tasks.workunit.client.0.smithi057.stderr: 18000K .......... .......... .......... .......... .......... 13% 23.1M 5s 2024-04-21T08:46:21.659 INFO:tasks.workunit.client.0.smithi057.stderr: 18050K .......... .......... .......... .......... .......... 13% 36.4M 5s 2024-04-21T08:46:21.662 INFO:tasks.workunit.client.0.smithi057.stderr: 18100K .......... .......... .......... .......... .......... 13% 20.0M 5s 2024-04-21T08:46:21.663 INFO:tasks.workunit.client.0.smithi057.stderr: 18150K .......... .......... .......... .......... .......... 13% 30.8M 5s 2024-04-21T08:46:21.664 INFO:tasks.workunit.client.0.smithi057.stderr: 18200K .......... .......... .......... .......... .......... 13% 57.8M 5s 2024-04-21T08:46:21.666 INFO:tasks.workunit.client.0.smithi057.stderr: 18250K .......... .......... .......... .......... .......... 13% 22.5M 5s 2024-04-21T08:46:21.667 INFO:tasks.workunit.client.0.smithi057.stderr: 18300K .......... .......... .......... .......... .......... 13% 40.6M 5s 2024-04-21T08:46:21.671 INFO:tasks.workunit.client.0.smithi057.stderr: 18350K .......... .......... .......... .......... .......... 13% 16.8M 5s 2024-04-21T08:46:21.672 INFO:tasks.workunit.client.0.smithi057.stderr: 18400K .......... .......... .......... .......... .......... 13% 39.9M 5s 2024-04-21T08:46:21.673 INFO:tasks.workunit.client.0.smithi057.stderr: 18450K .......... .......... .......... .......... .......... 13% 47.0M 5s 2024-04-21T08:46:21.675 INFO:tasks.workunit.client.0.smithi057.stderr: 18500K .......... .......... .......... .......... .......... 13% 23.5M 5s 2024-04-21T08:46:21.677 INFO:tasks.workunit.client.0.smithi057.stderr: 18550K .......... .......... .......... .......... .......... 13% 25.2M 5s 2024-04-21T08:46:21.678 INFO:tasks.workunit.client.0.smithi057.stderr: 18600K .......... .......... .......... .......... .......... 13% 37.4M 5s 2024-04-21T08:46:21.680 INFO:tasks.workunit.client.0.smithi057.stderr: 18650K .......... .......... .......... .......... .......... 13% 25.2M 5s 2024-04-21T08:46:21.681 INFO:tasks.workunit.client.0.smithi057.stderr: 18700K .......... .......... .......... .......... .......... 13% 44.1M 5s 2024-04-21T08:46:21.684 INFO:tasks.workunit.client.0.smithi057.stderr: 18750K .......... .......... .......... .......... .......... 13% 17.4M 5s 2024-04-21T08:46:21.685 INFO:tasks.workunit.client.0.smithi057.stderr: 18800K .......... .......... .......... .......... .......... 13% 37.9M 5s 2024-04-21T08:46:21.687 INFO:tasks.workunit.client.0.smithi057.stderr: 18850K .......... .......... .......... .......... .......... 13% 23.6M 5s 2024-04-21T08:46:21.690 INFO:tasks.workunit.client.0.smithi057.stderr: 18900K .......... .......... .......... .......... .......... 13% 24.4M 5s 2024-04-21T08:46:21.690 INFO:tasks.workunit.client.0.smithi057.stderr: 18950K .......... .......... .......... .......... .......... 13% 51.2M 5s 2024-04-21T08:46:21.692 INFO:tasks.workunit.client.0.smithi057.stderr: 19000K .......... .......... .......... .......... .......... 14% 23.1M 5s 2024-04-21T08:46:21.694 INFO:tasks.workunit.client.0.smithi057.stderr: 19050K .......... .......... .......... .......... .......... 14% 23.3M 5s 2024-04-21T08:46:21.695 INFO:tasks.workunit.client.0.smithi057.stderr: 19100K .......... .......... .......... .......... .......... 14% 42.3M 4s 2024-04-21T08:46:21.698 INFO:tasks.workunit.client.0.smithi057.stderr: 19150K .......... .......... .......... .......... .......... 14% 22.8M 4s 2024-04-21T08:46:21.699 INFO:tasks.workunit.client.0.smithi057.stderr: 19200K .......... .......... .......... .......... .......... 14% 57.1M 4s 2024-04-21T08:46:21.701 INFO:tasks.workunit.client.0.smithi057.stderr: 19250K .......... .......... .......... .......... .......... 14% 21.7M 4s 2024-04-21T08:46:21.702 INFO:tasks.workunit.client.0.smithi057.stderr: 19300K .......... .......... .......... .......... .......... 14% 38.4M 4s 2024-04-21T08:46:21.705 INFO:tasks.workunit.client.0.smithi057.stderr: 19350K .......... .......... .......... .......... .......... 14% 23.6M 4s 2024-04-21T08:46:21.707 INFO:tasks.workunit.client.0.smithi057.stderr: 19400K .......... .......... .......... .......... .......... 14% 23.3M 4s 2024-04-21T08:46:21.707 INFO:tasks.workunit.client.0.smithi057.stderr: 19450K .......... .......... .......... .......... .......... 14% 57.3M 4s 2024-04-21T08:46:21.709 INFO:tasks.workunit.client.0.smithi057.stderr: 19500K .......... .......... .......... .......... .......... 14% 28.4M 4s 2024-04-21T08:46:21.710 INFO:tasks.workunit.client.0.smithi057.stderr: 19550K .......... .......... .......... .......... .......... 14% 29.5M 4s 2024-04-21T08:46:21.713 INFO:tasks.workunit.client.0.smithi057.stderr: 19600K .......... .......... .......... .......... .......... 14% 19.6M 4s 2024-04-21T08:46:21.715 INFO:tasks.workunit.client.0.smithi057.stderr: 19650K .......... .......... .......... .......... .......... 14% 29.4M 4s 2024-04-21T08:46:21.715 INFO:tasks.workunit.client.0.smithi057.stderr: 19700K .......... .......... .......... .......... .......... 14% 56.8M 4s 2024-04-21T08:46:21.718 INFO:tasks.workunit.client.0.smithi057.stderr: 19750K .......... .......... .......... .......... .......... 14% 22.5M 4s 2024-04-21T08:46:21.719 INFO:tasks.workunit.client.0.smithi057.stderr: 19800K .......... .......... .......... .......... .......... 14% 39.8M 4s 2024-04-21T08:46:21.721 INFO:tasks.workunit.client.0.smithi057.stderr: 19850K .......... .......... .......... .......... .......... 14% 24.3M 4s 2024-04-21T08:46:21.723 INFO:tasks.workunit.client.0.smithi057.stderr: 19900K .......... .......... .......... .......... .......... 14% 22.2M 4s 2024-04-21T08:46:21.724 INFO:tasks.workunit.client.0.smithi057.stderr: 19950K .......... .......... .......... .......... .......... 14% 56.0M 4s 2024-04-21T08:46:21.726 INFO:tasks.workunit.client.0.smithi057.stderr: 20000K .......... .......... .......... .......... .......... 14% 25.2M 4s 2024-04-21T08:46:21.728 INFO:tasks.workunit.client.0.smithi057.stderr: 20050K .......... .......... .......... .......... .......... 14% 25.8M 4s 2024-04-21T08:46:21.729 INFO:tasks.workunit.client.0.smithi057.stderr: 20100K .......... .......... .......... .......... .......... 14% 33.2M 4s 2024-04-21T08:46:21.731 INFO:tasks.workunit.client.0.smithi057.stderr: 20150K .......... .......... .......... .......... .......... 14% 23.3M 4s 2024-04-21T08:46:21.733 INFO:tasks.workunit.client.0.smithi057.stderr: 20200K .......... .......... .......... .......... .......... 14% 36.6M 4s 2024-04-21T08:46:21.734 INFO:tasks.workunit.client.0.smithi057.stderr: 20250K .......... .......... .......... .......... .......... 14% 23.2M 4s 2024-04-21T08:46:21.737 INFO:tasks.workunit.client.0.smithi057.stderr: 20300K .......... .......... .......... .......... .......... 14% 40.7M 4s 2024-04-21T08:46:21.739 INFO:tasks.workunit.client.0.smithi057.stderr: 20350K .......... .......... .......... .......... .......... 15% 22.5M 4s 2024-04-21T08:46:21.740 INFO:tasks.workunit.client.0.smithi057.stderr: 20400K .......... .......... .......... .......... .......... 15% 23.1M 4s 2024-04-21T08:46:21.741 INFO:tasks.workunit.client.0.smithi057.stderr: 20450K .......... .......... .......... .......... .......... 15% 62.8M 4s 2024-04-21T08:46:21.744 INFO:tasks.workunit.client.0.smithi057.stderr: 20500K .......... .......... .......... .......... .......... 15% 22.7M 4s 2024-04-21T08:46:21.745 INFO:tasks.workunit.client.0.smithi057.stderr: 20550K .......... .......... .......... .......... .......... 15% 40.7M 4s 2024-04-21T08:46:21.746 INFO:tasks.workunit.client.0.smithi057.stderr: 20600K .......... .......... .......... .......... .......... 15% 22.7M 4s 2024-04-21T08:46:21.749 INFO:tasks.workunit.client.0.smithi057.stderr: 20650K .......... .......... .......... .......... .......... 15% 23.8M 4s 2024-04-21T08:46:21.749 INFO:tasks.workunit.client.0.smithi057.stderr: 20700K .......... .......... .......... .......... .......... 15% 53.1M 4s 2024-04-21T08:46:21.751 INFO:tasks.workunit.client.0.smithi057.stderr: 20750K .......... .......... .......... .......... .......... 15% 23.6M 4s 2024-04-21T08:46:21.753 INFO:tasks.workunit.client.0.smithi057.stderr: 20800K .......... .......... .......... .......... .......... 15% 39.1M 4s 2024-04-21T08:46:21.755 INFO:tasks.workunit.client.0.smithi057.stderr: 20850K .......... .......... .......... .......... .......... 15% 24.8M 4s 2024-04-21T08:46:21.757 INFO:tasks.workunit.client.0.smithi057.stderr: 20900K .......... .......... .......... .......... .......... 15% 22.5M 4s 2024-04-21T08:46:21.758 INFO:tasks.workunit.client.0.smithi057.stderr: 20950K .......... .......... .......... .......... .......... 15% 53.2M 4s 2024-04-21T08:46:21.760 INFO:tasks.workunit.client.0.smithi057.stderr: 21000K .......... .......... .......... .......... .......... 15% 24.2M 4s 2024-04-21T08:46:21.762 INFO:tasks.workunit.client.0.smithi057.stderr: 21050K .......... .......... .......... .......... .......... 15% 24.7M 4s 2024-04-21T08:46:21.763 INFO:tasks.workunit.client.0.smithi057.stderr: 21100K .......... .......... .......... .......... .......... 15% 39.3M 4s 2024-04-21T08:46:21.765 INFO:tasks.workunit.client.0.smithi057.stderr: 21150K .......... .......... .......... .......... .......... 15% 28.8M 4s 2024-04-21T08:46:21.767 INFO:tasks.workunit.client.0.smithi057.stderr: 21200K .......... .......... .......... .......... .......... 15% 39.5M 4s 2024-04-21T08:46:21.768 INFO:tasks.workunit.client.0.smithi057.stderr: 21250K .......... .......... .......... .......... .......... 15% 22.2M 4s 2024-04-21T08:46:21.770 INFO:tasks.workunit.client.0.smithi057.stderr: 21300K .......... .......... .......... .......... .......... 15% 28.8M 4s 2024-04-21T08:46:21.772 INFO:tasks.workunit.client.0.smithi057.stderr: 21350K .......... .......... .......... .......... .......... 15% 28.6M 4s 2024-04-21T08:46:21.773 INFO:tasks.workunit.client.0.smithi057.stderr: 21400K .......... .......... .......... .......... .......... 15% 32.7M 4s 2024-04-21T08:46:21.775 INFO:tasks.workunit.client.0.smithi057.stderr: 21450K .......... .......... .......... .......... .......... 15% 36.8M 4s 2024-04-21T08:46:21.776 INFO:tasks.workunit.client.0.smithi057.stderr: 21500K .......... .......... .......... .......... .......... 15% 23.5M 4s 2024-04-21T08:46:21.778 INFO:tasks.workunit.client.0.smithi057.stderr: 21550K .......... .......... .......... .......... .......... 15% 23.9M 4s 2024-04-21T08:46:21.779 INFO:tasks.workunit.client.0.smithi057.stderr: 21600K .......... .......... .......... .......... .......... 15% 48.1M 4s 2024-04-21T08:46:21.782 INFO:tasks.workunit.client.0.smithi057.stderr: 21650K .......... .......... .......... .......... .......... 15% 20.7M 4s 2024-04-21T08:46:21.784 INFO:tasks.workunit.client.0.smithi057.stderr: 21700K .......... .......... .......... .......... .......... 16% 25.0M 4s 2024-04-21T08:46:21.785 INFO:tasks.workunit.client.0.smithi057.stderr: 21750K .......... .......... .......... .......... .......... 16% 48.9M 4s 2024-04-21T08:46:21.787 INFO:tasks.workunit.client.0.smithi057.stderr: 21800K .......... .......... .......... .......... .......... 16% 22.9M 4s 2024-04-21T08:46:21.788 INFO:tasks.workunit.client.0.smithi057.stderr: 21850K .......... .......... .......... .......... .......... 16% 44.0M 4s 2024-04-21T08:46:21.790 INFO:tasks.workunit.client.0.smithi057.stderr: 21900K .......... .......... .......... .......... .......... 16% 22.0M 4s 2024-04-21T08:46:21.792 INFO:tasks.workunit.client.0.smithi057.stderr: 21950K .......... .......... .......... .......... .......... 16% 24.4M 4s 2024-04-21T08:46:21.794 INFO:tasks.workunit.client.0.smithi057.stderr: 22000K .......... .......... .......... .......... .......... 16% 27.4M 4s 2024-04-21T08:46:21.882 INFO:tasks.workunit.client.0.smithi057.stderr: 22050K .......... .......... .......... .......... .......... 16% 602K 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22100K .......... .......... .......... .......... .......... 16% 460M 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22150K .......... .......... .......... .......... .......... 16% 469M 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22200K .......... .......... .......... .......... .......... 16% 585M 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22250K .......... .......... .......... .......... .......... 16% 572M 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22300K .......... .......... .......... .......... .......... 16% 492M 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22350K .......... .......... .......... .......... .......... 16% 581M 5s 2024-04-21T08:46:21.883 INFO:tasks.workunit.client.0.smithi057.stderr: 22400K .......... .......... .......... .......... .......... 16% 579M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22450K .......... .......... .......... .......... .......... 16% 6.09M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22500K .......... .......... .......... .......... .......... 16% 428M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22550K .......... .......... .......... .......... .......... 16% 580M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22600K .......... .......... .......... .......... .......... 16% 583M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22650K .......... .......... .......... .......... .......... 16% 519M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22700K .......... .......... .......... .......... .......... 16% 590M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22750K .......... .......... .......... .......... .......... 16% 593M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22800K .......... .......... .......... .......... .......... 16% 578M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22850K .......... .......... .......... .......... .......... 16% 500M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22900K .......... .......... .......... .......... .......... 16% 600M 5s 2024-04-21T08:46:21.932 INFO:tasks.workunit.client.0.smithi057.stderr: 22950K .......... .......... .......... .......... .......... 16% 593M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23000K .......... .......... .......... .......... .......... 16% 502M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23050K .......... .......... .......... .......... .......... 17% 480M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23100K .......... .......... .......... .......... .......... 17% 577M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23150K .......... .......... .......... .......... .......... 17% 589M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23200K .......... .......... .......... .......... .......... 17% 13.9M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23250K .......... .......... .......... .......... .......... 17% 240M 5s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23300K .......... .......... .......... .......... .......... 17% 510M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23350K .......... .......... .......... .......... .......... 17% 483M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23400K .......... .......... .......... .......... .......... 17% 596M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23450K .......... .......... .......... .......... .......... 17% 584M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23500K .......... .......... .......... .......... .......... 17% 589M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23550K .......... .......... .......... .......... .......... 17% 504M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23600K .......... .......... .......... .......... .......... 17% 563M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23650K .......... .......... .......... .......... .......... 17% 406M 4s 2024-04-21T08:46:21.933 INFO:tasks.workunit.client.0.smithi057.stderr: 23700K .......... .......... .......... .......... .......... 17% 563M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 23750K .......... .......... .......... .......... .......... 17% 471M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 23800K .......... .......... .......... .......... .......... 17% 529M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 23850K .......... .......... .......... .......... .......... 17% 527M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 23900K .......... .......... .......... .......... .......... 17% 581M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 23950K .......... .......... .......... .......... .......... 17% 506M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24000K .......... .......... .......... .......... .......... 17% 613M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24050K .......... .......... .......... .......... .......... 17% 578M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24100K .......... .......... .......... .......... .......... 17% 588M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24150K .......... .......... .......... .......... .......... 17% 514M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24200K .......... .......... .......... .......... .......... 17% 579M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24250K .......... .......... .......... .......... .......... 17% 606M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24300K .......... .......... .......... .......... .......... 17% 24.1M 4s 2024-04-21T08:46:21.934 INFO:tasks.workunit.client.0.smithi057.stderr: 24350K .......... .......... .......... .......... .......... 17% 416M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24400K .......... .......... .......... .......... .......... 18% 591M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24450K .......... .......... .......... .......... .......... 18% 539M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24500K .......... .......... .......... .......... .......... 18% 571M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24550K .......... .......... .......... .......... .......... 18% 506M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24600K .......... .......... .......... .......... .......... 18% 573M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24650K .......... .......... .......... .......... .......... 18% 580M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24700K .......... .......... .......... .......... .......... 18% 577M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24750K .......... .......... .......... .......... .......... 18% 512M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24800K .......... .......... .......... .......... .......... 18% 581M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24850K .......... .......... .......... .......... .......... 18% 586M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24900K .......... .......... .......... .......... .......... 18% 10.6M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 24950K .......... .......... .......... .......... .......... 18% 31.4M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 25000K .......... .......... .......... .......... .......... 18% 35.0M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 25050K .......... .......... .......... .......... .......... 18% 27.7M 4s 2024-04-21T08:46:21.935 INFO:tasks.workunit.client.0.smithi057.stderr: 25100K .......... .......... .......... .......... .......... 18% 34.8M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25150K .......... .......... .......... .......... .......... 18% 461M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25200K .......... .......... .......... .......... .......... 18% 589M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25250K .......... .......... .......... .......... .......... 18% 611M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25300K .......... .......... .......... .......... .......... 18% 28.0M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25350K .......... .......... .......... .......... .......... 18% 18.0M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25400K .......... .......... .......... .......... .......... 18% 485M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25450K .......... .......... .......... .......... .......... 18% 36.6M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25500K .......... .......... .......... .......... .......... 18% 42.5M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25550K .......... .......... .......... .......... .......... 18% 25.0M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25600K .......... .......... .......... .......... .......... 18% 23.4M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25650K .......... .......... .......... .......... .......... 18% 41.0M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25700K .......... .......... .......... .......... .......... 18% 26.0M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25750K .......... .......... .......... .......... .......... 19% 44.9M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25800K .......... .......... .......... .......... .......... 19% 22.7M 4s 2024-04-21T08:46:21.936 INFO:tasks.workunit.client.0.smithi057.stderr: 25850K .......... .......... .......... .......... .......... 19% 25.4M 4s 2024-04-21T08:46:21.937 INFO:tasks.workunit.client.0.smithi057.stderr: 25900K .......... .......... .......... .......... .......... 19% 40.7M 4s 2024-04-21T08:46:21.937 INFO:tasks.workunit.client.0.smithi057.stderr: 25950K .......... .......... .......... .......... .......... 19% 23.5M 4s 2024-04-21T08:46:21.937 INFO:tasks.workunit.client.0.smithi057.stderr: 26000K .......... .......... .......... .......... .......... 19% 48.8M 4s 2024-04-21T08:46:21.940 INFO:tasks.workunit.client.0.smithi057.stderr: 26050K .......... .......... .......... .......... .......... 19% 24.5M 4s 2024-04-21T08:46:21.940 INFO:tasks.workunit.client.0.smithi057.stderr: 26100K .......... .......... .......... .......... .......... 19% 22.9M 4s 2024-04-21T08:46:21.940 INFO:tasks.workunit.client.0.smithi057.stderr: 26150K .......... .......... .......... .......... .......... 19% 40.3M 4s 2024-04-21T08:46:21.940 INFO:tasks.workunit.client.0.smithi057.stderr: 26200K .......... .......... .......... .......... .......... 19% 25.1M 4s 2024-04-21T08:46:21.940 INFO:tasks.workunit.client.0.smithi057.stderr: 26250K .......... .......... .......... .......... .......... 19% 45.4M 4s 2024-04-21T08:46:21.941 INFO:tasks.workunit.client.0.smithi057.stderr: 26300K .......... .......... .......... .......... .......... 19% 21.7M 4s 2024-04-21T08:46:21.943 INFO:tasks.workunit.client.0.smithi057.stderr: 26350K .......... .......... .......... .......... .......... 19% 31.2M 4s 2024-04-21T08:46:21.944 INFO:tasks.workunit.client.0.smithi057.stderr: 26400K .......... .......... .......... .......... .......... 19% 25.5M 4s 2024-04-21T08:46:21.947 INFO:tasks.workunit.client.0.smithi057.stderr: 26450K .......... .......... .......... .......... .......... 19% 24.5M 4s 2024-04-21T08:46:21.947 INFO:tasks.workunit.client.0.smithi057.stderr: 26500K .......... .......... .......... .......... .......... 19% 55.4M 4s 2024-04-21T08:46:21.949 INFO:tasks.workunit.client.0.smithi057.stderr: 26550K .......... .......... .......... .......... .......... 19% 34.6M 4s 2024-04-21T08:46:21.951 INFO:tasks.workunit.client.0.smithi057.stderr: 26600K .......... .......... .......... .......... .......... 19% 22.2M 4s 2024-04-21T08:46:21.954 INFO:tasks.workunit.client.0.smithi057.stderr: 26650K .......... .......... .......... .......... .......... 19% 22.3M 4s 2024-04-21T08:46:21.956 INFO:tasks.workunit.client.0.smithi057.stderr: 26700K .......... .......... .......... .......... .......... 19% 24.4M 4s 2024-04-21T08:46:21.956 INFO:tasks.workunit.client.0.smithi057.stderr: 26750K .......... .......... .......... .......... .......... 19% 91.9M 4s 2024-04-21T08:46:21.958 INFO:tasks.workunit.client.0.smithi057.stderr: 26800K .......... .......... .......... .......... .......... 19% 21.4M 4s 2024-04-21T08:46:21.961 INFO:tasks.workunit.client.0.smithi057.stderr: 26850K .......... .......... .......... .......... .......... 19% 17.5M 4s 2024-04-21T08:46:21.963 INFO:tasks.workunit.client.0.smithi057.stderr: 26900K .......... .......... .......... .......... .......... 19% 23.3M 4s 2024-04-21T08:46:21.965 INFO:tasks.workunit.client.0.smithi057.stderr: 26950K .......... .......... .......... .......... .......... 19% 24.6M 4s 2024-04-21T08:46:21.965 INFO:tasks.workunit.client.0.smithi057.stderr: 27000K .......... .......... .......... .......... .......... 19% 471M 4s 2024-04-21T08:46:21.967 INFO:tasks.workunit.client.0.smithi057.stderr: 27050K .......... .......... .......... .......... .......... 19% 27.2M 4s 2024-04-21T08:46:21.969 INFO:tasks.workunit.client.0.smithi057.stderr: 27100K .......... .......... .......... .......... .......... 19% 21.9M 4s 2024-04-21T08:46:21.971 INFO:tasks.workunit.client.0.smithi057.stderr: 27150K .......... .......... .......... .......... .......... 20% 21.1M 4s 2024-04-21T08:46:21.973 INFO:tasks.workunit.client.0.smithi057.stderr: 27200K .......... .......... .......... .......... .......... 20% 29.1M 4s 2024-04-21T08:46:21.973 INFO:tasks.workunit.client.0.smithi057.stderr: 27250K .......... .......... .......... .......... .......... 20% 140M 4s 2024-04-21T08:46:21.976 INFO:tasks.workunit.client.0.smithi057.stderr: 27300K .......... .......... .......... .......... .......... 20% 26.6M 4s 2024-04-21T08:46:21.977 INFO:tasks.workunit.client.0.smithi057.stderr: 27350K .......... .......... .......... .......... .......... 20% 23.8M 4s 2024-04-21T08:46:21.980 INFO:tasks.workunit.client.0.smithi057.stderr: 27400K .......... .......... .......... .......... .......... 20% 23.2M 4s 2024-04-21T08:46:21.982 INFO:tasks.workunit.client.0.smithi057.stderr: 27450K .......... .......... .......... .......... .......... 20% 19.7M 4s 2024-04-21T08:46:21.982 INFO:tasks.workunit.client.0.smithi057.stderr: 27500K .......... .......... .......... .......... .......... 20% 569M 4s 2024-04-21T08:46:21.984 INFO:tasks.workunit.client.0.smithi057.stderr: 27550K .......... .......... .......... .......... .......... 20% 21.8M 4s 2024-04-21T08:46:21.989 INFO:tasks.workunit.client.0.smithi057.stderr: 27600K .......... .......... .......... .......... .......... 20% 25.1M 4s 2024-04-21T08:46:21.989 INFO:tasks.workunit.client.0.smithi057.stderr: 27650K .......... .......... .......... .......... .......... 20% 23.3M 4s 2024-04-21T08:46:21.990 INFO:tasks.workunit.client.0.smithi057.stderr: 27700K .......... .......... .......... .......... .......... 20% 26.9M 4s 2024-04-21T08:46:21.990 INFO:tasks.workunit.client.0.smithi057.stderr: 27750K .......... .......... .......... .......... .......... 20% 311M 4s 2024-04-21T08:46:21.992 INFO:tasks.workunit.client.0.smithi057.stderr: 27800K .......... .......... .......... .......... .......... 20% 26.8M 4s 2024-04-21T08:46:21.994 INFO:tasks.workunit.client.0.smithi057.stderr: 27850K .......... .......... .......... .......... .......... 20% 25.5M 4s 2024-04-21T08:46:21.996 INFO:tasks.workunit.client.0.smithi057.stderr: 27900K .......... .......... .......... .......... .......... 20% 21.4M 4s 2024-04-21T08:46:21.998 INFO:tasks.workunit.client.0.smithi057.stderr: 27950K .......... .......... .......... .......... .......... 20% 23.8M 4s 2024-04-21T08:46:21.999 INFO:tasks.workunit.client.0.smithi057.stderr: 28000K .......... .......... .......... .......... .......... 20% 90.0M 4s 2024-04-21T08:46:22.001 INFO:tasks.workunit.client.0.smithi057.stderr: 28050K .......... .......... .......... .......... .......... 20% 23.8M 4s 2024-04-21T08:46:22.003 INFO:tasks.workunit.client.0.smithi057.stderr: 28100K .......... .......... .......... .......... .......... 20% 31.1M 4s 2024-04-21T08:46:22.005 INFO:tasks.workunit.client.0.smithi057.stderr: 28150K .......... .......... .......... .......... .......... 20% 24.5M 4s 2024-04-21T08:46:22.007 INFO:tasks.workunit.client.0.smithi057.stderr: 28200K .......... .......... .......... .......... .......... 20% 24.3M 4s 2024-04-21T08:46:22.009 INFO:tasks.workunit.client.0.smithi057.stderr: 28250K .......... .......... .......... .......... .......... 20% 64.7M 4s 2024-04-21T08:46:22.009 INFO:tasks.workunit.client.0.smithi057.stderr: 28300K .......... .......... .......... .......... .......... 20% 23.7M 4s 2024-04-21T08:46:22.011 INFO:tasks.workunit.client.0.smithi057.stderr: 28350K .......... .......... .......... .......... .......... 20% 19.6M 4s 2024-04-21T08:46:22.013 INFO:tasks.workunit.client.0.smithi057.stderr: 28400K .......... .......... .......... .......... .......... 20% 35.4M 4s 2024-04-21T08:46:22.015 INFO:tasks.workunit.client.0.smithi057.stderr: 28450K .......... .......... .......... .......... .......... 20% 24.3M 4s 2024-04-21T08:46:22.017 INFO:tasks.workunit.client.0.smithi057.stderr: 28500K .......... .......... .......... .......... .......... 21% 22.4M 4s 2024-04-21T08:46:22.018 INFO:tasks.workunit.client.0.smithi057.stderr: 28550K .......... .......... .......... .......... .......... 21% 59.4M 4s 2024-04-21T08:46:22.020 INFO:tasks.workunit.client.0.smithi057.stderr: 28600K .......... .......... .......... .......... .......... 21% 36.9M 4s 2024-04-21T08:46:22.022 INFO:tasks.workunit.client.0.smithi057.stderr: 28650K .......... .......... .......... .......... .......... 21% 22.5M 4s 2024-04-21T08:46:22.023 INFO:tasks.workunit.client.0.smithi057.stderr: 28700K .......... .......... .......... .......... .......... 21% 24.6M 4s 2024-04-21T08:46:22.026 INFO:tasks.workunit.client.0.smithi057.stderr: 28750K .......... .......... .......... .......... .......... 21% 24.3M 4s 2024-04-21T08:46:22.026 INFO:tasks.workunit.client.0.smithi057.stderr: 28800K .......... .......... .......... .......... .......... 21% 60.2M 4s 2024-04-21T08:46:22.028 INFO:tasks.workunit.client.0.smithi057.stderr: 28850K .......... .......... .......... .......... .......... 21% 23.5M 4s 2024-04-21T08:46:22.030 INFO:tasks.workunit.client.0.smithi057.stderr: 28900K .......... .......... .......... .......... .......... 21% 35.5M 4s 2024-04-21T08:46:22.032 INFO:tasks.workunit.client.0.smithi057.stderr: 28950K .......... .......... .......... .......... .......... 21% 23.9M 4s 2024-04-21T08:46:22.034 INFO:tasks.workunit.client.0.smithi057.stderr: 29000K .......... .......... .......... .......... .......... 21% 23.4M 4s 2024-04-21T08:46:22.035 INFO:tasks.workunit.client.0.smithi057.stderr: 29050K .......... .......... .......... .......... .......... 21% 46.3M 4s 2024-04-21T08:46:22.037 INFO:tasks.workunit.client.0.smithi057.stderr: 29100K .......... .......... .......... .......... .......... 21% 24.1M 4s 2024-04-21T08:46:22.039 INFO:tasks.workunit.client.0.smithi057.stderr: 29150K .......... .......... .......... .......... .......... 21% 40.1M 4s 2024-04-21T08:46:22.040 INFO:tasks.workunit.client.0.smithi057.stderr: 29200K .......... .......... .......... .......... .......... 21% 26.0M 4s 2024-04-21T08:46:22.042 INFO:tasks.workunit.client.0.smithi057.stderr: 29250K .......... .......... .......... .......... .......... 21% 23.3M 4s 2024-04-21T08:46:22.043 INFO:tasks.workunit.client.0.smithi057.stderr: 29300K .......... .......... .......... .......... .......... 21% 47.5M 4s 2024-04-21T08:46:22.045 INFO:tasks.workunit.client.0.smithi057.stderr: 29350K .......... .......... .......... .......... .......... 21% 23.1M 4s 2024-04-21T08:46:22.047 INFO:tasks.workunit.client.0.smithi057.stderr: 29400K .......... .......... .......... .......... .......... 21% 38.6M 4s 2024-04-21T08:46:22.049 INFO:tasks.workunit.client.0.smithi057.stderr: 29450K .......... .......... .......... .......... .......... 21% 23.6M 4s 2024-04-21T08:46:22.051 INFO:tasks.workunit.client.0.smithi057.stderr: 29500K .......... .......... .......... .......... .......... 21% 21.7M 4s 2024-04-21T08:46:22.052 INFO:tasks.workunit.client.0.smithi057.stderr: 29550K .......... .......... .......... .......... .......... 21% 50.5M 4s 2024-04-21T08:46:22.054 INFO:tasks.workunit.client.0.smithi057.stderr: 29600K .......... .......... .......... .......... .......... 21% 24.2M 4s 2024-04-21T08:46:22.055 INFO:tasks.workunit.client.0.smithi057.stderr: 29650K .......... .......... .......... .......... .......... 21% 40.2M 4s 2024-04-21T08:46:22.057 INFO:tasks.workunit.client.0.smithi057.stderr: 29700K .......... .......... .......... .......... .......... 21% 24.8M 4s 2024-04-21T08:46:22.059 INFO:tasks.workunit.client.0.smithi057.stderr: 29750K .......... .......... .......... .......... .......... 21% 23.7M 4s 2024-04-21T08:46:22.060 INFO:tasks.workunit.client.0.smithi057.stderr: 29800K .......... .......... .......... .......... .......... 21% 44.6M 4s 2024-04-21T08:46:22.062 INFO:tasks.workunit.client.0.smithi057.stderr: 29850K .......... .......... .......... .......... .......... 22% 32.8M 4s 2024-04-21T08:46:22.064 INFO:tasks.workunit.client.0.smithi057.stderr: 29900K .......... .......... .......... .......... .......... 22% 25.3M 4s 2024-04-21T08:46:22.067 INFO:tasks.workunit.client.0.smithi057.stderr: 29950K .......... .......... .......... .......... .......... 22% 22.3M 4s 2024-04-21T08:46:22.067 INFO:tasks.workunit.client.0.smithi057.stderr: 30000K .......... .......... .......... .......... .......... 22% 65.5M 4s 2024-04-21T08:46:22.068 INFO:tasks.workunit.client.0.smithi057.stderr: 30050K .......... .......... .......... .......... .......... 22% 24.9M 4s 2024-04-21T08:46:22.071 INFO:tasks.workunit.client.0.smithi057.stderr: 30100K .......... .......... .......... .......... .......... 22% 23.2M 4s 2024-04-21T08:46:22.072 INFO:tasks.workunit.client.0.smithi057.stderr: 30150K .......... .......... .......... .......... .......... 22% 38.3M 4s 2024-04-21T08:46:22.074 INFO:tasks.workunit.client.0.smithi057.stderr: 30200K .......... .......... .......... .......... .......... 22% 26.0M 4s 2024-04-21T08:46:22.075 INFO:tasks.workunit.client.0.smithi057.stderr: 30250K .......... .......... .......... .......... .......... 22% 36.2M 4s 2024-04-21T08:46:22.078 INFO:tasks.workunit.client.0.smithi057.stderr: 30300K .......... .......... .......... .......... .......... 22% 18.2M 4s 2024-04-21T08:46:22.080 INFO:tasks.workunit.client.0.smithi057.stderr: 30350K .......... .......... .......... .......... .......... 22% 23.0M 4s 2024-04-21T08:46:22.082 INFO:tasks.workunit.client.0.smithi057.stderr: 30400K .......... .......... .......... .......... .......... 22% 19.7M 4s 2024-04-21T08:46:22.084 INFO:tasks.workunit.client.0.smithi057.stderr: 30450K .......... .......... .......... .......... .......... 22% 35.8M 4s 2024-04-21T08:46:22.086 INFO:tasks.workunit.client.0.smithi057.stderr: 30500K .......... .......... .......... .......... .......... 22% 24.9M 4s 2024-04-21T08:46:22.087 INFO:tasks.workunit.client.0.smithi057.stderr: 30550K .......... .......... .......... .......... .......... 22% 51.2M 4s 2024-04-21T08:46:22.089 INFO:tasks.workunit.client.0.smithi057.stderr: 30600K .......... .......... .......... .......... .......... 22% 24.1M 4s 2024-04-21T08:46:22.090 INFO:tasks.workunit.client.0.smithi057.stderr: 30650K .......... .......... .......... .......... .......... 22% 36.2M 4s 2024-04-21T08:46:22.092 INFO:tasks.workunit.client.0.smithi057.stderr: 30700K .......... .......... .......... .......... .......... 22% 23.6M 4s 2024-04-21T08:46:22.094 INFO:tasks.workunit.client.0.smithi057.stderr: 30750K .......... .......... .......... .......... .......... 22% 24.8M 4s 2024-04-21T08:46:22.095 INFO:tasks.workunit.client.0.smithi057.stderr: 30800K .......... .......... .......... .......... .......... 22% 46.5M 4s 2024-04-21T08:46:22.097 INFO:tasks.workunit.client.0.smithi057.stderr: 30850K .......... .......... .......... .......... .......... 22% 22.5M 4s 2024-04-21T08:46:22.099 INFO:tasks.workunit.client.0.smithi057.stderr: 30900K .......... .......... .......... .......... .......... 22% 39.3M 4s 2024-04-21T08:46:22.102 INFO:tasks.workunit.client.0.smithi057.stderr: 30950K .......... .......... .......... .......... .......... 22% 19.6M 4s 2024-04-21T08:46:22.102 INFO:tasks.workunit.client.0.smithi057.stderr: 31000K .......... .......... .......... .......... .......... 22% 65.9M 4s 2024-04-21T08:46:22.104 INFO:tasks.workunit.client.0.smithi057.stderr: 31050K .......... .......... .......... .......... .......... 22% 27.8M 4s 2024-04-21T08:46:22.105 INFO:tasks.workunit.client.0.smithi057.stderr: 31100K .......... .......... .......... .......... .......... 22% 28.9M 4s 2024-04-21T08:46:22.108 INFO:tasks.workunit.client.0.smithi057.stderr: 31150K .......... .......... .......... .......... .......... 22% 25.8M 4s 2024-04-21T08:46:22.109 INFO:tasks.workunit.client.0.smithi057.stderr: 31200K .......... .......... .......... .......... .......... 23% 23.9M 4s 2024-04-21T08:46:22.111 INFO:tasks.workunit.client.0.smithi057.stderr: 31250K .......... .......... .......... .......... .......... 23% 25.3M 4s 2024-04-21T08:46:22.112 INFO:tasks.workunit.client.0.smithi057.stderr: 31300K .......... .......... .......... .......... .......... 23% 53.9M 4s 2024-04-21T08:46:22.114 INFO:tasks.workunit.client.0.smithi057.stderr: 31350K .......... .......... .......... .......... .......... 23% 23.8M 4s 2024-04-21T08:46:22.115 INFO:tasks.workunit.client.0.smithi057.stderr: 31400K .......... .......... .......... .......... .......... 23% 39.1M 4s 2024-04-21T08:46:22.117 INFO:tasks.workunit.client.0.smithi057.stderr: 31450K .......... .......... .......... .......... .......... 23% 23.1M 4s 2024-04-21T08:46:22.119 INFO:tasks.workunit.client.0.smithi057.stderr: 31500K .......... .......... .......... .......... .......... 23% 32.5M 4s 2024-04-21T08:46:22.120 INFO:tasks.workunit.client.0.smithi057.stderr: 31550K .......... .......... .......... .......... .......... 23% 37.3M 4s 2024-04-21T08:46:22.122 INFO:tasks.workunit.client.0.smithi057.stderr: 31600K .......... .......... .......... .......... .......... 23% 22.2M 4s 2024-04-21T08:46:22.124 INFO:tasks.workunit.client.0.smithi057.stderr: 31650K .......... .......... .......... .......... .......... 23% 37.4M 4s 2024-04-21T08:46:22.126 INFO:tasks.workunit.client.0.smithi057.stderr: 31700K .......... .......... .......... .......... .......... 23% 24.7M 4s 2024-04-21T08:46:22.128 INFO:tasks.workunit.client.0.smithi057.stderr: 31750K .......... .......... .......... .......... .......... 23% 23.4M 4s 2024-04-21T08:46:22.129 INFO:tasks.workunit.client.0.smithi057.stderr: 31800K .......... .......... .......... .......... .......... 23% 53.4M 4s 2024-04-21T08:46:22.131 INFO:tasks.workunit.client.0.smithi057.stderr: 31850K .......... .......... .......... .......... .......... 23% 23.5M 4s 2024-04-21T08:46:22.133 INFO:tasks.workunit.client.0.smithi057.stderr: 31900K .......... .......... .......... .......... .......... 23% 38.1M 4s 2024-04-21T08:46:22.134 INFO:tasks.workunit.client.0.smithi057.stderr: 31950K .......... .......... .......... .......... .......... 23% 23.8M 4s 2024-04-21T08:46:22.136 INFO:tasks.workunit.client.0.smithi057.stderr: 32000K .......... .......... .......... .......... .......... 23% 22.9M 4s 2024-04-21T08:46:22.137 INFO:tasks.workunit.client.0.smithi057.stderr: 32050K .......... .......... .......... .......... .......... 23% 52.6M 4s 2024-04-21T08:46:22.140 INFO:tasks.workunit.client.0.smithi057.stderr: 32100K .......... .......... .......... .......... .......... 23% 20.3M 4s 2024-04-21T08:46:22.142 INFO:tasks.workunit.client.0.smithi057.stderr: 32150K .......... .......... .......... .......... .......... 23% 32.7M 4s 2024-04-21T08:46:22.144 INFO:tasks.workunit.client.0.smithi057.stderr: 32200K .......... .......... .......... .......... .......... 23% 23.6M 4s 2024-04-21T08:46:22.145 INFO:tasks.workunit.client.0.smithi057.stderr: 32250K .......... .......... .......... .......... .......... 23% 24.5M 4s 2024-04-21T08:46:22.146 INFO:tasks.workunit.client.0.smithi057.stderr: 32300K .......... .......... .......... .......... .......... 23% 53.6M 4s 2024-04-21T08:46:22.148 INFO:tasks.workunit.client.0.smithi057.stderr: 32350K .......... .......... .......... .......... .......... 23% 24.1M 4s 2024-04-21T08:46:22.150 INFO:tasks.workunit.client.0.smithi057.stderr: 32400K .......... .......... .......... .......... .......... 23% 22.9M 4s 2024-04-21T08:46:22.152 INFO:tasks.workunit.client.0.smithi057.stderr: 32450K .......... .......... .......... .......... .......... 23% 28.1M 4s 2024-04-21T08:46:22.154 INFO:tasks.workunit.client.0.smithi057.stderr: 32500K .......... .......... .......... .......... .......... 23% 23.4M 4s 2024-04-21T08:46:22.155 INFO:tasks.workunit.client.0.smithi057.stderr: 32550K .......... .......... .......... .......... .......... 24% 52.7M 4s 2024-04-21T08:46:22.157 INFO:tasks.workunit.client.0.smithi057.stderr: 32600K .......... .......... .......... .......... .......... 24% 23.1M 4s 2024-04-21T08:46:22.159 INFO:tasks.workunit.client.0.smithi057.stderr: 32650K .......... .......... .......... .......... .......... 24% 24.8M 4s 2024-04-21T08:46:22.160 INFO:tasks.workunit.client.0.smithi057.stderr: 32700K .......... .......... .......... .......... .......... 24% 42.0M 4s 2024-04-21T08:46:22.162 INFO:tasks.workunit.client.0.smithi057.stderr: 32750K .......... .......... .......... .......... .......... 24% 23.1M 4s 2024-04-21T08:46:22.163 INFO:tasks.workunit.client.0.smithi057.stderr: 32800K .......... .......... .......... .......... .......... 24% 46.8M 4s 2024-04-21T08:46:22.165 INFO:tasks.workunit.client.0.smithi057.stderr: 32850K .......... .......... .......... .......... .......... 24% 23.7M 4s 2024-04-21T08:46:22.167 INFO:tasks.workunit.client.0.smithi057.stderr: 32900K .......... .......... .......... .......... .......... 24% 23.8M 4s 2024-04-21T08:46:22.169 INFO:tasks.workunit.client.0.smithi057.stderr: 32950K .......... .......... .......... .......... .......... 24% 40.2M 4s 2024-04-21T08:46:22.171 INFO:tasks.workunit.client.0.smithi057.stderr: 33000K .......... .......... .......... .......... .......... 24% 21.2M 4s 2024-04-21T08:46:22.172 INFO:tasks.workunit.client.0.smithi057.stderr: 33050K .......... .......... .......... .......... .......... 24% 53.3M 4s 2024-04-21T08:46:22.174 INFO:tasks.workunit.client.0.smithi057.stderr: 33100K .......... .......... .......... .......... .......... 24% 24.9M 4s 2024-04-21T08:46:22.176 INFO:tasks.workunit.client.0.smithi057.stderr: 33150K .......... .......... .......... .......... .......... 24% 20.5M 4s 2024-04-21T08:46:22.177 INFO:tasks.workunit.client.0.smithi057.stderr: 33200K .......... .......... .......... .......... .......... 24% 37.8M 4s 2024-04-21T08:46:22.180 INFO:tasks.workunit.client.0.smithi057.stderr: 33250K .......... .......... .......... .......... .......... 24% 22.8M 4s 2024-04-21T08:46:22.180 INFO:tasks.workunit.client.0.smithi057.stderr: 33300K .......... .......... .......... .......... .......... 24% 63.4M 4s 2024-04-21T08:46:22.183 INFO:tasks.workunit.client.0.smithi057.stderr: 33350K .......... .......... .......... .......... .......... 24% 22.1M 4s 2024-04-21T08:46:22.184 INFO:tasks.workunit.client.0.smithi057.stderr: 33400K .......... .......... .......... .......... .......... 24% 24.2M 4s 2024-04-21T08:46:22.186 INFO:tasks.workunit.client.0.smithi057.stderr: 33450K .......... .......... .......... .......... .......... 24% 40.7M 4s 2024-04-21T08:46:22.188 INFO:tasks.workunit.client.0.smithi057.stderr: 33500K .......... .......... .......... .......... .......... 24% 22.0M 4s 2024-04-21T08:46:22.190 INFO:tasks.workunit.client.0.smithi057.stderr: 33550K .......... .......... .......... .......... .......... 24% 57.8M 4s 2024-04-21T08:46:22.191 INFO:tasks.workunit.client.0.smithi057.stderr: 33600K .......... .......... .......... .......... .......... 24% 22.8M 4s 2024-04-21T08:46:22.193 INFO:tasks.workunit.client.0.smithi057.stderr: 33650K .......... .......... .......... .......... .......... 24% 23.4M 4s 2024-04-21T08:46:22.194 INFO:tasks.workunit.client.0.smithi057.stderr: 33700K .......... .......... .......... .......... .......... 24% 38.0M 4s 2024-04-21T08:46:22.197 INFO:tasks.workunit.client.0.smithi057.stderr: 33750K .......... .......... .......... .......... .......... 24% 24.1M 4s 2024-04-21T08:46:22.199 INFO:tasks.workunit.client.0.smithi057.stderr: 33800K .......... .......... .......... .......... .......... 24% 24.1M 4s 2024-04-21T08:46:22.200 INFO:tasks.workunit.client.0.smithi057.stderr: 33850K .......... .......... .......... .......... .......... 24% 44.9M 4s 2024-04-21T08:46:22.201 INFO:tasks.workunit.client.0.smithi057.stderr: 33900K .......... .......... .......... .......... .......... 25% 24.7M 4s 2024-04-21T08:46:22.203 INFO:tasks.workunit.client.0.smithi057.stderr: 33950K .......... .......... .......... .......... .......... 25% 41.5M 4s 2024-04-21T08:46:22.206 INFO:tasks.workunit.client.0.smithi057.stderr: 34000K .......... .......... .......... .......... .......... 25% 22.5M 4s 2024-04-21T08:46:22.207 INFO:tasks.workunit.client.0.smithi057.stderr: 34050K .......... .......... .......... .......... .......... 25% 22.7M 4s 2024-04-21T08:46:22.208 INFO:tasks.workunit.client.0.smithi057.stderr: 34100K .......... .......... .......... .......... .......... 25% 48.5M 4s 2024-04-21T08:46:22.210 INFO:tasks.workunit.client.0.smithi057.stderr: 34150K .......... .......... .......... .......... .......... 25% 24.4M 4s 2024-04-21T08:46:22.211 INFO:tasks.workunit.client.0.smithi057.stderr: 34200K .......... .......... .......... .......... .......... 25% 43.8M 4s 2024-04-21T08:46:22.213 INFO:tasks.workunit.client.0.smithi057.stderr: 34250K .......... .......... .......... .......... .......... 25% 23.5M 4s 2024-04-21T08:46:22.215 INFO:tasks.workunit.client.0.smithi057.stderr: 34300K .......... .......... .......... .......... .......... 25% 22.5M 4s 2024-04-21T08:46:22.216 INFO:tasks.workunit.client.0.smithi057.stderr: 34350K .......... .......... .......... .......... .......... 25% 45.6M 4s 2024-04-21T08:46:22.218 INFO:tasks.workunit.client.0.smithi057.stderr: 34400K .......... .......... .......... .......... .......... 25% 25.2M 4s 2024-04-21T08:46:22.220 INFO:tasks.workunit.client.0.smithi057.stderr: 34450K .......... .......... .......... .......... .......... 25% 34.7M 4s 2024-04-21T08:46:22.222 INFO:tasks.workunit.client.0.smithi057.stderr: 34500K .......... .......... .......... .......... .......... 25% 25.2M 4s 2024-04-21T08:46:22.224 INFO:tasks.workunit.client.0.smithi057.stderr: 34550K .......... .......... .......... .......... .......... 25% 22.4M 4s 2024-04-21T08:46:22.225 INFO:tasks.workunit.client.0.smithi057.stderr: 34600K .......... .......... .......... .......... .......... 25% 52.1M 4s 2024-04-21T08:46:22.227 INFO:tasks.workunit.client.0.smithi057.stderr: 34650K .......... .......... .......... .......... .......... 25% 24.0M 4s 2024-04-21T08:46:22.228 INFO:tasks.workunit.client.0.smithi057.stderr: 34700K .......... .......... .......... .......... .......... 25% 41.3M 4s 2024-04-21T08:46:22.230 INFO:tasks.workunit.client.0.smithi057.stderr: 34750K .......... .......... .......... .......... .......... 25% 23.5M 4s 2024-04-21T08:46:22.232 INFO:tasks.workunit.client.0.smithi057.stderr: 34800K .......... .......... .......... .......... .......... 25% 23.4M 4s 2024-04-21T08:46:22.233 INFO:tasks.workunit.client.0.smithi057.stderr: 34850K .......... .......... .......... .......... .......... 25% 45.8M 4s 2024-04-21T08:46:22.235 INFO:tasks.workunit.client.0.smithi057.stderr: 34900K .......... .......... .......... .......... .......... 25% 25.0M 4s 2024-04-21T08:46:22.238 INFO:tasks.workunit.client.0.smithi057.stderr: 34950K .......... .......... .......... .......... .......... 25% 37.1M 4s 2024-04-21T08:46:22.239 INFO:tasks.workunit.client.0.smithi057.stderr: 35000K .......... .......... .......... .......... .......... 25% 23.3M 4s 2024-04-21T08:46:22.241 INFO:tasks.workunit.client.0.smithi057.stderr: 35050K .......... .......... .......... .......... .......... 25% 24.8M 4s 2024-04-21T08:46:22.242 INFO:tasks.workunit.client.0.smithi057.stderr: 35100K .......... .......... .......... .......... .......... 25% 51.6M 4s 2024-04-21T08:46:22.244 INFO:tasks.workunit.client.0.smithi057.stderr: 35150K .......... .......... .......... .......... .......... 25% 19.0M 4s 2024-04-21T08:46:22.245 INFO:tasks.workunit.client.0.smithi057.stderr: 35200K .......... .......... .......... .......... .......... 25% 34.8M 4s 2024-04-21T08:46:22.248 INFO:tasks.workunit.client.0.smithi057.stderr: 35250K .......... .......... .......... .......... .......... 26% 21.9M 4s 2024-04-21T08:46:22.250 INFO:tasks.workunit.client.0.smithi057.stderr: 35300K .......... .......... .......... .......... .......... 26% 24.9M 4s 2024-04-21T08:46:22.250 INFO:tasks.workunit.client.0.smithi057.stderr: 35350K .......... .......... .......... .......... .......... 26% 60.1M 4s 2024-04-21T08:46:22.253 INFO:tasks.workunit.client.0.smithi057.stderr: 35400K .......... .......... .......... .......... .......... 26% 22.5M 4s 2024-04-21T08:46:22.254 INFO:tasks.workunit.client.0.smithi057.stderr: 35450K .......... .......... .......... .......... .......... 26% 40.3M 4s 2024-04-21T08:46:22.256 INFO:tasks.workunit.client.0.smithi057.stderr: 35500K .......... .......... .......... .......... .......... 26% 22.7M 4s 2024-04-21T08:46:22.258 INFO:tasks.workunit.client.0.smithi057.stderr: 35550K .......... .......... .......... .......... .......... 26% 25.3M 4s 2024-04-21T08:46:22.259 INFO:tasks.workunit.client.0.smithi057.stderr: 35600K .......... .......... .......... .......... .......... 26% 43.6M 4s 2024-04-21T08:46:22.260 INFO:tasks.workunit.client.0.smithi057.stderr: 35650K .......... .......... .......... .......... .......... 26% 38.9M 4s 2024-04-21T08:46:22.262 INFO:tasks.workunit.client.0.smithi057.stderr: 35700K .......... .......... .......... .......... .......... 26% 26.1M 4s 2024-04-21T08:46:22.264 INFO:tasks.workunit.client.0.smithi057.stderr: 35750K .......... .......... .......... .......... .......... 26% 22.2M 4s 2024-04-21T08:46:22.266 INFO:tasks.workunit.client.0.smithi057.stderr: 35800K .......... .......... .......... .......... .......... 26% 25.4M 4s 2024-04-21T08:46:22.268 INFO:tasks.workunit.client.0.smithi057.stderr: 35850K .......... .......... .......... .......... .......... 26% 44.7M 4s 2024-04-21T08:46:22.269 INFO:tasks.workunit.client.0.smithi057.stderr: 35900K .......... .......... .......... .......... .......... 26% 23.7M 4s 2024-04-21T08:46:22.271 INFO:tasks.workunit.client.0.smithi057.stderr: 35950K .......... .......... .......... .......... .......... 26% 40.9M 4s 2024-04-21T08:46:22.273 INFO:tasks.workunit.client.0.smithi057.stderr: 36000K .......... .......... .......... .......... .......... 26% 23.6M 4s 2024-04-21T08:46:22.276 INFO:tasks.workunit.client.0.smithi057.stderr: 36050K .......... .......... .......... .......... .......... 26% 24.2M 4s 2024-04-21T08:46:22.276 INFO:tasks.workunit.client.0.smithi057.stderr: 36100K .......... .......... .......... .......... .......... 26% 54.9M 4s 2024-04-21T08:46:22.278 INFO:tasks.workunit.client.0.smithi057.stderr: 36150K .......... .......... .......... .......... .......... 26% 23.6M 4s 2024-04-21T08:46:22.279 INFO:tasks.workunit.client.0.smithi057.stderr: 36200K .......... .......... .......... .......... .......... 26% 40.9M 4s 2024-04-21T08:46:22.281 INFO:tasks.workunit.client.0.smithi057.stderr: 36250K .......... .......... .......... .......... .......... 26% 23.4M 4s 2024-04-21T08:46:22.283 INFO:tasks.workunit.client.0.smithi057.stderr: 36300K .......... .......... .......... .......... .......... 26% 24.1M 4s 2024-04-21T08:46:22.284 INFO:tasks.workunit.client.0.smithi057.stderr: 36350K .......... .......... .......... .......... .......... 26% 43.0M 4s 2024-04-21T08:46:22.286 INFO:tasks.workunit.client.0.smithi057.stderr: 36400K .......... .......... .......... .......... .......... 26% 25.1M 4s 2024-04-21T08:46:22.287 INFO:tasks.workunit.client.0.smithi057.stderr: 36450K .......... .......... .......... .......... .......... 26% 41.9M 4s 2024-04-21T08:46:22.290 INFO:tasks.workunit.client.0.smithi057.stderr: 36500K .......... .......... .......... .......... .......... 26% 22.0M 4s 2024-04-21T08:46:22.291 INFO:tasks.workunit.client.0.smithi057.stderr: 36550K .......... .......... .......... .......... .......... 26% 25.8M 4s 2024-04-21T08:46:22.292 INFO:tasks.workunit.client.0.smithi057.stderr: 36600K .......... .......... .......... .......... .......... 26% 47.6M 4s 2024-04-21T08:46:22.294 INFO:tasks.workunit.client.0.smithi057.stderr: 36650K .......... .......... .......... .......... .......... 27% 23.8M 4s 2024-04-21T08:46:22.296 INFO:tasks.workunit.client.0.smithi057.stderr: 36700K .......... .......... .......... .......... .......... 27% 36.7M 4s 2024-04-21T08:46:22.298 INFO:tasks.workunit.client.0.smithi057.stderr: 36750K .......... .......... .......... .......... .......... 27% 21.1M 4s 2024-04-21T08:46:22.300 INFO:tasks.workunit.client.0.smithi057.stderr: 36800K .......... .......... .......... .......... .......... 27% 24.9M 4s 2024-04-21T08:46:22.301 INFO:tasks.workunit.client.0.smithi057.stderr: 36850K .......... .......... .......... .......... .......... 27% 48.9M 4s 2024-04-21T08:46:22.303 INFO:tasks.workunit.client.0.smithi057.stderr: 36900K .......... .......... .......... .......... .......... 27% 22.7M 4s 2024-04-21T08:46:22.305 INFO:tasks.workunit.client.0.smithi057.stderr: 36950K .......... .......... .......... .......... .......... 27% 25.8M 4s 2024-04-21T08:46:22.307 INFO:tasks.workunit.client.0.smithi057.stderr: 37000K .......... .......... .......... .......... .......... 27% 36.2M 4s 2024-04-21T08:46:22.309 INFO:tasks.workunit.client.0.smithi057.stderr: 37050K .......... .......... .......... .......... .......... 27% 24.4M 4s 2024-04-21T08:46:22.310 INFO:tasks.workunit.client.0.smithi057.stderr: 37100K .......... .......... .......... .......... .......... 27% 45.0M 4s 2024-04-21T08:46:22.312 INFO:tasks.workunit.client.0.smithi057.stderr: 37150K .......... .......... .......... .......... .......... 27% 23.0M 4s 2024-04-21T08:46:22.314 INFO:tasks.workunit.client.0.smithi057.stderr: 37200K .......... .......... .......... .......... .......... 27% 33.9M 4s 2024-04-21T08:46:22.315 INFO:tasks.workunit.client.0.smithi057.stderr: 37250K .......... .......... .......... .......... .......... 27% 24.4M 4s 2024-04-21T08:46:22.317 INFO:tasks.workunit.client.0.smithi057.stderr: 37300K .......... .......... .......... .......... .......... 27% 24.6M 4s 2024-04-21T08:46:22.318 INFO:tasks.workunit.client.0.smithi057.stderr: 37350K .......... .......... .......... .......... .......... 27% 53.1M 4s 2024-04-21T08:46:22.320 INFO:tasks.workunit.client.0.smithi057.stderr: 37400K .......... .......... .......... .......... .......... 27% 23.6M 4s 2024-04-21T08:46:22.322 INFO:tasks.workunit.client.0.smithi057.stderr: 37450K .......... .......... .......... .......... .......... 27% 40.7M 4s 2024-04-21T08:46:22.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:21 smithi129 ceph-mon[31272]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-21T08:46:22.324 INFO:tasks.workunit.client.0.smithi057.stderr: 37500K .......... .......... .......... .......... .......... 27% 22.0M 4s 2024-04-21T08:46:22.326 INFO:tasks.workunit.client.0.smithi057.stderr: 37550K .......... .......... .......... .......... .......... 27% 24.1M 4s 2024-04-21T08:46:22.326 INFO:tasks.workunit.client.0.smithi057.stderr: 37600K .......... .......... .......... .......... .......... 27% 56.4M 4s 2024-04-21T08:46:22.329 INFO:tasks.workunit.client.0.smithi057.stderr: 37650K .......... .......... .......... .......... .......... 27% 23.1M 4s 2024-04-21T08:46:22.331 INFO:tasks.workunit.client.0.smithi057.stderr: 37700K .......... .......... .......... .......... .......... 27% 24.3M 4s 2024-04-21T08:46:22.332 INFO:tasks.workunit.client.0.smithi057.stderr: 37750K .......... .......... .......... .......... .......... 27% 41.6M 4s 2024-04-21T08:46:22.334 INFO:tasks.workunit.client.0.smithi057.stderr: 37800K .......... .......... .......... .......... .......... 27% 23.4M 4s 2024-04-21T08:46:22.335 INFO:tasks.workunit.client.0.smithi057.stderr: 37850K .......... .......... .......... .......... .......... 27% 50.7M 4s 2024-04-21T08:46:22.337 INFO:tasks.workunit.client.0.smithi057.stderr: 37900K .......... .......... .......... .......... .......... 27% 24.0M 4s 2024-04-21T08:46:22.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:21 smithi057 ceph-mon[27968]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-21T08:46:22.340 INFO:tasks.workunit.client.0.smithi057.stderr: 37950K .......... .......... .......... .......... .......... 27% 27.3M 4s 2024-04-21T08:46:22.340 INFO:tasks.workunit.client.0.smithi057.stderr: 38000K .......... .......... .......... .......... .......... 28% 33.8M 4s 2024-04-21T08:46:22.342 INFO:tasks.workunit.client.0.smithi057.stderr: 38050K .......... .......... .......... .......... .......... 28% 23.4M 4s 2024-04-21T08:46:22.343 INFO:tasks.workunit.client.0.smithi057.stderr: 38100K .......... .......... .......... .......... .......... 28% 51.4M 4s 2024-04-21T08:46:22.345 INFO:tasks.workunit.client.0.smithi057.stderr: 38150K .......... .......... .......... .......... .......... 28% 24.9M 4s 2024-04-21T08:46:22.348 INFO:tasks.workunit.client.0.smithi057.stderr: 38200K .......... .......... .......... .......... .......... 28% 21.5M 4s 2024-04-21T08:46:22.349 INFO:tasks.workunit.client.0.smithi057.stderr: 38250K .......... .......... .......... .......... .......... 28% 38.2M 4s 2024-04-21T08:46:22.350 INFO:tasks.workunit.client.0.smithi057.stderr: 38300K .......... .......... .......... .......... .......... 28% 25.4M 4s 2024-04-21T08:46:22.351 INFO:tasks.workunit.client.0.smithi057.stderr: 38350K .......... .......... .......... .......... .......... 28% 58.9M 4s 2024-04-21T08:46:22.356 INFO:tasks.workunit.client.0.smithi057.stderr: 38400K .......... .......... .......... .......... .......... 28% 21.7M 4s 2024-04-21T08:46:22.356 INFO:tasks.workunit.client.0.smithi057.stderr: 38450K .......... .......... .......... .......... .......... 28% 23.6M 4s 2024-04-21T08:46:22.357 INFO:tasks.workunit.client.0.smithi057.stderr: 38500K .......... .......... .......... .......... .......... 28% 42.3M 4s 2024-04-21T08:46:22.359 INFO:tasks.workunit.client.0.smithi057.stderr: 38550K .......... .......... .......... .......... .......... 28% 22.2M 4s 2024-04-21T08:46:22.361 INFO:tasks.workunit.client.0.smithi057.stderr: 38600K .......... .......... .......... .......... .......... 28% 58.4M 4s 2024-04-21T08:46:22.362 INFO:tasks.workunit.client.0.smithi057.stderr: 38650K .......... .......... .......... .......... .......... 28% 19.5M 4s 2024-04-21T08:46:22.365 INFO:tasks.workunit.client.0.smithi057.stderr: 38700K .......... .......... .......... .......... .......... 28% 20.6M 4s 2024-04-21T08:46:22.366 INFO:tasks.workunit.client.0.smithi057.stderr: 38750K .......... .......... .......... .......... .......... 28% 39.2M 4s 2024-04-21T08:46:22.368 INFO:tasks.workunit.client.0.smithi057.stderr: 38800K .......... .......... .......... .......... .......... 28% 23.4M 4s 2024-04-21T08:46:22.370 INFO:tasks.workunit.client.0.smithi057.stderr: 38850K .......... .......... .......... .......... .......... 28% 49.1M 4s 2024-04-21T08:46:22.371 INFO:tasks.workunit.client.0.smithi057.stderr: 38900K .......... .......... .......... .......... .......... 28% 24.2M 4s 2024-04-21T08:46:22.373 INFO:tasks.workunit.client.0.smithi057.stderr: 38950K .......... .......... .......... .......... .......... 28% 23.4M 4s 2024-04-21T08:46:22.374 INFO:tasks.workunit.client.0.smithi057.stderr: 39000K .......... .......... .......... .......... .......... 28% 39.5M 4s 2024-04-21T08:46:22.377 INFO:tasks.workunit.client.0.smithi057.stderr: 39050K .......... .......... .......... .......... .......... 28% 24.1M 4s 2024-04-21T08:46:22.379 INFO:tasks.workunit.client.0.smithi057.stderr: 39100K .......... .......... .......... .......... .......... 28% 22.6M 4s 2024-04-21T08:46:22.379 INFO:tasks.workunit.client.0.smithi057.stderr: 39150K .......... .......... .......... .......... .......... 28% 59.9M 4s 2024-04-21T08:46:22.381 INFO:tasks.workunit.client.0.smithi057.stderr: 39200K .......... .......... .......... .......... .......... 28% 23.0M 4s 2024-04-21T08:46:22.383 INFO:tasks.workunit.client.0.smithi057.stderr: 39250K .......... .......... .......... .......... .......... 28% 37.9M 3s 2024-04-21T08:46:22.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:21 smithi136 ceph-mon[31791]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.6 KiB/s wr, 4 op/s 2024-04-21T08:46:22.385 INFO:tasks.workunit.client.0.smithi057.stderr: 39300K .......... .......... .......... .......... .......... 28% 22.3M 3s 2024-04-21T08:46:22.387 INFO:tasks.workunit.client.0.smithi057.stderr: 39350K .......... .......... .......... .......... .......... 29% 25.6M 3s 2024-04-21T08:46:22.388 INFO:tasks.workunit.client.0.smithi057.stderr: 39400K .......... .......... .......... .......... .......... 29% 44.6M 3s 2024-04-21T08:46:22.390 INFO:tasks.workunit.client.0.smithi057.stderr: 39450K .......... .......... .......... .......... .......... 29% 24.0M 3s 2024-04-21T08:46:22.391 INFO:tasks.workunit.client.0.smithi057.stderr: 39500K .......... .......... .......... .......... .......... 29% 42.7M 3s 2024-04-21T08:46:22.393 INFO:tasks.workunit.client.0.smithi057.stderr: 39550K .......... .......... .......... .......... .......... 29% 24.5M 3s 2024-04-21T08:46:22.395 INFO:tasks.workunit.client.0.smithi057.stderr: 39600K .......... .......... .......... .......... .......... 29% 23.5M 3s 2024-04-21T08:46:22.397 INFO:tasks.workunit.client.0.smithi057.stderr: 39650K .......... .......... .......... .......... .......... 29% 39.2M 3s 2024-04-21T08:46:22.398 INFO:tasks.workunit.client.0.smithi057.stderr: 39700K .......... .......... .......... .......... .......... 29% 24.9M 3s 2024-04-21T08:46:22.401 INFO:tasks.workunit.client.0.smithi057.stderr: 39750K .......... .......... .......... .......... .......... 29% 26.0M 3s 2024-04-21T08:46:22.402 INFO:tasks.workunit.client.0.smithi057.stderr: 39800K .......... .......... .......... .......... .......... 29% 30.7M 3s 2024-04-21T08:46:22.403 INFO:tasks.workunit.client.0.smithi057.stderr: 39850K .......... .......... .......... .......... .......... 29% 49.4M 3s 2024-04-21T08:46:22.405 INFO:tasks.workunit.client.0.smithi057.stderr: 39900K .......... .......... .......... .......... .......... 29% 28.1M 3s 2024-04-21T08:46:22.407 INFO:tasks.workunit.client.0.smithi057.stderr: 39950K .......... .......... .......... .......... .......... 29% 26.5M 3s 2024-04-21T08:46:22.408 INFO:tasks.workunit.client.0.smithi057.stderr: 40000K .......... .......... .......... .......... .......... 29% 30.6M 3s 2024-04-21T08:46:22.410 INFO:tasks.workunit.client.0.smithi057.stderr: 40050K .......... .......... .......... .......... .......... 29% 24.6M 3s 2024-04-21T08:46:22.412 INFO:tasks.workunit.client.0.smithi057.stderr: 40100K .......... .......... .......... .......... .......... 29% 23.8M 3s 2024-04-21T08:46:22.413 INFO:tasks.workunit.client.0.smithi057.stderr: 40150K .......... .......... .......... .......... .......... 29% 53.3M 3s 2024-04-21T08:46:22.415 INFO:tasks.workunit.client.0.smithi057.stderr: 40200K .......... .......... .......... .......... .......... 29% 28.8M 3s 2024-04-21T08:46:22.416 INFO:tasks.workunit.client.0.smithi057.stderr: 40250K .......... .......... .......... .......... .......... 29% 30.5M 3s 2024-04-21T08:46:22.418 INFO:tasks.workunit.client.0.smithi057.stderr: 40300K .......... .......... .......... .......... .......... 29% 30.7M 3s 2024-04-21T08:46:22.421 INFO:tasks.workunit.client.0.smithi057.stderr: 40350K .......... .......... .......... .......... .......... 29% 18.4M 3s 2024-04-21T08:46:22.422 INFO:tasks.workunit.client.0.smithi057.stderr: 40400K .......... .......... .......... .......... .......... 29% 51.1M 3s 2024-04-21T08:46:22.424 INFO:tasks.workunit.client.0.smithi057.stderr: 40450K .......... .......... .......... .......... .......... 29% 23.0M 3s 2024-04-21T08:46:22.425 INFO:tasks.workunit.client.0.smithi057.stderr: 40500K .......... .......... .......... .......... .......... 29% 35.2M 3s 2024-04-21T08:46:22.428 INFO:tasks.workunit.client.0.smithi057.stderr: 40550K .......... .......... .......... .......... .......... 29% 19.4M 3s 2024-04-21T08:46:22.430 INFO:tasks.workunit.client.0.smithi057.stderr: 40600K .......... .......... .......... .......... .......... 29% 25.3M 3s 2024-04-21T08:46:22.431 INFO:tasks.workunit.client.0.smithi057.stderr: 40650K .......... .......... .......... .......... .......... 29% 56.1M 3s 2024-04-21T08:46:22.433 INFO:tasks.workunit.client.0.smithi057.stderr: 40700K .......... .......... .......... .......... .......... 30% 23.6M 3s 2024-04-21T08:46:22.434 INFO:tasks.workunit.client.0.smithi057.stderr: 40750K .......... .......... .......... .......... .......... 30% 35.6M 3s 2024-04-21T08:46:22.436 INFO:tasks.workunit.client.0.smithi057.stderr: 40800K .......... .......... .......... .......... .......... 30% 23.6M 3s 2024-04-21T08:46:22.438 INFO:tasks.workunit.client.0.smithi057.stderr: 40850K .......... .......... .......... .......... .......... 30% 20.1M 3s 2024-04-21T08:46:22.439 INFO:tasks.workunit.client.0.smithi057.stderr: 40900K .......... .......... .......... .......... .......... 30% 46.9M 3s 2024-04-21T08:46:22.441 INFO:tasks.workunit.client.0.smithi057.stderr: 40950K .......... .......... .......... .......... .......... 30% 24.2M 3s 2024-04-21T08:46:22.443 INFO:tasks.workunit.client.0.smithi057.stderr: 41000K .......... .......... .......... .......... .......... 30% 41.0M 3s 2024-04-21T08:46:22.445 INFO:tasks.workunit.client.0.smithi057.stderr: 41050K .......... .......... .......... .......... .......... 30% 21.6M 3s 2024-04-21T08:46:22.447 INFO:tasks.workunit.client.0.smithi057.stderr: 41100K .......... .......... .......... .......... .......... 30% 25.2M 3s 2024-04-21T08:46:22.448 INFO:tasks.workunit.client.0.smithi057.stderr: 41150K .......... .......... .......... .......... .......... 30% 55.5M 3s 2024-04-21T08:46:22.450 INFO:tasks.workunit.client.0.smithi057.stderr: 41200K .......... .......... .......... .......... .......... 30% 22.4M 3s 2024-04-21T08:46:22.452 INFO:tasks.workunit.client.0.smithi057.stderr: 41250K .......... .......... .......... .......... .......... 30% 38.3M 3s 2024-04-21T08:46:22.453 INFO:tasks.workunit.client.0.smithi057.stderr: 41300K .......... .......... .......... .......... .......... 30% 25.8M 3s 2024-04-21T08:46:22.456 INFO:tasks.workunit.client.0.smithi057.stderr: 41350K .......... .......... .......... .......... .......... 30% 24.3M 3s 2024-04-21T08:46:22.456 INFO:tasks.workunit.client.0.smithi057.stderr: 41400K .......... .......... .......... .......... .......... 30% 48.2M 3s 2024-04-21T08:46:22.458 INFO:tasks.workunit.client.0.smithi057.stderr: 41450K .......... .......... .......... .......... .......... 30% 21.3M 3s 2024-04-21T08:46:22.459 INFO:tasks.workunit.client.0.smithi057.stderr: 41500K .......... .......... .......... .......... .......... 30% 44.8M 3s 2024-04-21T08:46:22.462 INFO:tasks.workunit.client.0.smithi057.stderr: 41550K .......... .......... .......... .......... .......... 30% 19.1M 3s 2024-04-21T08:46:22.464 INFO:tasks.workunit.client.0.smithi057.stderr: 41600K .......... .......... .......... .......... .......... 30% 24.0M 3s 2024-04-21T08:46:22.465 INFO:tasks.workunit.client.0.smithi057.stderr: 41650K .......... .......... .......... .......... .......... 30% 44.2M 3s 2024-04-21T08:46:22.467 INFO:tasks.workunit.client.0.smithi057.stderr: 41700K .......... .......... .......... .......... .......... 30% 23.7M 3s 2024-04-21T08:46:22.468 INFO:tasks.workunit.client.0.smithi057.stderr: 41750K .......... .......... .......... .......... .......... 30% 41.9M 3s 2024-04-21T08:46:22.470 INFO:tasks.workunit.client.0.smithi057.stderr: 41800K .......... .......... .......... .......... .......... 30% 24.3M 3s 2024-04-21T08:46:22.472 INFO:tasks.workunit.client.0.smithi057.stderr: 41850K .......... .......... .......... .......... .......... 30% 23.4M 3s 2024-04-21T08:46:22.473 INFO:tasks.workunit.client.0.smithi057.stderr: 41900K .......... .......... .......... .......... .......... 30% 46.7M 3s 2024-04-21T08:46:22.475 INFO:tasks.workunit.client.0.smithi057.stderr: 41950K .......... .......... .......... .......... .......... 30% 27.3M 3s 2024-04-21T08:46:22.477 INFO:tasks.workunit.client.0.smithi057.stderr: 42000K .......... .......... .......... .......... .......... 30% 32.8M 3s 2024-04-21T08:46:22.479 INFO:tasks.workunit.client.0.smithi057.stderr: 42050K .......... .......... .......... .......... .......... 31% 22.2M 3s 2024-04-21T08:46:22.481 INFO:tasks.workunit.client.0.smithi057.stderr: 42100K .......... .......... .......... .......... .......... 31% 25.0M 3s 2024-04-21T08:46:22.482 INFO:tasks.workunit.client.0.smithi057.stderr: 42150K .......... .......... .......... .......... .......... 31% 47.4M 3s 2024-04-21T08:46:22.484 INFO:tasks.workunit.client.0.smithi057.stderr: 42200K .......... .......... .......... .......... .......... 31% 23.6M 3s 2024-04-21T08:46:22.486 INFO:tasks.workunit.client.0.smithi057.stderr: 42250K .......... .......... .......... .......... .......... 31% 25.1M 3s 2024-04-21T08:46:22.487 INFO:tasks.workunit.client.0.smithi057.stderr: 42300K .......... .......... .......... .......... .......... 31% 40.0M 3s 2024-04-21T08:46:22.489 INFO:tasks.workunit.client.0.smithi057.stderr: 42350K .......... .......... .......... .......... .......... 31% 23.3M 3s 2024-04-21T08:46:22.490 INFO:tasks.workunit.client.0.smithi057.stderr: 42400K .......... .......... .......... .......... .......... 31% 45.1M 3s 2024-04-21T08:46:22.492 INFO:tasks.workunit.client.0.smithi057.stderr: 42450K .......... .......... .......... .......... .......... 31% 24.0M 3s 2024-04-21T08:46:22.494 INFO:tasks.workunit.client.0.smithi057.stderr: 42500K .......... .......... .......... .......... .......... 31% 37.8M 3s 2024-04-21T08:46:22.495 INFO:tasks.workunit.client.0.smithi057.stderr: 42550K .......... .......... .......... .......... .......... 31% 25.5M 3s 2024-04-21T08:46:22.498 INFO:tasks.workunit.client.0.smithi057.stderr: 42600K .......... .......... .......... .......... .......... 31% 23.4M 3s 2024-04-21T08:46:22.499 INFO:tasks.workunit.client.0.smithi057.stderr: 42650K .......... .......... .......... .......... .......... 31% 47.9M 3s 2024-04-21T08:46:22.501 INFO:tasks.workunit.client.0.smithi057.stderr: 42700K .......... .......... .......... .......... .......... 31% 23.5M 3s 2024-04-21T08:46:22.503 INFO:tasks.workunit.client.0.smithi057.stderr: 42750K .......... .......... .......... .......... .......... 31% 40.9M 3s 2024-04-21T08:46:22.505 INFO:tasks.workunit.client.0.smithi057.stderr: 42800K .......... .......... .......... .......... .......... 31% 32.2M 3s 2024-04-21T08:46:22.506 INFO:tasks.workunit.client.0.smithi057.stderr: 42850K .......... .......... .......... .......... .......... 31% 18.3M 3s 2024-04-21T08:46:22.507 INFO:tasks.workunit.client.0.smithi057.stderr: 42900K .......... .......... .......... .......... .......... 31% 44.7M 3s 2024-04-21T08:46:22.509 INFO:tasks.workunit.client.0.smithi057.stderr: 42950K .......... .......... .......... .......... .......... 31% 24.9M 3s 2024-04-21T08:46:22.511 INFO:tasks.workunit.client.0.smithi057.stderr: 43000K .......... .......... .......... .......... .......... 31% 23.3M 3s 2024-04-21T08:46:22.513 INFO:tasks.workunit.client.0.smithi057.stderr: 43050K .......... .......... .......... .......... .......... 31% 39.0M 3s 2024-04-21T08:46:22.514 INFO:tasks.workunit.client.0.smithi057.stderr: 43100K .......... .......... .......... .......... .......... 31% 27.2M 3s 2024-04-21T08:46:22.515 INFO:tasks.workunit.client.0.smithi057.stderr: 43150K .......... .......... .......... .......... .......... 31% 43.4M 3s 2024-04-21T08:46:22.518 INFO:tasks.workunit.client.0.smithi057.stderr: 43200K .......... .......... .......... .......... .......... 31% 22.0M 3s 2024-04-21T08:46:22.520 INFO:tasks.workunit.client.0.smithi057.stderr: 43250K .......... .......... .......... .......... .......... 31% 25.3M 3s 2024-04-21T08:46:22.521 INFO:tasks.workunit.client.0.smithi057.stderr: 43300K .......... .......... .......... .......... .......... 31% 42.7M 3s 2024-04-21T08:46:22.523 INFO:tasks.workunit.client.0.smithi057.stderr: 43350K .......... .......... .......... .......... .......... 31% 20.5M 3s 2024-04-21T08:46:22.524 INFO:tasks.workunit.client.0.smithi057.stderr: 43400K .......... .......... .......... .......... .......... 32% 59.7M 3s 2024-04-21T08:46:22.526 INFO:tasks.workunit.client.0.smithi057.stderr: 43450K .......... .......... .......... .......... .......... 32% 22.2M 3s 2024-04-21T08:46:22.528 INFO:tasks.workunit.client.0.smithi057.stderr: 43500K .......... .......... .......... .......... .......... 32% 23.9M 3s 2024-04-21T08:46:22.530 INFO:tasks.workunit.client.0.smithi057.stderr: 43550K .......... .......... .......... .......... .......... 32% 27.3M 3s 2024-04-21T08:46:22.532 INFO:tasks.workunit.client.0.smithi057.stderr: 43600K .......... .......... .......... .......... .......... 32% 23.9M 3s 2024-04-21T08:46:22.533 INFO:tasks.workunit.client.0.smithi057.stderr: 43650K .......... .......... .......... .......... .......... 32% 56.8M 3s 2024-04-21T08:46:22.535 INFO:tasks.workunit.client.0.smithi057.stderr: 43700K .......... .......... .......... .......... .......... 32% 23.5M 3s 2024-04-21T08:46:22.537 INFO:tasks.workunit.client.0.smithi057.stderr: 43750K .......... .......... .......... .......... .......... 32% 23.2M 3s 2024-04-21T08:46:22.538 INFO:tasks.workunit.client.0.smithi057.stderr: 43800K .......... .......... .......... .......... .......... 32% 38.7M 3s 2024-04-21T08:46:22.541 INFO:tasks.workunit.client.0.smithi057.stderr: 43850K .......... .......... .......... .......... .......... 32% 21.9M 3s 2024-04-21T08:46:22.543 INFO:tasks.workunit.client.0.smithi057.stderr: 43900K .......... .......... .......... .......... .......... 32% 56.6M 3s 2024-04-21T08:46:22.544 INFO:tasks.workunit.client.0.smithi057.stderr: 43950K .......... .......... .......... .......... .......... 32% 24.4M 3s 2024-04-21T08:46:22.546 INFO:tasks.workunit.client.0.smithi057.stderr: 44000K .......... .......... .......... .......... .......... 32% 23.2M 3s 2024-04-21T08:46:22.547 INFO:tasks.workunit.client.0.smithi057.stderr: 44050K .......... .......... .......... .......... .......... 32% 41.6M 3s 2024-04-21T08:46:22.549 INFO:tasks.workunit.client.0.smithi057.stderr: 44100K .......... .......... .......... .......... .......... 32% 22.4M 3s 2024-04-21T08:46:22.551 INFO:tasks.workunit.client.0.smithi057.stderr: 44150K .......... .......... .......... .......... .......... 32% 54.9M 3s 2024-04-21T08:46:22.552 INFO:tasks.workunit.client.0.smithi057.stderr: 44200K .......... .......... .......... .......... .......... 32% 22.8M 3s 2024-04-21T08:46:22.554 INFO:tasks.workunit.client.0.smithi057.stderr: 44250K .......... .......... .......... .......... .......... 32% 23.9M 3s 2024-04-21T08:46:22.555 INFO:tasks.workunit.client.0.smithi057.stderr: 44300K .......... .......... .......... .......... .......... 32% 41.4M 3s 2024-04-21T08:46:22.557 INFO:tasks.workunit.client.0.smithi057.stderr: 44350K .......... .......... .......... .......... .......... 32% 23.6M 3s 2024-04-21T08:46:22.559 INFO:tasks.workunit.client.0.smithi057.stderr: 44400K .......... .......... .......... .......... .......... 32% 23.4M 3s 2024-04-21T08:46:22.560 INFO:tasks.workunit.client.0.smithi057.stderr: 44450K .......... .......... .......... .......... .......... 32% 51.2M 3s 2024-04-21T08:46:22.562 INFO:tasks.workunit.client.0.smithi057.stderr: 44500K .......... .......... .......... .......... .......... 32% 23.5M 3s 2024-04-21T08:46:22.563 INFO:tasks.workunit.client.0.smithi057.stderr: 44550K .......... .......... .......... .......... .......... 32% 40.8M 3s 2024-04-21T08:46:22.565 INFO:tasks.workunit.client.0.smithi057.stderr: 44600K .......... .......... .......... .......... .......... 32% 24.8M 3s 2024-04-21T08:46:22.568 INFO:tasks.workunit.client.0.smithi057.stderr: 44650K .......... .......... .......... .......... .......... 32% 21.9M 3s 2024-04-21T08:46:22.569 INFO:tasks.workunit.client.0.smithi057.stderr: 44700K .......... .......... .......... .......... .......... 32% 43.8M 3s 2024-04-21T08:46:22.571 INFO:tasks.workunit.client.0.smithi057.stderr: 44750K .......... .......... .......... .......... .......... 33% 25.8M 3s 2024-04-21T08:46:22.572 INFO:tasks.workunit.client.0.smithi057.stderr: 44800K .......... .......... .......... .......... .......... 33% 39.0M 3s 2024-04-21T08:46:22.574 INFO:tasks.workunit.client.0.smithi057.stderr: 44850K .......... .......... .......... .......... .......... 33% 24.2M 3s 2024-04-21T08:46:22.576 INFO:tasks.workunit.client.0.smithi057.stderr: 44900K .......... .......... .......... .......... .......... 33% 22.4M 3s 2024-04-21T08:46:22.577 INFO:tasks.workunit.client.0.smithi057.stderr: 44950K .......... .......... .......... .......... .......... 33% 56.4M 3s 2024-04-21T08:46:22.579 INFO:tasks.workunit.client.0.smithi057.stderr: 45000K .......... .......... .......... .......... .......... 33% 22.7M 3s 2024-04-21T08:46:22.580 INFO:tasks.workunit.client.0.smithi057.stderr: 45050K .......... .......... .......... .......... .......... 33% 42.7M 3s 2024-04-21T08:46:22.582 INFO:tasks.workunit.client.0.smithi057.stderr: 45100K .......... .......... .......... .......... .......... 33% 23.6M 3s 2024-04-21T08:46:22.585 INFO:tasks.workunit.client.0.smithi057.stderr: 45150K .......... .......... .......... .......... .......... 33% 21.2M 3s 2024-04-21T08:46:22.586 INFO:tasks.workunit.client.0.smithi057.stderr: 45200K .......... .......... .......... .......... .......... 33% 51.3M 3s 2024-04-21T08:46:22.588 INFO:tasks.workunit.client.0.smithi057.stderr: 45250K .......... .......... .......... .......... .......... 33% 22.6M 3s 2024-04-21T08:46:22.589 INFO:tasks.workunit.client.0.smithi057.stderr: 45300K .......... .......... .......... .......... .......... 33% 36.1M 3s 2024-04-21T08:46:22.591 INFO:tasks.workunit.client.0.smithi057.stderr: 45350K .......... .......... .......... .......... .......... 33% 25.0M 3s 2024-04-21T08:46:22.593 INFO:tasks.workunit.client.0.smithi057.stderr: 45400K .......... .......... .......... .......... .......... 33% 23.5M 3s 2024-04-21T08:46:22.594 INFO:tasks.workunit.client.0.smithi057.stderr: 45450K .......... .......... .......... .......... .......... 33% 50.5M 3s 2024-04-21T08:46:22.596 INFO:tasks.workunit.client.0.smithi057.stderr: 45500K .......... .......... .......... .......... .......... 33% 24.0M 3s 2024-04-21T08:46:22.597 INFO:tasks.workunit.client.0.smithi057.stderr: 45550K .......... .......... .......... .......... .......... 33% 39.5M 3s 2024-04-21T08:46:22.600 INFO:tasks.workunit.client.0.smithi057.stderr: 45600K .......... .......... .......... .......... .......... 33% 22.4M 3s 2024-04-21T08:46:22.602 INFO:tasks.workunit.client.0.smithi057.stderr: 45650K .......... .......... .......... .......... .......... 33% 24.2M 3s 2024-04-21T08:46:22.602 INFO:tasks.workunit.client.0.smithi057.stderr: 45700K .......... .......... .......... .......... .......... 33% 54.9M 3s 2024-04-21T08:46:22.605 INFO:tasks.workunit.client.0.smithi057.stderr: 45750K .......... .......... .......... .......... .......... 33% 23.8M 3s 2024-04-21T08:46:22.607 INFO:tasks.workunit.client.0.smithi057.stderr: 45800K .......... .......... .......... .......... .......... 33% 22.3M 3s 2024-04-21T08:46:22.608 INFO:tasks.workunit.client.0.smithi057.stderr: 45850K .......... .......... .......... .......... .......... 33% 43.9M 3s 2024-04-21T08:46:22.610 INFO:tasks.workunit.client.0.smithi057.stderr: 45900K .......... .......... .......... .......... .......... 33% 23.6M 3s 2024-04-21T08:46:22.611 INFO:tasks.workunit.client.0.smithi057.stderr: 45950K .......... .......... .......... .......... .......... 33% 50.5M 3s 2024-04-21T08:46:22.613 INFO:tasks.workunit.client.0.smithi057.stderr: 46000K .......... .......... .......... .......... .......... 33% 24.0M 3s 2024-04-21T08:46:22.614 INFO:tasks.workunit.client.0.smithi057.stderr: 46050K .......... .......... .......... .......... .......... 33% 41.3M 3s 2024-04-21T08:46:22.616 INFO:tasks.workunit.client.0.smithi057.stderr: 46100K .......... .......... .......... .......... .......... 33% 23.1M 3s 2024-04-21T08:46:22.619 INFO:tasks.workunit.client.0.smithi057.stderr: 46150K .......... .......... .......... .......... .......... 34% 23.1M 3s 2024-04-21T08:46:22.619 INFO:tasks.workunit.client.0.smithi057.stderr: 46200K .......... .......... .......... .......... .......... 34% 51.1M 3s 2024-04-21T08:46:22.621 INFO:tasks.workunit.client.0.smithi057.stderr: 46250K .......... .......... .......... .......... .......... 34% 31.8M 3s 2024-04-21T08:46:22.623 INFO:tasks.workunit.client.0.smithi057.stderr: 46300K .......... .......... .......... .......... .......... 34% 25.9M 3s 2024-04-21T08:46:22.625 INFO:tasks.workunit.client.0.smithi057.stderr: 46350K .......... .......... .......... .......... .......... 34% 25.2M 3s 2024-04-21T08:46:22.627 INFO:tasks.workunit.client.0.smithi057.stderr: 46400K .......... .......... .......... .......... .......... 34% 24.6M 3s 2024-04-21T08:46:22.628 INFO:tasks.workunit.client.0.smithi057.stderr: 46450K .......... .......... .......... .......... .......... 34% 45.4M 3s 2024-04-21T08:46:22.630 INFO:tasks.workunit.client.0.smithi057.stderr: 46500K .......... .......... .......... .......... .......... 34% 23.4M 3s 2024-04-21T08:46:22.631 INFO:tasks.workunit.client.0.smithi057.stderr: 46550K .......... .......... .......... .......... .......... 34% 39.8M 3s 2024-04-21T08:46:22.633 INFO:tasks.workunit.client.0.smithi057.stderr: 46600K .......... .......... .......... .......... .......... 34% 24.3M 3s 2024-04-21T08:46:22.636 INFO:tasks.workunit.client.0.smithi057.stderr: 46650K .......... .......... .......... .......... .......... 34% 23.8M 3s 2024-04-21T08:46:22.636 INFO:tasks.workunit.client.0.smithi057.stderr: 46700K .......... .......... .......... .......... .......... 34% 54.6M 3s 2024-04-21T08:46:22.638 INFO:tasks.workunit.client.0.smithi057.stderr: 46750K .......... .......... .......... .......... .......... 34% 29.6M 3s 2024-04-21T08:46:22.639 INFO:tasks.workunit.client.0.smithi057.stderr: 46800K .......... .......... .......... .......... .......... 34% 28.9M 3s 2024-04-21T08:46:22.642 INFO:tasks.workunit.client.0.smithi057.stderr: 46850K .......... .......... .......... .......... .......... 34% 21.9M 3s 2024-04-21T08:46:22.644 INFO:tasks.workunit.client.0.smithi057.stderr: 46900K .......... .......... .......... .......... .......... 34% 25.3M 3s 2024-04-21T08:46:22.645 INFO:tasks.workunit.client.0.smithi057.stderr: 46950K .......... .......... .......... .......... .......... 34% 46.8M 3s 2024-04-21T08:46:22.647 INFO:tasks.workunit.client.0.smithi057.stderr: 47000K .......... .......... .......... .......... .......... 34% 22.5M 3s 2024-04-21T08:46:22.648 INFO:tasks.workunit.client.0.smithi057.stderr: 47050K .......... .......... .......... .......... .......... 34% 31.2M 3s 2024-04-21T08:46:22.651 INFO:tasks.workunit.client.0.smithi057.stderr: 47100K .......... .......... .......... .......... .......... 34% 20.6M 3s 2024-04-21T08:46:22.654 INFO:tasks.workunit.client.0.smithi057.stderr: 47150K .......... .......... .......... .......... .......... 34% 24.3M 3s 2024-04-21T08:46:22.654 INFO:tasks.workunit.client.0.smithi057.stderr: 47200K .......... .......... .......... .......... .......... 34% 61.5M 3s 2024-04-21T08:46:22.656 INFO:tasks.workunit.client.0.smithi057.stderr: 47250K .......... .......... .......... .......... .......... 34% 22.1M 3s 2024-04-21T08:46:22.657 INFO:tasks.workunit.client.0.smithi057.stderr: 47300K .......... .......... .......... .......... .......... 34% 42.1M 3s 2024-04-21T08:46:22.659 INFO:tasks.workunit.client.0.smithi057.stderr: 47350K .......... .......... .......... .......... .......... 34% 23.5M 3s 2024-04-21T08:46:22.661 INFO:tasks.workunit.client.0.smithi057.stderr: 47400K .......... .......... .......... .......... .......... 34% 23.3M 3s 2024-04-21T08:46:22.662 INFO:tasks.workunit.client.0.smithi057.stderr: 47450K .......... .......... .......... .......... .......... 34% 53.9M 3s 2024-04-21T08:46:22.664 INFO:tasks.workunit.client.0.smithi057.stderr: 47500K .......... .......... .......... .......... .......... 35% 23.2M 3s 2024-04-21T08:46:22.665 INFO:tasks.workunit.client.0.smithi057.stderr: 47550K .......... .......... .......... .......... .......... 35% 41.9M 3s 2024-04-21T08:46:22.667 INFO:tasks.workunit.client.0.smithi057.stderr: 47600K .......... .......... .......... .......... .......... 35% 23.9M 3s 2024-04-21T08:46:22.669 INFO:tasks.workunit.client.0.smithi057.stderr: 47650K .......... .......... .......... .......... .......... 35% 22.8M 3s 2024-04-21T08:46:22.670 INFO:tasks.workunit.client.0.smithi057.stderr: 47700K .......... .......... .......... .......... .......... 35% 53.1M 3s 2024-04-21T08:46:22.672 INFO:tasks.workunit.client.0.smithi057.stderr: 47750K .......... .......... .......... .......... .......... 35% 22.4M 3s 2024-04-21T08:46:22.674 INFO:tasks.workunit.client.0.smithi057.stderr: 47800K .......... .......... .......... .......... .......... 35% 37.7M 3s 2024-04-21T08:46:22.677 INFO:tasks.workunit.client.0.smithi057.stderr: 47850K .......... .......... .......... .......... .......... 35% 26.7M 3s 2024-04-21T08:46:22.678 INFO:tasks.workunit.client.0.smithi057.stderr: 47900K .......... .......... .......... .......... .......... 35% 22.5M 3s 2024-04-21T08:46:22.679 INFO:tasks.workunit.client.0.smithi057.stderr: 47950K .......... .......... .......... .......... .......... 35% 54.4M 3s 2024-04-21T08:46:22.681 INFO:tasks.workunit.client.0.smithi057.stderr: 48000K .......... .......... .......... .......... .......... 35% 23.4M 3s 2024-04-21T08:46:22.683 INFO:tasks.workunit.client.0.smithi057.stderr: 48050K .......... .......... .......... .......... .......... 35% 23.4M 3s 2024-04-21T08:46:22.685 INFO:tasks.workunit.client.0.smithi057.stderr: 48100K .......... .......... .......... .......... .......... 35% 37.0M 3s 2024-04-21T08:46:22.686 INFO:tasks.workunit.client.0.smithi057.stderr: 48150K .......... .......... .......... .......... .......... 35% 25.5M 3s 2024-04-21T08:46:22.687 INFO:tasks.workunit.client.0.smithi057.stderr: 48200K .......... .......... .......... .......... .......... 35% 52.6M 3s 2024-04-21T08:46:22.689 INFO:tasks.workunit.client.0.smithi057.stderr: 48250K .......... .......... .......... .......... .......... 35% 22.4M 3s 2024-04-21T08:46:22.691 INFO:tasks.workunit.client.0.smithi057.stderr: 48300K .......... .......... .......... .......... .......... 35% 23.7M 3s 2024-04-21T08:46:22.693 INFO:tasks.workunit.client.0.smithi057.stderr: 48350K .......... .......... .......... .......... .......... 35% 35.6M 3s 2024-04-21T08:46:22.695 INFO:tasks.workunit.client.0.smithi057.stderr: 48400K .......... .......... .......... .......... .......... 35% 23.9M 3s 2024-04-21T08:46:22.696 INFO:tasks.workunit.client.0.smithi057.stderr: 48450K .......... .......... .......... .......... .......... 35% 59.9M 3s 2024-04-21T08:46:22.698 INFO:tasks.workunit.client.0.smithi057.stderr: 48500K .......... .......... .......... .......... .......... 35% 23.5M 3s 2024-04-21T08:46:22.700 INFO:tasks.workunit.client.0.smithi057.stderr: 48550K .......... .......... .......... .......... .......... 35% 23.0M 3s 2024-04-21T08:46:22.701 INFO:tasks.workunit.client.0.smithi057.stderr: 48600K .......... .......... .......... .......... .......... 35% 40.0M 3s 2024-04-21T08:46:22.703 INFO:tasks.workunit.client.0.smithi057.stderr: 48650K .......... .......... .......... .......... .......... 35% 23.3M 3s 2024-04-21T08:46:22.704 INFO:tasks.workunit.client.0.smithi057.stderr: 48700K .......... .......... .......... .......... .......... 35% 46.1M 3s 2024-04-21T08:46:22.706 INFO:tasks.workunit.client.0.smithi057.stderr: 48750K .......... .......... .......... .......... .......... 35% 24.8M 3s 2024-04-21T08:46:22.708 INFO:tasks.workunit.client.0.smithi057.stderr: 48800K .......... .......... .......... .......... .......... 35% 23.1M 3s 2024-04-21T08:46:22.709 INFO:tasks.workunit.client.0.smithi057.stderr: 48850K .......... .......... .......... .......... .......... 36% 40.2M 3s 2024-04-21T08:46:22.711 INFO:tasks.workunit.client.0.smithi057.stderr: 48900K .......... .......... .......... .......... .......... 36% 24.1M 3s 2024-04-21T08:46:22.713 INFO:tasks.workunit.client.0.smithi057.stderr: 48950K .......... .......... .......... .......... .......... 36% 32.9M 3s 2024-04-21T08:46:22.715 INFO:tasks.workunit.client.0.smithi057.stderr: 49000K .......... .......... .......... .......... .......... 36% 23.1M 3s 2024-04-21T08:46:22.717 INFO:tasks.workunit.client.0.smithi057.stderr: 49050K .......... .......... .......... .......... .......... 36% 24.2M 3s 2024-04-21T08:46:22.718 INFO:tasks.workunit.client.0.smithi057.stderr: 49100K .......... .......... .......... .......... .......... 36% 41.4M 3s 2024-04-21T08:46:22.720 INFO:tasks.workunit.client.0.smithi057.stderr: 49150K .......... .......... .......... .......... .......... 36% 23.9M 3s 2024-04-21T08:46:22.722 INFO:tasks.workunit.client.0.smithi057.stderr: 49200K .......... .......... .......... .......... .......... 36% 45.2M 3s 2024-04-21T08:46:22.723 INFO:tasks.workunit.client.0.smithi057.stderr: 49250K .......... .......... .......... .......... .......... 36% 22.9M 3s 2024-04-21T08:46:22.726 INFO:tasks.workunit.client.0.smithi057.stderr: 49300K .......... .......... .......... .......... .......... 36% 20.8M 3s 2024-04-21T08:46:22.727 INFO:tasks.workunit.client.0.smithi057.stderr: 49350K .......... .......... .......... .......... .......... 36% 34.5M 3s 2024-04-21T08:46:22.729 INFO:tasks.workunit.client.0.smithi057.stderr: 49400K .......... .......... .......... .......... .......... 36% 26.0M 3s 2024-04-21T08:46:22.731 INFO:tasks.workunit.client.0.smithi057.stderr: 49450K .......... .......... .......... .......... .......... 36% 49.2M 3s 2024-04-21T08:46:22.732 INFO:tasks.workunit.client.0.smithi057.stderr: 49500K .......... .......... .......... .......... .......... 36% 23.2M 3s 2024-04-21T08:46:22.734 INFO:tasks.workunit.client.0.smithi057.stderr: 49550K .......... .......... .......... .......... .......... 36% 23.3M 3s 2024-04-21T08:46:22.735 INFO:tasks.workunit.client.0.smithi057.stderr: 49600K .......... .......... .......... .......... .......... 36% 41.8M 3s 2024-04-21T08:46:22.737 INFO:tasks.workunit.client.0.smithi057.stderr: 49650K .......... .......... .......... .......... .......... 36% 22.8M 3s 2024-04-21T08:46:22.740 INFO:tasks.workunit.client.0.smithi057.stderr: 49700K .......... .......... .......... .......... .......... 36% 24.5M 3s 2024-04-21T08:46:22.740 INFO:tasks.workunit.client.0.smithi057.stderr: 49750K .......... .......... .......... .......... .......... 36% 53.2M 3s 2024-04-21T08:46:22.743 INFO:tasks.workunit.client.0.smithi057.stderr: 49800K .......... .......... .......... .......... .......... 36% 22.4M 3s 2024-04-21T08:46:22.744 INFO:tasks.workunit.client.0.smithi057.stderr: 49850K .......... .......... .......... .......... .......... 36% 39.3M 3s 2024-04-21T08:46:22.746 INFO:tasks.workunit.client.0.smithi057.stderr: 49900K .......... .......... .......... .......... .......... 36% 24.5M 3s 2024-04-21T08:46:22.748 INFO:tasks.workunit.client.0.smithi057.stderr: 49950K .......... .......... .......... .......... .......... 36% 28.0M 3s 2024-04-21T08:46:22.749 INFO:tasks.workunit.client.0.smithi057.stderr: 50000K .......... .......... .......... .......... .......... 36% 38.4M 3s 2024-04-21T08:46:22.751 INFO:tasks.workunit.client.0.smithi057.stderr: 50050K .......... .......... .......... .......... .......... 36% 23.7M 3s 2024-04-21T08:46:22.753 INFO:tasks.workunit.client.0.smithi057.stderr: 50100K .......... .......... .......... .......... .......... 36% 30.4M 3s 2024-04-21T08:46:22.754 INFO:tasks.workunit.client.0.smithi057.stderr: 50150K .......... .......... .......... .......... .......... 36% 25.5M 3s 2024-04-21T08:46:22.757 INFO:tasks.workunit.client.0.smithi057.stderr: 50200K .......... .......... .......... .......... .......... 37% 23.7M 3s 2024-04-21T08:46:22.758 INFO:tasks.workunit.client.0.smithi057.stderr: 50250K .......... .......... .......... .......... .......... 37% 38.2M 3s 2024-04-21T08:46:22.760 INFO:tasks.workunit.client.0.smithi057.stderr: 50300K .......... .......... .......... .......... .......... 37% 23.8M 3s 2024-04-21T08:46:22.761 INFO:tasks.workunit.client.0.smithi057.stderr: 50350K .......... .......... .......... .......... .......... 37% 38.3M 3s 2024-04-21T08:46:22.763 INFO:tasks.workunit.client.0.smithi057.stderr: 50400K .......... .......... .......... .......... .......... 37% 24.9M 3s 2024-04-21T08:46:22.764 INFO:tasks.workunit.client.0.smithi057.stderr: 50450K .......... .......... .......... .......... .......... 37% 34.0M 3s 2024-04-21T08:46:22.766 INFO:tasks.workunit.client.0.smithi057.stderr: 50500K .......... .......... .......... .......... .......... 37% 30.3M 3s 2024-04-21T08:46:22.768 INFO:tasks.workunit.client.0.smithi057.stderr: 50550K .......... .......... .......... .......... .......... 37% 22.8M 3s 2024-04-21T08:46:22.770 INFO:tasks.workunit.client.0.smithi057.stderr: 50600K .......... .......... .......... .......... .......... 37% 36.9M 3s 2024-04-21T08:46:22.771 INFO:tasks.workunit.client.0.smithi057.stderr: 50650K .......... .......... .......... .......... .......... 37% 33.9M 3s 2024-04-21T08:46:22.773 INFO:tasks.workunit.client.0.smithi057.stderr: 50700K .......... .......... .......... .......... .......... 37% 28.8M 3s 2024-04-21T08:46:22.774 INFO:tasks.workunit.client.0.smithi057.stderr: 50750K .......... .......... .......... .......... .......... 37% 27.1M 3s 2024-04-21T08:46:22.777 INFO:tasks.workunit.client.0.smithi057.stderr: 50800K .......... .......... .......... .......... .......... 37% 23.5M 3s 2024-04-21T08:46:22.778 INFO:tasks.workunit.client.0.smithi057.stderr: 50850K .......... .......... .......... .......... .......... 37% 37.5M 3s 2024-04-21T08:46:22.780 INFO:tasks.workunit.client.0.smithi057.stderr: 50900K .......... .......... .......... .......... .......... 37% 29.8M 3s 2024-04-21T08:46:22.782 INFO:tasks.workunit.client.0.smithi057.stderr: 50950K .......... .......... .......... .......... .......... 37% 20.3M 3s 2024-04-21T08:46:22.783 INFO:tasks.workunit.client.0.smithi057.stderr: 51000K .......... .......... .......... .......... .......... 37% 47.9M 3s 2024-04-21T08:46:22.785 INFO:tasks.workunit.client.0.smithi057.stderr: 51050K .......... .......... .......... .......... .......... 37% 24.1M 3s 2024-04-21T08:46:22.786 INFO:tasks.workunit.client.0.smithi057.stderr: 51100K .......... .......... .......... .......... .......... 37% 41.8M 3s 2024-04-21T08:46:22.789 INFO:tasks.workunit.client.0.smithi057.stderr: 51150K .......... .......... .......... .......... .......... 37% 24.0M 3s 2024-04-21T08:46:22.791 INFO:tasks.workunit.client.0.smithi057.stderr: 51200K .......... .......... .......... .......... .......... 37% 20.5M 3s 2024-04-21T08:46:22.791 INFO:tasks.workunit.client.0.smithi057.stderr: 51250K .......... .......... .......... .......... .......... 37% 51.7M 3s 2024-04-21T08:46:22.793 INFO:tasks.workunit.client.0.smithi057.stderr: 51300K .......... .......... .......... .......... .......... 37% 24.7M 3s 2024-04-21T08:46:22.795 INFO:tasks.workunit.client.0.smithi057.stderr: 51350K .......... .......... .......... .......... .......... 37% 35.4M 3s 2024-04-21T08:46:22.797 INFO:tasks.workunit.client.0.smithi057.stderr: 51400K .......... .......... .......... .......... .......... 37% 26.3M 3s 2024-04-21T08:46:22.799 INFO:tasks.workunit.client.0.smithi057.stderr: 51450K .......... .......... .......... .......... .......... 37% 22.5M 3s 2024-04-21T08:46:22.800 INFO:tasks.workunit.client.0.smithi057.stderr: 51500K .......... .......... .......... .......... .......... 37% 50.1M 3s 2024-04-21T08:46:22.802 INFO:tasks.workunit.client.0.smithi057.stderr: 51550K .......... .......... .......... .......... .......... 38% 24.0M 3s 2024-04-21T08:46:22.803 INFO:tasks.workunit.client.0.smithi057.stderr: 51600K .......... .......... .......... .......... .......... 38% 41.8M 3s 2024-04-21T08:46:22.805 INFO:tasks.workunit.client.0.smithi057.stderr: 51650K .......... .......... .......... .......... .......... 38% 22.8M 3s 2024-04-21T08:46:22.807 INFO:tasks.workunit.client.0.smithi057.stderr: 51700K .......... .......... .......... .......... .......... 38% 23.8M 3s 2024-04-21T08:46:22.808 INFO:tasks.workunit.client.0.smithi057.stderr: 51750K .......... .......... .......... .......... .......... 38% 45.9M 3s 2024-04-21T08:46:22.810 INFO:tasks.workunit.client.0.smithi057.stderr: 51800K .......... .......... .......... .......... .......... 38% 22.2M 3s 2024-04-21T08:46:22.812 INFO:tasks.workunit.client.0.smithi057.stderr: 51850K .......... .......... .......... .......... .......... 38% 36.0M 3s 2024-04-21T08:46:22.814 INFO:tasks.workunit.client.0.smithi057.stderr: 51900K .......... .......... .......... .......... .......... 38% 25.7M 3s 2024-04-21T08:46:22.816 INFO:tasks.workunit.client.0.smithi057.stderr: 51950K .......... .......... .......... .......... .......... 38% 23.1M 3s 2024-04-21T08:46:22.817 INFO:tasks.workunit.client.0.smithi057.stderr: 52000K .......... .......... .......... .......... .......... 38% 31.3M 3s 2024-04-21T08:46:22.819 INFO:tasks.workunit.client.0.smithi057.stderr: 52050K .......... .......... .......... .......... .......... 38% 24.1M 3s 2024-04-21T08:46:22.821 INFO:tasks.workunit.client.0.smithi057.stderr: 52100K .......... .......... .......... .......... .......... 38% 42.1M 3s 2024-04-21T08:46:22.823 INFO:tasks.workunit.client.0.smithi057.stderr: 52150K .......... .......... .......... .......... .......... 38% 23.0M 3s 2024-04-21T08:46:22.825 INFO:tasks.workunit.client.0.smithi057.stderr: 52200K .......... .......... .......... .......... .......... 38% 24.0M 3s 2024-04-21T08:46:22.826 INFO:tasks.workunit.client.0.smithi057.stderr: 52250K .......... .......... .......... .......... .......... 38% 58.3M 3s 2024-04-21T08:46:22.828 INFO:tasks.workunit.client.0.smithi057.stderr: 52300K .......... .......... .......... .......... .......... 38% 22.9M 3s 2024-04-21T08:46:22.829 INFO:tasks.workunit.client.0.smithi057.stderr: 52350K .......... .......... .......... .......... .......... 38% 40.1M 3s 2024-04-21T08:46:22.831 INFO:tasks.workunit.client.0.smithi057.stderr: 52400K .......... .......... .......... .......... .......... 38% 25.7M 3s 2024-04-21T08:46:22.833 INFO:tasks.workunit.client.0.smithi057.stderr: 52450K .......... .......... .......... .......... .......... 38% 22.5M 3s 2024-04-21T08:46:22.834 INFO:tasks.workunit.client.0.smithi057.stderr: 52500K .......... .......... .......... .......... .......... 38% 49.1M 3s 2024-04-21T08:46:22.836 INFO:tasks.workunit.client.0.smithi057.stderr: 52550K .......... .......... .......... .......... .......... 38% 23.5M 3s 2024-04-21T08:46:22.839 INFO:tasks.workunit.client.0.smithi057.stderr: 52600K .......... .......... .......... .......... .......... 38% 21.3M 3s 2024-04-21T08:46:22.840 INFO:tasks.workunit.client.0.smithi057.stderr: 52650K .......... .......... .......... .......... .......... 38% 41.8M 3s 2024-04-21T08:46:22.842 INFO:tasks.workunit.client.0.smithi057.stderr: 52700K .......... .......... .......... .......... .......... 38% 25.6M 3s 2024-04-21T08:46:22.843 INFO:tasks.workunit.client.0.smithi057.stderr: 52750K .......... .......... .......... .......... .......... 38% 37.6M 3s 2024-04-21T08:46:22.844 INFO:tasks.workunit.client.0.smithi057.stderr: 52800K .......... .......... .......... .......... .......... 38% 37.0M 3s 2024-04-21T08:46:22.846 INFO:tasks.workunit.client.0.smithi057.stderr: 52850K .......... .......... .......... .......... .......... 38% 24.8M 3s 2024-04-21T08:46:22.850 INFO:tasks.workunit.client.0.smithi057.stderr: 52900K .......... .......... .......... .......... .......... 39% 25.1M 3s 2024-04-21T08:46:22.851 INFO:tasks.workunit.client.0.smithi057.stderr: 52950K .......... .......... .......... .......... .......... 39% 25.8M 3s 2024-04-21T08:46:22.851 INFO:tasks.workunit.client.0.smithi057.stderr: 53000K .......... .......... .......... .......... .......... 39% 41.8M 3s 2024-04-21T08:46:22.853 INFO:tasks.workunit.client.0.smithi057.stderr: 53050K .......... .......... .......... .......... .......... 39% 24.2M 3s 2024-04-21T08:46:22.855 INFO:tasks.workunit.client.0.smithi057.stderr: 53100K .......... .......... .......... .......... .......... 39% 40.5M 3s 2024-04-21T08:46:22.857 INFO:tasks.workunit.client.0.smithi057.stderr: 53150K .......... .......... .......... .......... .......... 39% 22.7M 3s 2024-04-21T08:46:22.859 INFO:tasks.workunit.client.0.smithi057.stderr: 53200K .......... .......... .......... .......... .......... 39% 23.4M 3s 2024-04-21T08:46:22.860 INFO:tasks.workunit.client.0.smithi057.stderr: 53250K .......... .......... .......... .......... .......... 39% 46.7M 3s 2024-04-21T08:46:22.861 INFO:tasks.workunit.client.0.smithi057.stderr: 53300K .......... .......... .......... .......... .......... 39% 27.9M 3s 2024-04-21T08:46:22.864 INFO:tasks.workunit.client.0.smithi057.stderr: 53350K .......... .......... .......... .......... .......... 39% 21.7M 3s 2024-04-21T08:46:22.865 INFO:tasks.workunit.client.0.smithi057.stderr: 53400K .......... .......... .......... .......... .......... 39% 38.1M 3s 2024-04-21T08:46:22.866 INFO:tasks.workunit.client.0.smithi057.stderr: 53450K .......... .......... .......... .......... .......... 39% 29.7M 3s 2024-04-21T08:46:22.868 INFO:tasks.workunit.client.0.smithi057.stderr: 53500K .......... .......... .......... .......... .......... 39% 39.9M 3s 2024-04-21T08:46:22.870 INFO:tasks.workunit.client.0.smithi057.stderr: 53550K .......... .......... .......... .......... .......... 39% 22.8M 3s 2024-04-21T08:46:22.872 INFO:tasks.workunit.client.0.smithi057.stderr: 53600K .......... .......... .......... .......... .......... 39% 21.7M 3s 2024-04-21T08:46:22.873 INFO:tasks.workunit.client.0.smithi057.stderr: 53650K .......... .......... .......... .......... .......... 39% 40.9M 3s 2024-04-21T08:46:22.876 INFO:tasks.workunit.client.0.smithi057.stderr: 53700K .......... .......... .......... .......... .......... 39% 21.8M 3s 2024-04-21T08:46:22.877 INFO:tasks.workunit.client.0.smithi057.stderr: 53750K .......... .......... .......... .......... .......... 39% 45.8M 3s 2024-04-21T08:46:22.879 INFO:tasks.workunit.client.0.smithi057.stderr: 53800K .......... .......... .......... .......... .......... 39% 24.8M 3s 2024-04-21T08:46:22.881 INFO:tasks.workunit.client.0.smithi057.stderr: 53850K .......... .......... .......... .......... .......... 39% 25.0M 3s 2024-04-21T08:46:22.882 INFO:tasks.workunit.client.0.smithi057.stderr: 53900K .......... .......... .......... .......... .......... 39% 40.7M 3s 2024-04-21T08:46:22.884 INFO:tasks.workunit.client.0.smithi057.stderr: 53950K .......... .......... .......... .......... .......... 39% 23.3M 3s 2024-04-21T08:46:22.885 INFO:tasks.workunit.client.0.smithi057.stderr: 54000K .......... .......... .......... .......... .......... 39% 43.6M 3s 2024-04-21T08:46:22.887 INFO:tasks.workunit.client.0.smithi057.stderr: 54050K .......... .......... .......... .......... .......... 39% 26.2M 3s 2024-04-21T08:46:22.889 INFO:tasks.workunit.client.0.smithi057.stderr: 54100K .......... .......... .......... .......... .......... 39% 23.6M 3s 2024-04-21T08:46:22.890 INFO:tasks.workunit.client.0.smithi057.stderr: 54150K .......... .......... .......... .......... .......... 39% 41.4M 3s 2024-04-21T08:46:22.892 INFO:tasks.workunit.client.0.smithi057.stderr: 54200K .......... .......... .......... .......... .......... 39% 22.3M 3s 2024-04-21T08:46:22.893 INFO:tasks.workunit.client.0.smithi057.stderr: 54250K .......... .......... .......... .......... .......... 39% 52.6M 3s 2024-04-21T08:46:22.895 INFO:tasks.workunit.client.0.smithi057.stderr: 54300K .......... .......... .......... .......... .......... 40% 23.4M 3s 2024-04-21T08:46:22.897 INFO:tasks.workunit.client.0.smithi057.stderr: 54350K .......... .......... .......... .......... .......... 40% 24.0M 3s 2024-04-21T08:46:22.899 INFO:tasks.workunit.client.0.smithi057.stderr: 54400K .......... .......... .......... .......... .......... 40% 41.6M 3s 2024-04-21T08:46:22.901 INFO:tasks.workunit.client.0.smithi057.stderr: 54450K .......... .......... .......... .......... .......... 40% 22.8M 3s 2024-04-21T08:46:22.903 INFO:tasks.workunit.client.0.smithi057.stderr: 54500K .......... .......... .......... .......... .......... 40% 46.7M 3s 2024-04-21T08:46:22.904 INFO:tasks.workunit.client.0.smithi057.stderr: 54550K .......... .......... .......... .......... .......... 40% 24.9M 3s 2024-04-21T08:46:22.906 INFO:tasks.workunit.client.0.smithi057.stderr: 54600K .......... .......... .......... .......... .......... 40% 24.1M 3s 2024-04-21T08:46:22.907 INFO:tasks.workunit.client.0.smithi057.stderr: 54650K .......... .......... .......... .......... .......... 40% 35.7M 3s 2024-04-21T08:46:22.909 INFO:tasks.workunit.client.0.smithi057.stderr: 54700K .......... .......... .......... .......... .......... 40% 24.9M 3s 2024-04-21T08:46:22.911 INFO:tasks.workunit.client.0.smithi057.stderr: 54750K .......... .......... .......... .......... .......... 40% 24.8M 3s 2024-04-21T08:46:22.912 INFO:tasks.workunit.client.0.smithi057.stderr: 54800K .......... .......... .......... .......... .......... 40% 50.6M 3s 2024-04-21T08:46:22.914 INFO:tasks.workunit.client.0.smithi057.stderr: 54850K .......... .......... .......... .......... .......... 40% 23.6M 3s 2024-04-21T08:46:22.915 INFO:tasks.workunit.client.0.smithi057.stderr: 54900K .......... .......... .......... .......... .......... 40% 40.8M 3s 2024-04-21T08:46:22.917 INFO:tasks.workunit.client.0.smithi057.stderr: 54950K .......... .......... .......... .......... .......... 40% 23.6M 3s 2024-04-21T08:46:22.920 INFO:tasks.workunit.client.0.smithi057.stderr: 55000K .......... .......... .......... .......... .......... 40% 22.3M 3s 2024-04-21T08:46:22.920 INFO:tasks.workunit.client.0.smithi057.stderr: 55050K .......... .......... .......... .......... .......... 40% 46.2M 3s 2024-04-21T08:46:22.923 INFO:tasks.workunit.client.0.smithi057.stderr: 55100K .......... .......... .......... .......... .......... 40% 25.0M 3s 2024-04-21T08:46:22.923 INFO:tasks.workunit.client.0.smithi057.stderr: 55150K .......... .......... .......... .......... .......... 40% 41.4M 3s 2024-04-21T08:46:22.926 INFO:tasks.workunit.client.0.smithi057.stderr: 55200K .......... .......... .......... .......... .......... 40% 23.0M 3s 2024-04-21T08:46:22.928 INFO:tasks.workunit.client.0.smithi057.stderr: 55250K .......... .......... .......... .......... .......... 40% 24.5M 3s 2024-04-21T08:46:22.929 INFO:tasks.workunit.client.0.smithi057.stderr: 55300K .......... .......... .......... .......... .......... 40% 47.8M 3s 2024-04-21T08:46:22.931 INFO:tasks.workunit.client.0.smithi057.stderr: 55350K .......... .......... .......... .......... .......... 40% 24.0M 3s 2024-04-21T08:46:22.932 INFO:tasks.workunit.client.0.smithi057.stderr: 55400K .......... .......... .......... .......... .......... 40% 40.2M 3s 2024-04-21T08:46:22.935 INFO:tasks.workunit.client.0.smithi057.stderr: 55450K .......... .......... .......... .......... .......... 40% 20.2M 3s 2024-04-21T08:46:22.936 INFO:tasks.workunit.client.0.smithi057.stderr: 55500K .......... .......... .......... .......... .......... 40% 24.8M 3s 2024-04-21T08:46:22.938 INFO:tasks.workunit.client.0.smithi057.stderr: 55550K .......... .......... .......... .......... .......... 40% 39.1M 3s 2024-04-21T08:46:22.940 INFO:tasks.workunit.client.0.smithi057.stderr: 55600K .......... .......... .......... .......... .......... 40% 24.9M 3s 2024-04-21T08:46:22.941 INFO:tasks.workunit.client.0.smithi057.stderr: 55650K .......... .......... .......... .......... .......... 41% 29.6M 3s 2024-04-21T08:46:22.943 INFO:tasks.workunit.client.0.smithi057.stderr: 55700K .......... .......... .......... .......... .......... 41% 24.9M 3s 2024-04-21T08:46:22.945 INFO:tasks.workunit.client.0.smithi057.stderr: 55750K .......... .......... .......... .......... .......... 41% 23.2M 3s 2024-04-21T08:46:22.946 INFO:tasks.workunit.client.0.smithi057.stderr: 55800K .......... .......... .......... .......... .......... 41% 47.7M 3s 2024-04-21T08:46:22.948 INFO:tasks.workunit.client.0.smithi057.stderr: 55850K .......... .......... .......... .......... .......... 41% 24.8M 3s 2024-04-21T08:46:22.950 INFO:tasks.workunit.client.0.smithi057.stderr: 55900K .......... .......... .......... .......... .......... 41% 34.5M 3s 2024-04-21T08:46:22.952 INFO:tasks.workunit.client.0.smithi057.stderr: 55950K .......... .......... .......... .......... .......... 41% 22.1M 3s 2024-04-21T08:46:22.954 INFO:tasks.workunit.client.0.smithi057.stderr: 56000K .......... .......... .......... .......... .......... 41% 26.4M 3s 2024-04-21T08:46:22.955 INFO:tasks.workunit.client.0.smithi057.stderr: 56050K .......... .......... .......... .......... .......... 41% 55.1M 3s 2024-04-21T08:46:22.957 INFO:tasks.workunit.client.0.smithi057.stderr: 56100K .......... .......... .......... .......... .......... 41% 23.1M 3s 2024-04-21T08:46:22.958 INFO:tasks.workunit.client.0.smithi057.stderr: 56150K .......... .......... .......... .......... .......... 41% 37.9M 3s 2024-04-21T08:46:22.960 INFO:tasks.workunit.client.0.smithi057.stderr: 56200K .......... .......... .......... .......... .......... 41% 24.2M 3s 2024-04-21T08:46:22.963 INFO:tasks.workunit.client.0.smithi057.stderr: 56250K .......... .......... .......... .......... .......... 41% 22.6M 3s 2024-04-21T08:46:22.963 INFO:tasks.workunit.client.0.smithi057.stderr: 56300K .......... .......... .......... .......... .......... 41% 60.9M 3s 2024-04-21T08:46:22.965 INFO:tasks.workunit.client.0.smithi057.stderr: 56350K .......... .......... .......... .......... .......... 41% 22.8M 3s 2024-04-21T08:46:22.966 INFO:tasks.workunit.client.0.smithi057.stderr: 56400K .......... .......... .......... .......... .......... 41% 40.2M 3s 2024-04-21T08:46:22.969 INFO:tasks.workunit.client.0.smithi057.stderr: 56450K .......... .......... .......... .......... .......... 41% 21.6M 3s 2024-04-21T08:46:22.971 INFO:tasks.workunit.client.0.smithi057.stderr: 56500K .......... .......... .......... .......... .......... 41% 23.7M 3s 2024-04-21T08:46:22.972 INFO:tasks.workunit.client.0.smithi057.stderr: 56550K .......... .......... .......... .......... .......... 41% 56.2M 3s 2024-04-21T08:46:22.974 INFO:tasks.workunit.client.0.smithi057.stderr: 56600K .......... .......... .......... .......... .......... 41% 24.1M 3s 2024-04-21T08:46:22.976 INFO:tasks.workunit.client.0.smithi057.stderr: 56650K .......... .......... .......... .......... .......... 41% 31.4M 3s 2024-04-21T08:46:22.977 INFO:tasks.workunit.client.0.smithi057.stderr: 56700K .......... .......... .......... .......... .......... 41% 27.3M 3s 2024-04-21T08:46:22.978 INFO:tasks.workunit.client.0.smithi057.stderr: 56750K .......... .......... .......... .......... .......... 41% 32.3M 3s 2024-04-21T08:46:22.980 INFO:tasks.workunit.client.0.smithi057.stderr: 56800K .......... .......... .......... .......... .......... 41% 35.9M 3s 2024-04-21T08:46:22.982 INFO:tasks.workunit.client.0.smithi057.stderr: 56850K .......... .......... .......... .......... .......... 41% 22.8M 3s 2024-04-21T08:46:22.983 INFO:tasks.workunit.client.0.smithi057.stderr: 56900K .......... .......... .......... .......... .......... 41% 35.9M 3s 2024-04-21T08:46:22.986 INFO:tasks.workunit.client.0.smithi057.stderr: 56950K .......... .......... .......... .......... .......... 41% 22.3M 3s 2024-04-21T08:46:22.988 INFO:tasks.workunit.client.0.smithi057.stderr: 57000K .......... .......... .......... .......... .......... 42% 25.4M 3s 2024-04-21T08:46:22.988 INFO:tasks.workunit.client.0.smithi057.stderr: 57050K .......... .......... .......... .......... .......... 42% 49.7M 3s 2024-04-21T08:46:22.991 INFO:tasks.workunit.client.0.smithi057.stderr: 57100K .......... .......... .......... .......... .......... 42% 23.1M 3s 2024-04-21T08:46:22.992 INFO:tasks.workunit.client.0.smithi057.stderr: 57150K .......... .......... .......... .......... .......... 42% 33.3M 3s 2024-04-21T08:46:22.994 INFO:tasks.workunit.client.0.smithi057.stderr: 57200K .......... .......... .......... .......... .......... 42% 25.1M 3s 2024-04-21T08:46:22.998 INFO:tasks.workunit.client.0.smithi057.stderr: 57250K .......... .......... .......... .......... .......... 42% 24.3M 3s 2024-04-21T08:46:22.998 INFO:tasks.workunit.client.0.smithi057.stderr: 57300K .......... .......... .......... .......... .......... 42% 58.4M 3s 2024-04-21T08:46:22.999 INFO:tasks.workunit.client.0.smithi057.stderr: 57350K .......... .......... .......... .......... .......... 42% 23.4M 3s 2024-04-21T08:46:23.000 INFO:tasks.workunit.client.0.smithi057.stderr: 57400K .......... .......... .......... .......... .......... 42% 38.5M 3s 2024-04-21T08:46:23.003 INFO:tasks.workunit.client.0.smithi057.stderr: 57450K .......... .......... .......... .......... .......... 42% 18.5M 3s 2024-04-21T08:46:23.005 INFO:tasks.workunit.client.0.smithi057.stderr: 57500K .......... .......... .......... .......... .......... 42% 24.5M 3s 2024-04-21T08:46:23.006 INFO:tasks.workunit.client.0.smithi057.stderr: 57550K .......... .......... .......... .......... .......... 42% 50.0M 3s 2024-04-21T08:46:23.008 INFO:tasks.workunit.client.0.smithi057.stderr: 57600K .......... .......... .......... .......... .......... 42% 27.2M 3s 2024-04-21T08:46:23.009 INFO:tasks.workunit.client.0.smithi057.stderr: 57650K .......... .......... .......... .......... .......... 42% 33.1M 3s 2024-04-21T08:46:23.011 INFO:tasks.workunit.client.0.smithi057.stderr: 57700K .......... .......... .......... .......... .......... 42% 21.9M 3s 2024-04-21T08:46:23.014 INFO:tasks.workunit.client.0.smithi057.stderr: 57750K .......... .......... .......... .......... .......... 42% 22.3M 3s 2024-04-21T08:46:23.015 INFO:tasks.workunit.client.0.smithi057.stderr: 57800K .......... .......... .......... .......... .......... 42% 36.5M 3s 2024-04-21T08:46:23.017 INFO:tasks.workunit.client.0.smithi057.stderr: 57850K .......... .......... .......... .......... .......... 42% 25.2M 3s 2024-04-21T08:46:23.018 INFO:tasks.workunit.client.0.smithi057.stderr: 57900K .......... .......... .......... .......... .......... 42% 40.6M 3s 2024-04-21T08:46:23.020 INFO:tasks.workunit.client.0.smithi057.stderr: 57950K .......... .......... .......... .......... .......... 42% 22.5M 3s 2024-04-21T08:46:23.022 INFO:tasks.workunit.client.0.smithi057.stderr: 58000K .......... .......... .......... .......... .......... 42% 25.4M 3s 2024-04-21T08:46:23.023 INFO:tasks.workunit.client.0.smithi057.stderr: 58050K .......... .......... .......... .......... .......... 42% 50.0M 3s 2024-04-21T08:46:23.025 INFO:tasks.workunit.client.0.smithi057.stderr: 58100K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-04-21T08:46:23.027 INFO:tasks.workunit.client.0.smithi057.stderr: 58150K .......... .......... .......... .......... .......... 42% 40.3M 3s 2024-04-21T08:46:23.029 INFO:tasks.workunit.client.0.smithi057.stderr: 58200K .......... .......... .......... .......... .......... 42% 22.4M 3s 2024-04-21T08:46:23.031 INFO:tasks.workunit.client.0.smithi057.stderr: 58250K .......... .......... .......... .......... .......... 42% 22.6M 3s 2024-04-21T08:46:23.031 INFO:tasks.workunit.client.0.smithi057.stderr: 58300K .......... .......... .......... .......... .......... 42% 61.0M 3s 2024-04-21T08:46:23.034 INFO:tasks.workunit.client.0.smithi057.stderr: 58350K .......... .......... .......... .......... .......... 43% 21.9M 3s 2024-04-21T08:46:23.036 INFO:tasks.workunit.client.0.smithi057.stderr: 58400K .......... .......... .......... .......... .......... 43% 36.1M 3s 2024-04-21T08:46:23.037 INFO:tasks.workunit.client.0.smithi057.stderr: 58450K .......... .......... .......... .......... .......... 43% 27.3M 3s 2024-04-21T08:46:23.039 INFO:tasks.workunit.client.0.smithi057.stderr: 58500K .......... .......... .......... .......... .......... 43% 20.1M 3s 2024-04-21T08:46:23.040 INFO:tasks.workunit.client.0.smithi057.stderr: 58550K .......... .......... .......... .......... .......... 43% 43.6M 3s 2024-04-21T08:46:23.042 INFO:tasks.workunit.client.0.smithi057.stderr: 58600K .......... .......... .......... .......... .......... 43% 23.3M 3s 2024-04-21T08:46:23.045 INFO:tasks.workunit.client.0.smithi057.stderr: 58650K .......... .......... .......... .......... .......... 43% 23.8M 3s 2024-04-21T08:46:23.046 INFO:tasks.workunit.client.0.smithi057.stderr: 58700K .......... .......... .......... .......... .......... 43% 37.6M 3s 2024-04-21T08:46:23.048 INFO:tasks.workunit.client.0.smithi057.stderr: 58750K .......... .......... .......... .......... .......... 43% 24.1M 3s 2024-04-21T08:46:23.049 INFO:tasks.workunit.client.0.smithi057.stderr: 58800K .......... .......... .......... .......... .......... 43% 47.1M 3s 2024-04-21T08:46:23.051 INFO:tasks.workunit.client.0.smithi057.stderr: 58850K .......... .......... .......... .......... .......... 43% 25.4M 3s 2024-04-21T08:46:23.053 INFO:tasks.workunit.client.0.smithi057.stderr: 58900K .......... .......... .......... .......... .......... 43% 23.2M 3s 2024-04-21T08:46:23.054 INFO:tasks.workunit.client.0.smithi057.stderr: 58950K .......... .......... .......... .......... .......... 43% 43.7M 3s 2024-04-21T08:46:23.056 INFO:tasks.workunit.client.0.smithi057.stderr: 59000K .......... .......... .......... .......... .......... 43% 21.5M 3s 2024-04-21T08:46:23.057 INFO:tasks.workunit.client.0.smithi057.stderr: 59050K .......... .......... .......... .......... .......... 43% 56.6M 3s 2024-04-21T08:46:23.060 INFO:tasks.workunit.client.0.smithi057.stderr: 59100K .......... .......... .......... .......... .......... 43% 21.7M 3s 2024-04-21T08:46:23.062 INFO:tasks.workunit.client.0.smithi057.stderr: 59150K .......... .......... .......... .......... .......... 43% 22.7M 3s 2024-04-21T08:46:23.063 INFO:tasks.workunit.client.0.smithi057.stderr: 59200K .......... .......... .......... .......... .......... 43% 40.1M 3s 2024-04-21T08:46:23.065 INFO:tasks.workunit.client.0.smithi057.stderr: 59250K .......... .......... .......... .......... .......... 43% 24.3M 3s 2024-04-21T08:46:23.066 INFO:tasks.workunit.client.0.smithi057.stderr: 59300K .......... .......... .......... .......... .......... 43% 51.1M 3s 2024-04-21T08:46:23.068 INFO:tasks.workunit.client.0.smithi057.stderr: 59350K .......... .......... .......... .......... .......... 43% 23.8M 3s 2024-04-21T08:46:23.070 INFO:tasks.workunit.client.0.smithi057.stderr: 59400K .......... .......... .......... .......... .......... 43% 23.3M 3s 2024-04-21T08:46:23.071 INFO:tasks.workunit.client.0.smithi057.stderr: 59450K .......... .......... .......... .......... .......... 43% 37.7M 3s 2024-04-21T08:46:23.073 INFO:tasks.workunit.client.0.smithi057.stderr: 59500K .......... .......... .......... .......... .......... 43% 24.5M 3s 2024-04-21T08:46:23.074 INFO:tasks.workunit.client.0.smithi057.stderr: 59550K .......... .......... .......... .......... .......... 43% 47.6M 3s 2024-04-21T08:46:23.076 INFO:tasks.workunit.client.0.smithi057.stderr: 59600K .......... .......... .......... .......... .......... 43% 23.8M 3s 2024-04-21T08:46:23.078 INFO:tasks.workunit.client.0.smithi057.stderr: 59650K .......... .......... .......... .......... .......... 43% 24.4M 3s 2024-04-21T08:46:23.080 INFO:tasks.workunit.client.0.smithi057.stderr: 59700K .......... .......... .......... .......... .......... 44% 36.2M 3s 2024-04-21T08:46:23.082 INFO:tasks.workunit.client.0.smithi057.stderr: 59750K .......... .......... .......... .......... .......... 44% 22.9M 3s 2024-04-21T08:46:23.084 INFO:tasks.workunit.client.0.smithi057.stderr: 59800K .......... .......... .......... .......... .......... 44% 57.0M 3s 2024-04-21T08:46:23.085 INFO:tasks.workunit.client.0.smithi057.stderr: 59850K .......... .......... .......... .......... .......... 44% 21.4M 3s 2024-04-21T08:46:23.087 INFO:tasks.workunit.client.0.smithi057.stderr: 59900K .......... .......... .......... .......... .......... 44% 24.4M 3s 2024-04-21T08:46:23.088 INFO:tasks.workunit.client.0.smithi057.stderr: 59950K .......... .......... .......... .......... .......... 44% 42.9M 3s 2024-04-21T08:46:23.090 INFO:tasks.workunit.client.0.smithi057.stderr: 60000K .......... .......... .......... .......... .......... 44% 22.7M 3s 2024-04-21T08:46:23.092 INFO:tasks.workunit.client.0.smithi057.stderr: 60050K .......... .......... .......... .......... .......... 44% 23.4M 3s 2024-04-21T08:46:23.093 INFO:tasks.workunit.client.0.smithi057.stderr: 60100K .......... .......... .......... .......... .......... 44% 44.8M 3s 2024-04-21T08:46:23.095 INFO:tasks.workunit.client.0.smithi057.stderr: 60150K .......... .......... .......... .......... .......... 44% 25.3M 3s 2024-04-21T08:46:23.097 INFO:tasks.workunit.client.0.smithi057.stderr: 60200K .......... .......... .......... .......... .......... 44% 39.7M 3s 2024-04-21T08:46:23.098 INFO:tasks.workunit.client.0.smithi057.stderr: 60250K .......... .......... .......... .......... .......... 44% 23.1M 3s 2024-04-21T08:46:23.101 INFO:tasks.workunit.client.0.smithi057.stderr: 60300K .......... .......... .......... .......... .......... 44% 24.0M 3s 2024-04-21T08:46:23.102 INFO:tasks.workunit.client.0.smithi057.stderr: 60350K .......... .......... .......... .......... .......... 44% 48.2M 3s 2024-04-21T08:46:23.104 INFO:tasks.workunit.client.0.smithi057.stderr: 60400K .......... .......... .......... .......... .......... 44% 24.4M 3s 2024-04-21T08:46:23.105 INFO:tasks.workunit.client.0.smithi057.stderr: 60450K .......... .......... .......... .......... .......... 44% 37.8M 3s 2024-04-21T08:46:23.108 INFO:tasks.workunit.client.0.smithi057.stderr: 60500K .......... .......... .......... .......... .......... 44% 18.8M 3s 2024-04-21T08:46:23.110 INFO:tasks.workunit.client.0.smithi057.stderr: 60550K .......... .......... .......... .......... .......... 44% 23.3M 3s 2024-04-21T08:46:23.111 INFO:tasks.workunit.client.0.smithi057.stderr: 60600K .......... .......... .......... .......... .......... 44% 44.5M 3s 2024-04-21T08:46:23.113 INFO:tasks.workunit.client.0.smithi057.stderr: 60650K .......... .......... .......... .......... .......... 44% 25.1M 3s 2024-04-21T08:46:23.114 INFO:tasks.workunit.client.0.smithi057.stderr: 60700K .......... .......... .......... .......... .......... 44% 35.9M 3s 2024-04-21T08:46:23.116 INFO:tasks.workunit.client.0.smithi057.stderr: 60750K .......... .......... .......... .......... .......... 44% 26.3M 3s 2024-04-21T08:46:23.118 INFO:tasks.workunit.client.0.smithi057.stderr: 60800K .......... .......... .......... .......... .......... 44% 26.1M 3s 2024-04-21T08:46:23.119 INFO:tasks.workunit.client.0.smithi057.stderr: 60850K .......... .......... .......... .......... .......... 44% 36.6M 3s 2024-04-21T08:46:23.121 INFO:tasks.workunit.client.0.smithi057.stderr: 60900K .......... .......... .......... .......... .......... 44% 25.3M 3s 2024-04-21T08:46:23.122 INFO:tasks.workunit.client.0.smithi057.stderr: 60950K .......... .......... .......... .......... .......... 44% 47.0M 3s 2024-04-21T08:46:23.124 INFO:tasks.workunit.client.0.smithi057.stderr: 61000K .......... .......... .......... .......... .......... 44% 28.7M 3s 2024-04-21T08:46:23.126 INFO:tasks.workunit.client.0.smithi057.stderr: 61050K .......... .......... .......... .......... .......... 45% 19.2M 3s 2024-04-21T08:46:23.127 INFO:tasks.workunit.client.0.smithi057.stderr: 61100K .......... .......... .......... .......... .......... 45% 43.6M 3s 2024-04-21T08:46:23.129 INFO:tasks.workunit.client.0.smithi057.stderr: 61150K .......... .......... .......... .......... .......... 45% 23.4M 3s 2024-04-21T08:46:23.131 INFO:tasks.workunit.client.0.smithi057.stderr: 61200K .......... .......... .......... .......... .......... 45% 40.1M 3s 2024-04-21T08:46:23.132 INFO:tasks.workunit.client.0.smithi057.stderr: 61250K .......... .......... .......... .......... .......... 45% 27.7M 3s 2024-04-21T08:46:23.135 INFO:tasks.workunit.client.0.smithi057.stderr: 61300K .......... .......... .......... .......... .......... 45% 21.0M 3s 2024-04-21T08:46:23.136 INFO:tasks.workunit.client.0.smithi057.stderr: 61350K .......... .......... .......... .......... .......... 45% 40.1M 3s 2024-04-21T08:46:23.138 INFO:tasks.workunit.client.0.smithi057.stderr: 61400K .......... .......... .......... .......... .......... 45% 25.5M 3s 2024-04-21T08:46:23.139 INFO:tasks.workunit.client.0.smithi057.stderr: 61450K .......... .......... .......... .......... .......... 45% 45.1M 3s 2024-04-21T08:46:23.141 INFO:tasks.workunit.client.0.smithi057.stderr: 61500K .......... .......... .......... .......... .......... 45% 23.3M 3s 2024-04-21T08:46:23.143 INFO:tasks.workunit.client.0.smithi057.stderr: 61550K .......... .......... .......... .......... .......... 45% 22.7M 3s 2024-04-21T08:46:23.144 INFO:tasks.workunit.client.0.smithi057.stderr: 61600K .......... .......... .......... .......... .......... 45% 50.3M 3s 2024-04-21T08:46:23.146 INFO:tasks.workunit.client.0.smithi057.stderr: 61650K .......... .......... .......... .......... .......... 45% 23.5M 3s 2024-04-21T08:46:23.148 INFO:tasks.workunit.client.0.smithi057.stderr: 61700K .......... .......... .......... .......... .......... 45% 38.5M 3s 2024-04-21T08:46:23.150 INFO:tasks.workunit.client.0.smithi057.stderr: 61750K .......... .......... .......... .......... .......... 45% 23.1M 3s 2024-04-21T08:46:23.152 INFO:tasks.workunit.client.0.smithi057.stderr: 61800K .......... .......... .......... .......... .......... 45% 24.5M 3s 2024-04-21T08:46:23.153 INFO:tasks.workunit.client.0.smithi057.stderr: 61850K .......... .......... .......... .......... .......... 45% 51.7M 3s 2024-04-21T08:46:23.155 INFO:tasks.workunit.client.0.smithi057.stderr: 61900K .......... .......... .......... .......... .......... 45% 23.3M 3s 2024-04-21T08:46:23.156 INFO:tasks.workunit.client.0.smithi057.stderr: 61950K .......... .......... .......... .......... .......... 45% 30.8M 3s 2024-04-21T08:46:23.158 INFO:tasks.workunit.client.0.smithi057.stderr: 62000K .......... .......... .......... .......... .......... 45% 29.8M 3s 2024-04-21T08:46:23.160 INFO:tasks.workunit.client.0.smithi057.stderr: 62050K .......... .......... .......... .......... .......... 45% 23.9M 3s 2024-04-21T08:46:23.161 INFO:tasks.workunit.client.0.smithi057.stderr: 62100K .......... .......... .......... .......... .......... 45% 36.2M 3s 2024-04-21T08:46:23.163 INFO:tasks.workunit.client.0.smithi057.stderr: 62150K .......... .......... .......... .......... .......... 45% 24.9M 3s 2024-04-21T08:46:23.165 INFO:tasks.workunit.client.0.smithi057.stderr: 62200K .......... .......... .......... .......... .......... 45% 39.1M 3s 2024-04-21T08:46:23.166 INFO:tasks.workunit.client.0.smithi057.stderr: 62250K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-04-21T08:46:23.172 INFO:tasks.workunit.client.0.smithi057.stderr: 62300K .......... .......... .......... .......... .......... 45% 9.23M 3s 2024-04-21T08:46:23.176 INFO:tasks.workunit.client.0.smithi057.stderr: 62350K .......... .......... .......... .......... .......... 45% 41.3M 3s 2024-04-21T08:46:23.176 INFO:tasks.workunit.client.0.smithi057.stderr: 62400K .......... .......... .......... .......... .......... 46% 482M 3s 2024-04-21T08:46:23.176 INFO:tasks.workunit.client.0.smithi057.stderr: 62450K .......... .......... .......... .......... .......... 46% 610M 3s 2024-04-21T08:46:23.176 INFO:tasks.workunit.client.0.smithi057.stderr: 62500K .......... .......... .......... .......... .......... 46% 26.6M 3s 2024-04-21T08:46:23.177 INFO:tasks.workunit.client.0.smithi057.stderr: 62550K .......... .......... .......... .......... .......... 46% 25.1M 3s 2024-04-21T08:46:23.178 INFO:tasks.workunit.client.0.smithi057.stderr: 62600K .......... .......... .......... .......... .......... 46% 45.9M 3s 2024-04-21T08:46:23.180 INFO:tasks.workunit.client.0.smithi057.stderr: 62650K .......... .......... .......... .......... .......... 46% 24.6M 3s 2024-04-21T08:46:23.182 INFO:tasks.workunit.client.0.smithi057.stderr: 62700K .......... .......... .......... .......... .......... 46% 27.9M 3s 2024-04-21T08:46:23.184 INFO:tasks.workunit.client.0.smithi057.stderr: 62750K .......... .......... .......... .......... .......... 46% 30.2M 3s 2024-04-21T08:46:23.186 INFO:tasks.workunit.client.0.smithi057.stderr: 62800K .......... .......... .......... .......... .......... 46% 24.8M 3s 2024-04-21T08:46:23.186 INFO:tasks.workunit.client.0.smithi057.stderr: 62850K .......... .......... .......... .......... .......... 46% 46.8M 3s 2024-04-21T08:46:23.188 INFO:tasks.workunit.client.0.smithi057.stderr: 62900K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-04-21T08:46:23.190 INFO:tasks.workunit.client.0.smithi057.stderr: 62950K .......... .......... .......... .......... .......... 46% 37.8M 3s 2024-04-21T08:46:23.193 INFO:tasks.workunit.client.0.smithi057.stderr: 63000K .......... .......... .......... .......... .......... 46% 25.1M 3s 2024-04-21T08:46:23.194 INFO:tasks.workunit.client.0.smithi057.stderr: 63050K .......... .......... .......... .......... .......... 46% 24.2M 3s 2024-04-21T08:46:23.195 INFO:tasks.workunit.client.0.smithi057.stderr: 63100K .......... .......... .......... .......... .......... 46% 50.5M 3s 2024-04-21T08:46:23.197 INFO:tasks.workunit.client.0.smithi057.stderr: 63150K .......... .......... .......... .......... .......... 46% 22.0M 3s 2024-04-21T08:46:23.198 INFO:tasks.workunit.client.0.smithi057.stderr: 63200K .......... .......... .......... .......... .......... 46% 40.7M 3s 2024-04-21T08:46:23.200 INFO:tasks.workunit.client.0.smithi057.stderr: 63250K .......... .......... .......... .......... .......... 46% 23.6M 3s 2024-04-21T08:46:23.202 INFO:tasks.workunit.client.0.smithi057.stderr: 63300K .......... .......... .......... .......... .......... 46% 25.1M 3s 2024-04-21T08:46:23.203 INFO:tasks.workunit.client.0.smithi057.stderr: 63350K .......... .......... .......... .......... .......... 46% 48.8M 3s 2024-04-21T08:46:23.205 INFO:tasks.workunit.client.0.smithi057.stderr: 63400K .......... .......... .......... .......... .......... 46% 23.3M 3s 2024-04-21T08:46:23.207 INFO:tasks.workunit.client.0.smithi057.stderr: 63450K .......... .......... .......... .......... .......... 46% 41.2M 3s 2024-04-21T08:46:23.209 INFO:tasks.workunit.client.0.smithi057.stderr: 63500K .......... .......... .......... .......... .......... 46% 21.8M 3s 2024-04-21T08:46:23.211 INFO:tasks.workunit.client.0.smithi057.stderr: 63550K .......... .......... .......... .......... .......... 46% 24.8M 3s 2024-04-21T08:46:23.212 INFO:tasks.workunit.client.0.smithi057.stderr: 63600K .......... .......... .......... .......... .......... 46% 42.8M 3s 2024-04-21T08:46:23.214 INFO:tasks.workunit.client.0.smithi057.stderr: 63650K .......... .......... .......... .......... .......... 46% 27.0M 3s 2024-04-21T08:46:23.216 INFO:tasks.workunit.client.0.smithi057.stderr: 63700K .......... .......... .......... .......... .......... 46% 30.6M 3s 2024-04-21T08:46:23.217 INFO:tasks.workunit.client.0.smithi057.stderr: 63750K .......... .......... .......... .......... .......... 46% 24.0M 3s 2024-04-21T08:46:23.219 INFO:tasks.workunit.client.0.smithi057.stderr: 63800K .......... .......... .......... .......... .......... 47% 27.2M 3s 2024-04-21T08:46:23.221 INFO:tasks.workunit.client.0.smithi057.stderr: 63850K .......... .......... .......... .......... .......... 47% 42.7M 3s 2024-04-21T08:46:23.223 INFO:tasks.workunit.client.0.smithi057.stderr: 63900K .......... .......... .......... .......... .......... 47% 23.2M 3s 2024-04-21T08:46:23.225 INFO:tasks.workunit.client.0.smithi057.stderr: 63950K .......... .......... .......... .......... .......... 47% 20.2M 3s 2024-04-21T08:46:23.226 INFO:tasks.workunit.client.0.smithi057.stderr: 64000K .......... .......... .......... .......... .......... 47% 37.7M 3s 2024-04-21T08:46:23.229 INFO:tasks.workunit.client.0.smithi057.stderr: 64050K .......... .......... .......... .......... .......... 47% 19.6M 3s 2024-04-21T08:46:23.229 INFO:tasks.workunit.client.0.smithi057.stderr: 64100K .......... .......... .......... .......... .......... 47% 54.7M 3s 2024-04-21T08:46:23.231 INFO:tasks.workunit.client.0.smithi057.stderr: 64150K .......... .......... .......... .......... .......... 47% 24.1M 3s 2024-04-21T08:46:23.234 INFO:tasks.workunit.client.0.smithi057.stderr: 64200K .......... .......... .......... .......... .......... 47% 23.0M 3s 2024-04-21T08:46:23.235 INFO:tasks.workunit.client.0.smithi057.stderr: 64250K .......... .......... .......... .......... .......... 47% 41.8M 3s 2024-04-21T08:46:23.237 INFO:tasks.workunit.client.0.smithi057.stderr: 64300K .......... .......... .......... .......... .......... 47% 23.7M 3s 2024-04-21T08:46:23.238 INFO:tasks.workunit.client.0.smithi057.stderr: 64350K .......... .......... .......... .......... .......... 47% 42.5M 3s 2024-04-21T08:46:23.240 INFO:tasks.workunit.client.0.smithi057.stderr: 64400K .......... .......... .......... .......... .......... 47% 23.4M 3s 2024-04-21T08:46:23.242 INFO:tasks.workunit.client.0.smithi057.stderr: 64450K .......... .......... .......... .......... .......... 47% 24.4M 3s 2024-04-21T08:46:23.243 INFO:tasks.workunit.client.0.smithi057.stderr: 64500K .......... .......... .......... .......... .......... 47% 47.1M 3s 2024-04-21T08:46:23.245 INFO:tasks.workunit.client.0.smithi057.stderr: 64550K .......... .......... .......... .......... .......... 47% 21.9M 3s 2024-04-21T08:46:23.246 INFO:tasks.workunit.client.0.smithi057.stderr: 64600K .......... .......... .......... .......... .......... 47% 52.0M 3s 2024-04-21T08:46:23.248 INFO:tasks.workunit.client.0.smithi057.stderr: 64650K .......... .......... .......... .......... .......... 47% 22.9M 3s 2024-04-21T08:46:23.250 INFO:tasks.workunit.client.0.smithi057.stderr: 64700K .......... .......... .......... .......... .......... 47% 24.2M 3s 2024-04-21T08:46:23.251 INFO:tasks.workunit.client.0.smithi057.stderr: 64750K .......... .......... .......... .......... .......... 47% 42.9M 3s 2024-04-21T08:46:23.254 INFO:tasks.workunit.client.0.smithi057.stderr: 64800K .......... .......... .......... .......... .......... 47% 22.0M 3s 2024-04-21T08:46:23.256 INFO:tasks.workunit.client.0.smithi057.stderr: 64850K .......... .......... .......... .......... .......... 47% 54.4M 3s 2024-04-21T08:46:23.257 INFO:tasks.workunit.client.0.smithi057.stderr: 64900K .......... .......... .......... .......... .......... 47% 22.8M 3s 2024-04-21T08:46:23.259 INFO:tasks.workunit.client.0.smithi057.stderr: 64950K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-04-21T08:46:23.260 INFO:tasks.workunit.client.0.smithi057.stderr: 65000K .......... .......... .......... .......... .......... 47% 42.9M 3s 2024-04-21T08:46:23.262 INFO:tasks.workunit.client.0.smithi057.stderr: 65050K .......... .......... .......... .......... .......... 47% 23.4M 3s 2024-04-21T08:46:23.264 INFO:tasks.workunit.client.0.smithi057.stderr: 65100K .......... .......... .......... .......... .......... 47% 44.7M 2s 2024-04-21T08:46:23.265 INFO:tasks.workunit.client.0.smithi057.stderr: 65150K .......... .......... .......... .......... .......... 48% 25.8M 2s 2024-04-21T08:46:23.267 INFO:tasks.workunit.client.0.smithi057.stderr: 65200K .......... .......... .......... .......... .......... 48% 22.8M 2s 2024-04-21T08:46:23.268 INFO:tasks.workunit.client.0.smithi057.stderr: 65250K .......... .......... .......... .......... .......... 48% 42.6M 2s 2024-04-21T08:46:23.270 INFO:tasks.workunit.client.0.smithi057.stderr: 65300K .......... .......... .......... .......... .......... 48% 23.5M 2s 2024-04-21T08:46:23.273 INFO:tasks.workunit.client.0.smithi057.stderr: 65350K .......... .......... .......... .......... .......... 48% 23.1M 2s 2024-04-21T08:46:23.274 INFO:tasks.workunit.client.0.smithi057.stderr: 65400K .......... .......... .......... .......... .......... 48% 51.2M 2s 2024-04-21T08:46:23.275 INFO:tasks.workunit.client.0.smithi057.stderr: 65450K .......... .......... .......... .......... .......... 48% 23.9M 2s 2024-04-21T08:46:23.277 INFO:tasks.workunit.client.0.smithi057.stderr: 65500K .......... .......... .......... .......... .......... 48% 31.9M 2s 2024-04-21T08:46:23.279 INFO:tasks.workunit.client.0.smithi057.stderr: 65550K .......... .......... .......... .......... .......... 48% 24.6M 2s 2024-04-21T08:46:23.281 INFO:tasks.workunit.client.0.smithi057.stderr: 65600K .......... .......... .......... .......... .......... 48% 25.5M 2s 2024-04-21T08:46:23.282 INFO:tasks.workunit.client.0.smithi057.stderr: 65650K .......... .......... .......... .......... .......... 48% 41.3M 2s 2024-04-21T08:46:23.284 INFO:tasks.workunit.client.0.smithi057.stderr: 65700K .......... .......... .......... .......... .......... 48% 25.2M 2s 2024-04-21T08:46:23.285 INFO:tasks.workunit.client.0.smithi057.stderr: 65750K .......... .......... .......... .......... .......... 48% 43.1M 2s 2024-04-21T08:46:23.287 INFO:tasks.workunit.client.0.smithi057.stderr: 65800K .......... .......... .......... .......... .......... 48% 23.7M 2s 2024-04-21T08:46:23.290 INFO:tasks.workunit.client.0.smithi057.stderr: 65850K .......... .......... .......... .......... .......... 48% 22.5M 2s 2024-04-21T08:46:23.291 INFO:tasks.workunit.client.0.smithi057.stderr: 65900K .......... .......... .......... .......... .......... 48% 47.5M 2s 2024-04-21T08:46:23.293 INFO:tasks.workunit.client.0.smithi057.stderr: 65950K .......... .......... .......... .......... .......... 48% 18.8M 2s 2024-04-21T08:46:23.295 INFO:tasks.workunit.client.0.smithi057.stderr: 66000K .......... .......... .......... .......... .......... 48% 28.9M 2s 2024-04-21T08:46:23.297 INFO:tasks.workunit.client.0.smithi057.stderr: 66050K .......... .......... .......... .......... .......... 48% 23.4M 2s 2024-04-21T08:46:23.299 INFO:tasks.workunit.client.0.smithi057.stderr: 66100K .......... .......... .......... .......... .......... 48% 24.1M 2s 2024-04-21T08:46:23.299 INFO:tasks.workunit.client.0.smithi057.stderr: 66150K .......... .......... .......... .......... .......... 48% 56.6M 2s 2024-04-21T08:46:23.302 INFO:tasks.workunit.client.0.smithi057.stderr: 66200K .......... .......... .......... .......... .......... 48% 21.9M 2s 2024-04-21T08:46:23.303 INFO:tasks.workunit.client.0.smithi057.stderr: 66250K .......... .......... .......... .......... .......... 48% 40.0M 2s 2024-04-21T08:46:23.306 INFO:tasks.workunit.client.0.smithi057.stderr: 66300K .......... .......... .......... .......... .......... 48% 18.9M 2s 2024-04-21T08:46:23.308 INFO:tasks.workunit.client.0.smithi057.stderr: 66350K .......... .......... .......... .......... .......... 48% 25.0M 2s 2024-04-21T08:46:23.309 INFO:tasks.workunit.client.0.smithi057.stderr: 66400K .......... .......... .......... .......... .......... 48% 44.3M 2s 2024-04-21T08:46:23.311 INFO:tasks.workunit.client.0.smithi057.stderr: 66450K .......... .......... .......... .......... .......... 48% 25.2M 2s 2024-04-21T08:46:23.312 INFO:tasks.workunit.client.0.smithi057.stderr: 66500K .......... .......... .......... .......... .......... 49% 39.5M 2s 2024-04-21T08:46:23.314 INFO:tasks.workunit.client.0.smithi057.stderr: 66550K .......... .......... .......... .......... .......... 49% 25.5M 2s 2024-04-21T08:46:23.316 INFO:tasks.workunit.client.0.smithi057.stderr: 66600K .......... .......... .......... .......... .......... 49% 22.9M 2s 2024-04-21T08:46:23.317 INFO:tasks.workunit.client.0.smithi057.stderr: 66650K .......... .......... .......... .......... .......... 49% 48.9M 2s 2024-04-21T08:46:23.319 INFO:tasks.workunit.client.0.smithi057.stderr: 66700K .......... .......... .......... .......... .......... 49% 23.9M 2s 2024-04-21T08:46:23.320 INFO:tasks.workunit.client.0.smithi057.stderr: 66750K .......... .......... .......... .......... .......... 49% 41.7M 2s 2024-04-21T08:46:23.322 INFO:tasks.workunit.client.0.smithi057.stderr: 66800K .......... .......... .......... .......... .......... 49% 21.4M 2s 2024-04-21T08:46:23.325 INFO:tasks.workunit.client.0.smithi057.stderr: 66850K .......... .......... .......... .......... .......... 49% 23.4M 2s 2024-04-21T08:46:23.325 INFO:tasks.workunit.client.0.smithi057.stderr: 66900K .......... .......... .......... .......... .......... 49% 51.9M 2s 2024-04-21T08:46:23.327 INFO:tasks.workunit.client.0.smithi057.stderr: 66950K .......... .......... .......... .......... .......... 49% 23.9M 2s 2024-04-21T08:46:23.329 INFO:tasks.workunit.client.0.smithi057.stderr: 67000K .......... .......... .......... .......... .......... 49% 37.3M 2s 2024-04-21T08:46:23.331 INFO:tasks.workunit.client.0.smithi057.stderr: 67050K .......... .......... .......... .......... .......... 49% 19.6M 2s 2024-04-21T08:46:23.334 INFO:tasks.workunit.client.0.smithi057.stderr: 67100K .......... .......... .......... .......... .......... 49% 23.2M 2s 2024-04-21T08:46:23.334 INFO:tasks.workunit.client.0.smithi057.stderr: 67150K .......... .......... .......... .......... .......... 49% 55.4M 2s 2024-04-21T08:46:23.336 INFO:tasks.workunit.client.0.smithi057.stderr: 67200K .......... .......... .......... .......... .......... 49% 24.6M 2s 2024-04-21T08:46:23.338 INFO:tasks.workunit.client.0.smithi057.stderr: 67250K .......... .......... .......... .......... .......... 49% 38.3M 2s 2024-04-21T08:46:23.340 INFO:tasks.workunit.client.0.smithi057.stderr: 67300K .......... .......... .......... .......... .......... 49% 22.9M 2s 2024-04-21T08:46:23.342 INFO:tasks.workunit.client.0.smithi057.stderr: 67350K .......... .......... .......... .......... .......... 49% 24.1M 2s 2024-04-21T08:46:23.343 INFO:tasks.workunit.client.0.smithi057.stderr: 67400K .......... .......... .......... .......... .......... 49% 44.9M 2s 2024-04-21T08:46:23.345 INFO:tasks.workunit.client.0.smithi057.stderr: 67450K .......... .......... .......... .......... .......... 49% 23.9M 2s 2024-04-21T08:46:23.346 INFO:tasks.workunit.client.0.smithi057.stderr: 67500K .......... .......... .......... .......... .......... 49% 40.5M 2s 2024-04-21T08:46:23.349 INFO:tasks.workunit.client.0.smithi057.stderr: 67550K .......... .......... .......... .......... .......... 49% 22.2M 2s 2024-04-21T08:46:23.352 INFO:tasks.workunit.client.0.smithi057.stderr: 67600K .......... .......... .......... .......... .......... 49% 24.7M 2s 2024-04-21T08:46:23.352 INFO:tasks.workunit.client.0.smithi057.stderr: 67650K .......... .......... .......... .......... .......... 49% 45.0M 2s 2024-04-21T08:46:23.353 INFO:tasks.workunit.client.0.smithi057.stderr: 67700K .......... .......... .......... .......... .......... 49% 33.1M 2s 2024-04-21T08:46:23.355 INFO:tasks.workunit.client.0.smithi057.stderr: 67750K .......... .......... .......... .......... .......... 49% 27.5M 2s 2024-04-21T08:46:23.357 INFO:tasks.workunit.client.0.smithi057.stderr: 67800K .......... .......... .......... .......... .......... 49% 23.5M 2s 2024-04-21T08:46:23.358 INFO:tasks.workunit.client.0.smithi057.stderr: 67850K .......... .......... .......... .......... .......... 50% 29.7M 2s 2024-04-21T08:46:23.360 INFO:tasks.workunit.client.0.smithi057.stderr: 67900K .......... .......... .......... .......... .......... 50% 36.8M 2s 2024-04-21T08:46:23.362 INFO:tasks.workunit.client.0.smithi057.stderr: 67950K .......... .......... .......... .......... .......... 50% 23.3M 2s 2024-04-21T08:46:23.364 INFO:tasks.workunit.client.0.smithi057.stderr: 68000K .......... .......... .......... .......... .......... 50% 37.8M 2s 2024-04-21T08:46:23.368 INFO:tasks.workunit.client.0.smithi057.stderr: 68050K .......... .......... .......... .......... .......... 50% 23.7M 2s 2024-04-21T08:46:23.369 INFO:tasks.workunit.client.0.smithi057.stderr: 68100K .......... .......... .......... .......... .......... 50% 23.7M 2s 2024-04-21T08:46:23.369 INFO:tasks.workunit.client.0.smithi057.stderr: 68150K .......... .......... .......... .......... .......... 50% 45.5M 2s 2024-04-21T08:46:23.370 INFO:tasks.workunit.client.0.smithi057.stderr: 68200K .......... .......... .......... .......... .......... 50% 23.2M 2s 2024-04-21T08:46:23.371 INFO:tasks.workunit.client.0.smithi057.stderr: 68250K .......... .......... .......... .......... .......... 50% 43.5M 2s 2024-04-21T08:46:23.373 INFO:tasks.workunit.client.0.smithi057.stderr: 68300K .......... .......... .......... .......... .......... 50% 23.5M 2s 2024-04-21T08:46:23.376 INFO:tasks.workunit.client.0.smithi057.stderr: 68350K .......... .......... .......... .......... .......... 50% 21.5M 2s 2024-04-21T08:46:23.377 INFO:tasks.workunit.client.0.smithi057.stderr: 68400K .......... .......... .......... .......... .......... 50% 47.3M 2s 2024-04-21T08:46:23.378 INFO:tasks.workunit.client.0.smithi057.stderr: 68450K .......... .......... .......... .......... .......... 50% 34.8M 2s 2024-04-21T08:46:23.380 INFO:tasks.workunit.client.0.smithi057.stderr: 68500K .......... .......... .......... .......... .......... 50% 27.7M 2s 2024-04-21T08:46:23.383 INFO:tasks.workunit.client.0.smithi057.stderr: 68550K .......... .......... .......... .......... .......... 50% 22.9M 2s 2024-04-21T08:46:23.384 INFO:tasks.workunit.client.0.smithi057.stderr: 68600K .......... .......... .......... .......... .......... 50% 25.1M 2s 2024-04-21T08:46:23.385 INFO:tasks.workunit.client.0.smithi057.stderr: 68650K .......... .......... .......... .......... .......... 50% 43.6M 2s 2024-04-21T08:46:23.387 INFO:tasks.workunit.client.0.smithi057.stderr: 68700K .......... .......... .......... .......... .......... 50% 24.8M 2s 2024-04-21T08:46:23.389 INFO:tasks.workunit.client.0.smithi057.stderr: 68750K .......... .......... .......... .......... .......... 50% 40.1M 2s 2024-04-21T08:46:23.391 INFO:tasks.workunit.client.0.smithi057.stderr: 68800K .......... .......... .......... .......... .......... 50% 22.0M 2s 2024-04-21T08:46:23.393 INFO:tasks.workunit.client.0.smithi057.stderr: 68850K .......... .......... .......... .......... .......... 50% 24.1M 2s 2024-04-21T08:46:23.394 INFO:tasks.workunit.client.0.smithi057.stderr: 68900K .......... .......... .......... .......... .......... 50% 52.3M 2s 2024-04-21T08:46:23.396 INFO:tasks.workunit.client.0.smithi057.stderr: 68950K .......... .......... .......... .......... .......... 50% 23.0M 2s 2024-04-21T08:46:23.398 INFO:tasks.workunit.client.0.smithi057.stderr: 69000K .......... .......... .......... .......... .......... 50% 35.1M 2s 2024-04-21T08:46:23.399 INFO:tasks.workunit.client.0.smithi057.stderr: 69050K .......... .......... .......... .......... .......... 50% 20.1M 2s 2024-04-21T08:46:23.402 INFO:tasks.workunit.client.0.smithi057.stderr: 69100K .......... .......... .......... .......... .......... 50% 22.9M 2s 2024-04-21T08:46:23.402 INFO:tasks.workunit.client.0.smithi057.stderr: 69150K .......... .......... .......... .......... .......... 50% 61.2M 2s 2024-04-21T08:46:23.404 INFO:tasks.workunit.client.0.smithi057.stderr: 69200K .......... .......... .......... .......... .......... 51% 26.2M 2s 2024-04-21T08:46:23.407 INFO:tasks.workunit.client.0.smithi057.stderr: 69250K .......... .......... .......... .......... .......... 51% 25.8M 2s 2024-04-21T08:46:23.408 INFO:tasks.workunit.client.0.smithi057.stderr: 69300K .......... .......... .......... .......... .......... 51% 30.1M 2s 2024-04-21T08:46:23.410 INFO:tasks.workunit.client.0.smithi057.stderr: 69350K .......... .......... .......... .......... .......... 51% 22.6M 2s 2024-04-21T08:46:23.411 INFO:tasks.workunit.client.0.smithi057.stderr: 69400K .......... .......... .......... .......... .......... 51% 53.8M 2s 2024-04-21T08:46:23.413 INFO:tasks.workunit.client.0.smithi057.stderr: 69450K .......... .......... .......... .......... .......... 51% 22.1M 2s 2024-04-21T08:46:23.415 INFO:tasks.workunit.client.0.smithi057.stderr: 69500K .......... .......... .......... .......... .......... 51% 24.9M 2s 2024-04-21T08:46:23.417 INFO:tasks.workunit.client.0.smithi057.stderr: 69550K .......... .......... .......... .......... .......... 51% 37.5M 2s 2024-04-21T08:46:23.419 INFO:tasks.workunit.client.0.smithi057.stderr: 69600K .......... .......... .......... .......... .......... 51% 23.1M 2s 2024-04-21T08:46:23.419 INFO:tasks.workunit.client.0.smithi057.stderr: 69650K .......... .......... .......... .......... .......... 51% 56.1M 2s 2024-04-21T08:46:23.422 INFO:tasks.workunit.client.0.smithi057.stderr: 69700K .......... .......... .......... .......... .......... 51% 22.3M 2s 2024-04-21T08:46:23.423 INFO:tasks.workunit.client.0.smithi057.stderr: 69750K .......... .......... .......... .......... .......... 51% 24.3M 2s 2024-04-21T08:46:23.425 INFO:tasks.workunit.client.0.smithi057.stderr: 69800K .......... .......... .......... .......... .......... 51% 38.0M 2s 2024-04-21T08:46:23.427 INFO:tasks.workunit.client.0.smithi057.stderr: 69850K .......... .......... .......... .......... .......... 51% 23.6M 2s 2024-04-21T08:46:23.427 INFO:tasks.workunit.client.0.smithi057.stderr: 69900K .......... .......... .......... .......... .......... 51% 58.6M 2s 2024-04-21T08:46:23.430 INFO:tasks.workunit.client.0.smithi057.stderr: 69950K .......... .......... .......... .......... .......... 51% 22.6M 2s 2024-04-21T08:46:23.432 INFO:tasks.workunit.client.0.smithi057.stderr: 70000K .......... .......... .......... .......... .......... 51% 26.5M 2s 2024-04-21T08:46:23.433 INFO:tasks.workunit.client.0.smithi057.stderr: 70050K .......... .......... .......... .......... .......... 51% 33.0M 2s 2024-04-21T08:46:23.435 INFO:tasks.workunit.client.0.smithi057.stderr: 70100K .......... .......... .......... .......... .......... 51% 25.5M 2s 2024-04-21T08:46:23.437 INFO:tasks.workunit.client.0.smithi057.stderr: 70150K .......... .......... .......... .......... .......... 51% 49.5M 2s 2024-04-21T08:46:23.438 INFO:tasks.workunit.client.0.smithi057.stderr: 70200K .......... .......... .......... .......... .......... 51% 28.8M 2s 2024-04-21T08:46:23.440 INFO:tasks.workunit.client.0.smithi057.stderr: 70250K .......... .......... .......... .......... .......... 51% 18.9M 2s 2024-04-21T08:46:23.442 INFO:tasks.workunit.client.0.smithi057.stderr: 70300K .......... .......... .......... .......... .......... 51% 39.4M 2s 2024-04-21T08:46:23.444 INFO:tasks.workunit.client.0.smithi057.stderr: 70350K .......... .......... .......... .......... .......... 51% 24.1M 2s 2024-04-21T08:46:23.446 INFO:tasks.workunit.client.0.smithi057.stderr: 70400K .......... .......... .......... .......... .......... 51% 59.2M 2s 2024-04-21T08:46:23.447 INFO:tasks.workunit.client.0.smithi057.stderr: 70450K .......... .......... .......... .......... .......... 51% 21.9M 2s 2024-04-21T08:46:23.449 INFO:tasks.workunit.client.0.smithi057.stderr: 70500K .......... .......... .......... .......... .......... 51% 23.8M 2s 2024-04-21T08:46:23.450 INFO:tasks.workunit.client.0.smithi057.stderr: 70550K .......... .......... .......... .......... .......... 52% 38.2M 2s 2024-04-21T08:46:23.452 INFO:tasks.workunit.client.0.smithi057.stderr: 70600K .......... .......... .......... .......... .......... 52% 22.3M 2s 2024-04-21T08:46:23.454 INFO:tasks.workunit.client.0.smithi057.stderr: 70650K .......... .......... .......... .......... .......... 52% 24.4M 2s 2024-04-21T08:46:23.455 INFO:tasks.workunit.client.0.smithi057.stderr: 70700K .......... .......... .......... .......... .......... 52% 49.7M 2s 2024-04-21T08:46:23.457 INFO:tasks.workunit.client.0.smithi057.stderr: 70750K .......... .......... .......... .......... .......... 52% 23.3M 2s 2024-04-21T08:46:23.458 INFO:tasks.workunit.client.0.smithi057.stderr: 70800K .......... .......... .......... .......... .......... 52% 40.5M 2s 2024-04-21T08:46:23.463 INFO:tasks.workunit.client.0.smithi057.stderr: 70850K .......... .......... .......... .......... .......... 52% 23.9M 2s 2024-04-21T08:46:23.464 INFO:tasks.workunit.client.0.smithi057.stderr: 70900K .......... .......... .......... .......... .......... 52% 23.5M 2s 2024-04-21T08:46:23.465 INFO:tasks.workunit.client.0.smithi057.stderr: 70950K .......... .......... .......... .......... .......... 52% 47.3M 2s 2024-04-21T08:46:23.466 INFO:tasks.workunit.client.0.smithi057.stderr: 71000K .......... .......... .......... .......... .......... 52% 23.8M 2s 2024-04-21T08:46:23.467 INFO:tasks.workunit.client.0.smithi057.stderr: 71050K .......... .......... .......... .......... .......... 52% 42.7M 2s 2024-04-21T08:46:23.469 INFO:tasks.workunit.client.0.smithi057.stderr: 71100K .......... .......... .......... .......... .......... 52% 22.3M 2s 2024-04-21T08:46:23.471 INFO:tasks.workunit.client.0.smithi057.stderr: 71150K .......... .......... .......... .......... .......... 52% 24.6M 2s 2024-04-21T08:46:23.472 INFO:tasks.workunit.client.0.smithi057.stderr: 71200K .......... .......... .......... .......... .......... 52% 48.5M 2s 2024-04-21T08:46:23.474 INFO:tasks.workunit.client.0.smithi057.stderr: 71250K .......... .......... .......... .......... .......... 52% 24.9M 2s 2024-04-21T08:46:23.475 INFO:tasks.workunit.client.0.smithi057.stderr: 71300K .......... .......... .......... .......... .......... 52% 36.0M 2s 2024-04-21T08:46:23.477 INFO:tasks.workunit.client.0.smithi057.stderr: 71350K .......... .......... .......... .......... .......... 52% 24.9M 2s 2024-04-21T08:46:23.480 INFO:tasks.workunit.client.0.smithi057.stderr: 71400K .......... .......... .......... .......... .......... 52% 23.8M 2s 2024-04-21T08:46:23.480 INFO:tasks.workunit.client.0.smithi057.stderr: 71450K .......... .......... .......... .......... .......... 52% 47.2M 2s 2024-04-21T08:46:23.482 INFO:tasks.workunit.client.0.smithi057.stderr: 71500K .......... .......... .......... .......... .......... 52% 24.9M 2s 2024-04-21T08:46:23.484 INFO:tasks.workunit.client.0.smithi057.stderr: 71550K .......... .......... .......... .......... .......... 52% 37.2M 2s 2024-04-21T08:46:23.486 INFO:tasks.workunit.client.0.smithi057.stderr: 71600K .......... .......... .......... .......... .......... 52% 24.7M 2s 2024-04-21T08:46:23.488 INFO:tasks.workunit.client.0.smithi057.stderr: 71650K .......... .......... .......... .......... .......... 52% 23.0M 2s 2024-04-21T08:46:23.488 INFO:tasks.workunit.client.0.smithi057.stderr: 71700K .......... .......... .......... .......... .......... 52% 54.2M 2s 2024-04-21T08:46:23.491 INFO:tasks.workunit.client.0.smithi057.stderr: 71750K .......... .......... .......... .......... .......... 52% 23.9M 2s 2024-04-21T08:46:23.495 INFO:tasks.workunit.client.0.smithi057.stderr: 71800K .......... .......... .......... .......... .......... 52% 37.2M 2s 2024-04-21T08:46:23.495 INFO:tasks.workunit.client.0.smithi057.stderr: 71850K .......... .......... .......... .......... .......... 52% 25.6M 2s 2024-04-21T08:46:23.496 INFO:tasks.workunit.client.0.smithi057.stderr: 71900K .......... .......... .......... .......... .......... 53% 23.1M 2s 2024-04-21T08:46:23.497 INFO:tasks.workunit.client.0.smithi057.stderr: 71950K .......... .......... .......... .......... .......... 53% 45.8M 2s 2024-04-21T08:46:23.499 INFO:tasks.workunit.client.0.smithi057.stderr: 72000K .......... .......... .......... .......... .......... 53% 20.3M 2s 2024-04-21T08:46:23.501 INFO:tasks.workunit.client.0.smithi057.stderr: 72050K .......... .......... .......... .......... .......... 53% 38.1M 2s 2024-04-21T08:46:23.503 INFO:tasks.workunit.client.0.smithi057.stderr: 72100K .......... .......... .......... .......... .......... 53% 26.2M 2s 2024-04-21T08:46:23.505 INFO:tasks.workunit.client.0.smithi057.stderr: 72150K .......... .......... .......... .......... .......... 53% 23.7M 2s 2024-04-21T08:46:23.505 INFO:tasks.workunit.client.0.smithi057.stderr: 72200K .......... .......... .......... .......... .......... 53% 49.5M 2s 2024-04-21T08:46:23.508 INFO:tasks.workunit.client.0.smithi057.stderr: 72250K .......... .......... .......... .......... .......... 53% 24.2M 2s 2024-04-21T08:46:23.509 INFO:tasks.workunit.client.0.smithi057.stderr: 72300K .......... .......... .......... .......... .......... 53% 37.7M 2s 2024-04-21T08:46:23.511 INFO:tasks.workunit.client.0.smithi057.stderr: 72350K .......... .......... .......... .......... .......... 53% 24.5M 2s 2024-04-21T08:46:23.513 INFO:tasks.workunit.client.0.smithi057.stderr: 72400K .......... .......... .......... .......... .......... 53% 23.4M 2s 2024-04-21T08:46:23.514 INFO:tasks.workunit.client.0.smithi057.stderr: 72450K .......... .......... .......... .......... .......... 53% 37.4M 2s 2024-04-21T08:46:23.516 INFO:tasks.workunit.client.0.smithi057.stderr: 72500K .......... .......... .......... .......... .......... 53% 22.9M 2s 2024-04-21T08:46:23.518 INFO:tasks.workunit.client.0.smithi057.stderr: 72550K .......... .......... .......... .......... .......... 53% 32.4M 2s 2024-04-21T08:46:23.520 INFO:tasks.workunit.client.0.smithi057.stderr: 72600K .......... .......... .......... .......... .......... 53% 21.2M 2s 2024-04-21T08:46:23.521 INFO:tasks.workunit.client.0.smithi057.stderr: 72650K .......... .......... .......... .......... .......... 53% 41.4M 2s 2024-04-21T08:46:23.523 INFO:tasks.workunit.client.0.smithi057.stderr: 72700K .......... .......... .......... .......... .......... 53% 29.3M 2s 2024-04-21T08:46:23.525 INFO:tasks.workunit.client.0.smithi057.stderr: 72750K .......... .......... .......... .......... .......... 53% 24.9M 2s 2024-04-21T08:46:23.527 INFO:tasks.workunit.client.0.smithi057.stderr: 72800K .......... .......... .......... .......... .......... 53% 22.1M 2s 2024-04-21T08:46:23.529 INFO:tasks.workunit.client.0.smithi057.stderr: 72850K .......... .......... .......... .......... .......... 53% 38.9M 2s 2024-04-21T08:46:23.530 INFO:tasks.workunit.client.0.smithi057.stderr: 72900K .......... .......... .......... .......... .......... 53% 45.8M 2s 2024-04-21T08:46:23.531 INFO:tasks.workunit.client.0.smithi057.stderr: 72950K .......... .......... .......... .......... .......... 53% 25.7M 2s 2024-04-21T08:46:23.534 INFO:tasks.workunit.client.0.smithi057.stderr: 73000K .......... .......... .......... .......... .......... 53% 22.8M 2s 2024-04-21T08:46:23.535 INFO:tasks.workunit.client.0.smithi057.stderr: 73050K .......... .......... .......... .......... .......... 53% 30.0M 2s 2024-04-21T08:46:23.537 INFO:tasks.workunit.client.0.smithi057.stderr: 73100K .......... .......... .......... .......... .......... 53% 34.3M 2s 2024-04-21T08:46:23.539 INFO:tasks.workunit.client.0.smithi057.stderr: 73150K .......... .......... .......... .......... .......... 53% 24.4M 2s 2024-04-21T08:46:23.540 INFO:tasks.workunit.client.0.smithi057.stderr: 73200K .......... .......... .......... .......... .......... 53% 40.0M 2s 2024-04-21T08:46:23.542 INFO:tasks.workunit.client.0.smithi057.stderr: 73250K .......... .......... .......... .......... .......... 53% 23.1M 2s 2024-04-21T08:46:23.544 INFO:tasks.workunit.client.0.smithi057.stderr: 73300K .......... .......... .......... .......... .......... 54% 24.1M 2s 2024-04-21T08:46:23.545 INFO:tasks.workunit.client.0.smithi057.stderr: 73350K .......... .......... .......... .......... .......... 54% 39.3M 2s 2024-04-21T08:46:23.548 INFO:tasks.workunit.client.0.smithi057.stderr: 73400K .......... .......... .......... .......... .......... 54% 23.6M 2s 2024-04-21T08:46:23.548 INFO:tasks.workunit.client.0.smithi057.stderr: 73450K .......... .......... .......... .......... .......... 54% 51.3M 2s 2024-04-21T08:46:23.550 INFO:tasks.workunit.client.0.smithi057.stderr: 73500K .......... .......... .......... .......... .......... 54% 22.8M 2s 2024-04-21T08:46:23.552 INFO:tasks.workunit.client.0.smithi057.stderr: 73550K .......... .......... .......... .......... .......... 54% 39.1M 2s 2024-04-21T08:46:23.553 INFO:tasks.workunit.client.0.smithi057.stderr: 73600K .......... .......... .......... .......... .......... 54% 26.9M 2s 2024-04-21T08:46:23.556 INFO:tasks.workunit.client.0.smithi057.stderr: 73650K .......... .......... .......... .......... .......... 54% 23.2M 2s 2024-04-21T08:46:23.557 INFO:tasks.workunit.client.0.smithi057.stderr: 73700K .......... .......... .......... .......... .......... 54% 38.2M 2s 2024-04-21T08:46:23.559 INFO:tasks.workunit.client.0.smithi057.stderr: 73750K .......... .......... .......... .......... .......... 54% 24.3M 2s 2024-04-21T08:46:23.560 INFO:tasks.workunit.client.0.smithi057.stderr: 73800K .......... .......... .......... .......... .......... 54% 39.3M 2s 2024-04-21T08:46:23.563 INFO:tasks.workunit.client.0.smithi057.stderr: 73850K .......... .......... .......... .......... .......... 54% 18.0M 2s 2024-04-21T08:46:23.564 INFO:tasks.workunit.client.0.smithi057.stderr: 73900K .......... .......... .......... .......... .......... 54% 35.3M 2s 2024-04-21T08:46:23.565 INFO:tasks.workunit.client.0.smithi057.stderr: 73950K .......... .......... .......... .......... .......... 54% 52.1M 2s 2024-04-21T08:46:23.567 INFO:tasks.workunit.client.0.smithi057.stderr: 74000K .......... .......... .......... .......... .......... 54% 31.4M 2s 2024-04-21T08:46:23.569 INFO:tasks.workunit.client.0.smithi057.stderr: 74050K .......... .......... .......... .......... .......... 54% 27.6M 2s 2024-04-21T08:46:23.571 INFO:tasks.workunit.client.0.smithi057.stderr: 74100K .......... .......... .......... .......... .......... 54% 23.0M 2s 2024-04-21T08:46:23.573 INFO:tasks.workunit.client.0.smithi057.stderr: 74150K .......... .......... .......... .......... .......... 54% 25.3M 2s 2024-04-21T08:46:23.574 INFO:tasks.workunit.client.0.smithi057.stderr: 74200K .......... .......... .......... .......... .......... 54% 44.3M 2s 2024-04-21T08:46:23.576 INFO:tasks.workunit.client.0.smithi057.stderr: 74250K .......... .......... .......... .......... .......... 54% 23.4M 2s 2024-04-21T08:46:23.577 INFO:tasks.workunit.client.0.smithi057.stderr: 74300K .......... .......... .......... .......... .......... 54% 45.2M 2s 2024-04-21T08:46:23.579 INFO:tasks.workunit.client.0.smithi057.stderr: 74350K .......... .......... .......... .......... .......... 54% 22.7M 2s 2024-04-21T08:46:23.583 INFO:tasks.workunit.client.0.smithi057.stderr: 74400K .......... .......... .......... .......... .......... 54% 31.2M 2s 2024-04-21T08:46:23.583 INFO:tasks.workunit.client.0.smithi057.stderr: 74450K .......... .......... .......... .......... .......... 54% 33.0M 2s 2024-04-21T08:46:23.585 INFO:tasks.workunit.client.0.smithi057.stderr: 74500K .......... .......... .......... .......... .......... 54% 18.5M 2s 2024-04-21T08:46:23.587 INFO:tasks.workunit.client.0.smithi057.stderr: 74550K .......... .......... .......... .......... .......... 54% 23.4M 2s 2024-04-21T08:46:23.589 INFO:tasks.workunit.client.0.smithi057.stderr: 74600K .......... .......... .......... .......... .......... 54% 39.8M 2s 2024-04-21T08:46:23.592 INFO:tasks.workunit.client.0.smithi057.stderr: 74650K .......... .......... .......... .......... .......... 55% 25.3M 2s 2024-04-21T08:46:23.593 INFO:tasks.workunit.client.0.smithi057.stderr: 74700K .......... .......... .......... .......... .......... 55% 48.4M 2s 2024-04-21T08:46:23.594 INFO:tasks.workunit.client.0.smithi057.stderr: 74750K .......... .......... .......... .......... .......... 55% 23.8M 2s 2024-04-21T08:46:23.595 INFO:tasks.workunit.client.0.smithi057.stderr: 74800K .......... .......... .......... .......... .......... 55% 21.6M 2s 2024-04-21T08:46:23.597 INFO:tasks.workunit.client.0.smithi057.stderr: 74850K .......... .......... .......... .......... .......... 55% 34.4M 2s 2024-04-21T08:46:23.599 INFO:tasks.workunit.client.0.smithi057.stderr: 74900K .......... .......... .......... .......... .......... 55% 21.5M 2s 2024-04-21T08:46:23.600 INFO:tasks.workunit.client.0.smithi057.stderr: 74950K .......... .......... .......... .......... .......... 55% 53.7M 2s 2024-04-21T08:46:23.602 INFO:tasks.workunit.client.0.smithi057.stderr: 75000K .......... .......... .......... .......... .......... 55% 21.9M 2s 2024-04-21T08:46:23.604 INFO:tasks.workunit.client.0.smithi057.stderr: 75050K .......... .......... .......... .......... .......... 55% 25.2M 2s 2024-04-21T08:46:23.605 INFO:tasks.workunit.client.0.smithi057.stderr: 75100K .......... .......... .......... .......... .......... 55% 38.4M 2s 2024-04-21T08:46:23.607 INFO:tasks.workunit.client.0.smithi057.stderr: 75150K .......... .......... .......... .......... .......... 55% 24.0M 2s 2024-04-21T08:46:23.608 INFO:tasks.workunit.client.0.smithi057.stderr: 75200K .......... .......... .......... .......... .......... 55% 55.3M 2s 2024-04-21T08:46:23.610 INFO:tasks.workunit.client.0.smithi057.stderr: 75250K .......... .......... .......... .......... .......... 55% 22.1M 2s 2024-04-21T08:46:23.613 INFO:tasks.workunit.client.0.smithi057.stderr: 75300K .......... .......... .......... .......... .......... 55% 23.7M 2s 2024-04-21T08:46:23.614 INFO:tasks.workunit.client.0.smithi057.stderr: 75350K .......... .......... .......... .......... .......... 55% 35.7M 2s 2024-04-21T08:46:23.616 INFO:tasks.workunit.client.0.smithi057.stderr: 75400K .......... .......... .......... .......... .......... 55% 25.2M 2s 2024-04-21T08:46:23.618 INFO:tasks.workunit.client.0.smithi057.stderr: 75450K .......... .......... .......... .......... .......... 55% 46.3M 2s 2024-04-21T08:46:23.619 INFO:tasks.workunit.client.0.smithi057.stderr: 75500K .......... .......... .......... .......... .......... 55% 23.4M 2s 2024-04-21T08:46:23.621 INFO:tasks.workunit.client.0.smithi057.stderr: 75550K .......... .......... .......... .......... .......... 55% 24.1M 2s 2024-04-21T08:46:23.622 INFO:tasks.workunit.client.0.smithi057.stderr: 75600K .......... .......... .......... .......... .......... 55% 38.4M 2s 2024-04-21T08:46:23.625 INFO:tasks.workunit.client.0.smithi057.stderr: 75650K .......... .......... .......... .......... .......... 55% 18.2M 2s 2024-04-21T08:46:23.627 INFO:tasks.workunit.client.0.smithi057.stderr: 75700K .......... .......... .......... .......... .......... 55% 54.1M 2s 2024-04-21T08:46:23.628 INFO:tasks.workunit.client.0.smithi057.stderr: 75750K .......... .......... .......... .......... .......... 55% 24.0M 2s 2024-04-21T08:46:23.630 INFO:tasks.workunit.client.0.smithi057.stderr: 75800K .......... .......... .......... .......... .......... 55% 23.3M 2s 2024-04-21T08:46:23.632 INFO:tasks.workunit.client.0.smithi057.stderr: 75850K .......... .......... .......... .......... .......... 55% 42.2M 2s 2024-04-21T08:46:23.633 INFO:tasks.workunit.client.0.smithi057.stderr: 75900K .......... .......... .......... .......... .......... 55% 28.1M 2s 2024-04-21T08:46:23.636 INFO:tasks.workunit.client.0.smithi057.stderr: 75950K .......... .......... .......... .......... .......... 55% 20.3M 2s 2024-04-21T08:46:23.637 INFO:tasks.workunit.client.0.smithi057.stderr: 76000K .......... .......... .......... .......... .......... 56% 43.7M 2s 2024-04-21T08:46:23.639 INFO:tasks.workunit.client.0.smithi057.stderr: 76050K .......... .......... .......... .......... .......... 56% 24.9M 2s 2024-04-21T08:46:23.641 INFO:tasks.workunit.client.0.smithi057.stderr: 76100K .......... .......... .......... .......... .......... 56% 36.8M 2s 2024-04-21T08:46:23.642 INFO:tasks.workunit.client.0.smithi057.stderr: 76150K .......... .......... .......... .......... .......... 56% 22.4M 2s 2024-04-21T08:46:23.644 INFO:tasks.workunit.client.0.smithi057.stderr: 76200K .......... .......... .......... .......... .......... 56% 25.6M 2s 2024-04-21T08:46:23.647 INFO:tasks.workunit.client.0.smithi057.stderr: 76250K .......... .......... .......... .......... .......... 56% 41.1M 2s 2024-04-21T08:46:23.647 INFO:tasks.workunit.client.0.smithi057.stderr: 76300K .......... .......... .......... .......... .......... 56% 26.0M 2s 2024-04-21T08:46:23.651 INFO:tasks.workunit.client.0.smithi057.stderr: 76350K .......... .......... .......... .......... .......... 56% 39.4M 2s 2024-04-21T08:46:23.651 INFO:tasks.workunit.client.0.smithi057.stderr: 76400K .......... .......... .......... .......... .......... 56% 22.9M 2s 2024-04-21T08:46:23.653 INFO:tasks.workunit.client.0.smithi057.stderr: 76450K .......... .......... .......... .......... .......... 56% 24.3M 2s 2024-04-21T08:46:23.654 INFO:tasks.workunit.client.0.smithi057.stderr: 76500K .......... .......... .......... .......... .......... 56% 40.4M 2s 2024-04-21T08:46:23.658 INFO:tasks.workunit.client.0.smithi057.stderr: 76550K .......... .......... .......... .......... .......... 56% 23.2M 2s 2024-04-21T08:46:23.658 INFO:tasks.workunit.client.0.smithi057.stderr: 76600K .......... .......... .......... .......... .......... 56% 42.8M 2s 2024-04-21T08:46:23.659 INFO:tasks.workunit.client.0.smithi057.stderr: 76650K .......... .......... .......... .......... .......... 56% 22.6M 2s 2024-04-21T08:46:23.663 INFO:tasks.workunit.client.0.smithi057.stderr: 76700K .......... .......... .......... .......... .......... 56% 24.1M 2s 2024-04-21T08:46:23.663 INFO:tasks.workunit.client.0.smithi057.stderr: 76750K .......... .......... .......... .......... .......... 56% 44.9M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 76800K .......... .......... .......... .......... .......... 56% 25.1M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 76850K .......... .......... .......... .......... .......... 56% 39.7M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 76900K .......... .......... .......... .......... .......... 56% 23.0M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 76950K .......... .......... .......... .......... .......... 56% 24.3M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 77000K .......... .......... .......... .......... .......... 56% 55.7M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 77050K .......... .......... .......... .......... .......... 56% 23.1M 2s 2024-04-21T08:46:23.675 INFO:tasks.workunit.client.0.smithi057.stderr: 77100K .......... .......... .......... .......... .......... 56% 38.0M 2s 2024-04-21T08:46:23.676 INFO:tasks.workunit.client.0.smithi057.stderr: 77150K .......... .......... .......... .......... .......... 56% 24.1M 2s 2024-04-21T08:46:23.678 INFO:tasks.workunit.client.0.smithi057.stderr: 77200K .......... .......... .......... .......... .......... 56% 23.4M 2s 2024-04-21T08:46:23.681 INFO:tasks.workunit.client.0.smithi057.stderr: 77250K .......... .......... .......... .......... .......... 56% 54.3M 2s 2024-04-21T08:46:23.681 INFO:tasks.workunit.client.0.smithi057.stderr: 77300K .......... .......... .......... .......... .......... 56% 22.8M 2s 2024-04-21T08:46:23.682 INFO:tasks.workunit.client.0.smithi057.stderr: 77350K .......... .......... .......... .......... .......... 57% 37.1M 2s 2024-04-21T08:46:23.684 INFO:tasks.workunit.client.0.smithi057.stderr: 77400K .......... .......... .......... .......... .......... 57% 26.4M 2s 2024-04-21T08:46:23.686 INFO:tasks.workunit.client.0.smithi057.stderr: 77450K .......... .......... .......... .......... .......... 57% 23.3M 2s 2024-04-21T08:46:23.687 INFO:tasks.workunit.client.0.smithi057.stderr: 77500K .......... .......... .......... .......... .......... 57% 52.2M 2s 2024-04-21T08:46:23.689 INFO:tasks.workunit.client.0.smithi057.stderr: 77550K .......... .......... .......... .......... .......... 57% 22.2M 2s 2024-04-21T08:46:23.691 INFO:tasks.workunit.client.0.smithi057.stderr: 77600K .......... .......... .......... .......... .......... 57% 30.0M 2s 2024-04-21T08:46:23.693 INFO:tasks.workunit.client.0.smithi057.stderr: 77650K .......... .......... .......... .......... .......... 57% 24.4M 2s 2024-04-21T08:46:23.695 INFO:tasks.workunit.client.0.smithi057.stderr: 77700K .......... .......... .......... .......... .......... 57% 23.2M 2s 2024-04-21T08:46:23.696 INFO:tasks.workunit.client.0.smithi057.stderr: 77750K .......... .......... .......... .......... .......... 57% 43.4M 2s 2024-04-21T08:46:23.698 INFO:tasks.workunit.client.0.smithi057.stderr: 77800K .......... .......... .......... .......... .......... 57% 23.1M 2s 2024-04-21T08:46:23.701 INFO:tasks.workunit.client.0.smithi057.stderr: 77850K .......... .......... .......... .......... .......... 57% 21.6M 2s 2024-04-21T08:46:23.703 INFO:tasks.workunit.client.0.smithi057.stderr: 77900K .......... .......... .......... .......... .......... 57% 37.4M 2s 2024-04-21T08:46:23.706 INFO:tasks.workunit.client.0.smithi057.stderr: 77950K .......... .......... .......... .......... .......... 57% 25.6M 2s 2024-04-21T08:46:23.706 INFO:tasks.workunit.client.0.smithi057.stderr: 78000K .......... .......... .......... .......... .......... 57% 52.1M 2s 2024-04-21T08:46:23.708 INFO:tasks.workunit.client.0.smithi057.stderr: 78050K .......... .......... .......... .......... .......... 57% 22.5M 2s 2024-04-21T08:46:23.709 INFO:tasks.workunit.client.0.smithi057.stderr: 78100K .......... .......... .......... .......... .......... 57% 23.3M 2s 2024-04-21T08:46:23.710 INFO:tasks.workunit.client.0.smithi057.stderr: 78150K .......... .......... .......... .......... .......... 57% 42.1M 2s 2024-04-21T08:46:23.712 INFO:tasks.workunit.client.0.smithi057.stderr: 78200K .......... .......... .......... .......... .......... 57% 25.0M 2s 2024-04-21T08:46:23.713 INFO:tasks.workunit.client.0.smithi057.stderr: 78250K .......... .......... .......... .......... .......... 57% 40.7M 2s 2024-04-21T08:46:23.715 INFO:tasks.workunit.client.0.smithi057.stderr: 78300K .......... .......... .......... .......... .......... 57% 26.4M 2s 2024-04-21T08:46:23.716 INFO:tasks.workunit.client.0.smithi057.stderr: 78350K .......... .......... .......... .......... .......... 57% 38.6M 2s 2024-04-21T08:46:23.718 INFO:tasks.workunit.client.0.smithi057.stderr: 78400K .......... .......... .......... .......... .......... 57% 24.5M 2s 2024-04-21T08:46:23.720 INFO:tasks.workunit.client.0.smithi057.stderr: 78450K .......... .......... .......... .......... .......... 57% 24.1M 2s 2024-04-21T08:46:23.721 INFO:tasks.workunit.client.0.smithi057.stderr: 78500K .......... .......... .......... .......... .......... 57% 47.1M 2s 2024-04-21T08:46:23.723 INFO:tasks.workunit.client.0.smithi057.stderr: 78550K .......... .......... .......... .......... .......... 57% 22.4M 2s 2024-04-21T08:46:23.725 INFO:tasks.workunit.client.0.smithi057.stderr: 78600K .......... .......... .......... .......... .......... 57% 24.4M 2s 2024-04-21T08:46:23.727 INFO:tasks.workunit.client.0.smithi057.stderr: 78650K .......... .......... .......... .......... .......... 57% 39.4M 2s 2024-04-21T08:46:23.729 INFO:tasks.workunit.client.0.smithi057.stderr: 78700K .......... .......... .......... .......... .......... 58% 24.2M 2s 2024-04-21T08:46:23.730 INFO:tasks.workunit.client.0.smithi057.stderr: 78750K .......... .......... .......... .......... .......... 58% 47.8M 2s 2024-04-21T08:46:23.732 INFO:tasks.workunit.client.0.smithi057.stderr: 78800K .......... .......... .......... .......... .......... 58% 23.1M 2s 2024-04-21T08:46:23.733 INFO:tasks.workunit.client.0.smithi057.stderr: 78850K .......... .......... .......... .......... .......... 58% 36.6M 2s 2024-04-21T08:46:23.735 INFO:tasks.workunit.client.0.smithi057.stderr: 78900K .......... .......... .......... .......... .......... 58% 28.8M 2s 2024-04-21T08:46:23.737 INFO:tasks.workunit.client.0.smithi057.stderr: 78950K .......... .......... .......... .......... .......... 58% 22.0M 2s 2024-04-21T08:46:23.738 INFO:tasks.workunit.client.0.smithi057.stderr: 79000K .......... .......... .......... .......... .......... 58% 50.2M 2s 2024-04-21T08:46:23.740 INFO:tasks.workunit.client.0.smithi057.stderr: 79050K .......... .......... .......... .......... .......... 58% 23.6M 2s 2024-04-21T08:46:23.742 INFO:tasks.workunit.client.0.smithi057.stderr: 79100K .......... .......... .......... .......... .......... 58% 38.5M 2s 2024-04-21T08:46:23.744 INFO:tasks.workunit.client.0.smithi057.stderr: 79150K .......... .......... .......... .......... .......... 58% 21.1M 2s 2024-04-21T08:46:23.746 INFO:tasks.workunit.client.0.smithi057.stderr: 79200K .......... .......... .......... .......... .......... 58% 24.1M 2s 2024-04-21T08:46:23.746 INFO:tasks.workunit.client.0.smithi057.stderr: 79250K .......... .......... .......... .......... .......... 58% 49.1M 2s 2024-04-21T08:46:23.749 INFO:tasks.workunit.client.0.smithi057.stderr: 79300K .......... .......... .......... .......... .......... 58% 23.5M 2s 2024-04-21T08:46:23.751 INFO:tasks.workunit.client.0.smithi057.stderr: 79350K .......... .......... .......... .......... .......... 58% 43.8M 2s 2024-04-21T08:46:23.752 INFO:tasks.workunit.client.0.smithi057.stderr: 79400K .......... .......... .......... .......... .......... 58% 23.2M 2s 2024-04-21T08:46:23.754 INFO:tasks.workunit.client.0.smithi057.stderr: 79450K .......... .......... .......... .......... .......... 58% 23.4M 2s 2024-04-21T08:46:23.755 INFO:tasks.workunit.client.0.smithi057.stderr: 79500K .......... .......... .......... .......... .......... 58% 45.4M 2s 2024-04-21T08:46:23.757 INFO:tasks.workunit.client.0.smithi057.stderr: 79550K .......... .......... .......... .......... .......... 58% 25.1M 2s 2024-04-21T08:46:23.758 INFO:tasks.workunit.client.0.smithi057.stderr: 79600K .......... .......... .......... .......... .......... 58% 35.2M 2s 2024-04-21T08:46:23.761 INFO:tasks.workunit.client.0.smithi057.stderr: 79650K .......... .......... .......... .......... .......... 58% 23.8M 2s 2024-04-21T08:46:23.763 INFO:tasks.workunit.client.0.smithi057.stderr: 79700K .......... .......... .......... .......... .......... 58% 23.8M 2s 2024-04-21T08:46:23.763 INFO:tasks.workunit.client.0.smithi057.stderr: 79750K .......... .......... .......... .......... .......... 58% 53.2M 2s 2024-04-21T08:46:23.765 INFO:tasks.workunit.client.0.smithi057.stderr: 79800K .......... .......... .......... .......... .......... 58% 25.0M 2s 2024-04-21T08:46:23.767 INFO:tasks.workunit.client.0.smithi057.stderr: 79850K .......... .......... .......... .......... .......... 58% 23.7M 2s 2024-04-21T08:46:23.770 INFO:tasks.workunit.client.0.smithi057.stderr: 79900K .......... .......... .......... .......... .......... 58% 23.6M 2s 2024-04-21T08:46:23.771 INFO:tasks.workunit.client.0.smithi057.stderr: 79950K .......... .......... .......... .......... .......... 58% 33.3M 2s 2024-04-21T08:46:23.772 INFO:tasks.workunit.client.0.smithi057.stderr: 80000K .......... .......... .......... .......... .......... 58% 52.0M 2s 2024-04-21T08:46:23.774 INFO:tasks.workunit.client.0.smithi057.stderr: 80050K .......... .......... .......... .......... .......... 59% 24.3M 2s 2024-04-21T08:46:23.781 INFO:tasks.workunit.client.0.smithi057.stderr: 80100K .......... .......... .......... .......... .......... 59% 24.2M 2s 2024-04-21T08:46:23.781 INFO:tasks.workunit.client.0.smithi057.stderr: 80150K .......... .......... .......... .......... .......... 59% 43.1M 2s 2024-04-21T08:46:23.781 INFO:tasks.workunit.client.0.smithi057.stderr: 80200K .......... .......... .......... .......... .......... 59% 23.7M 2s 2024-04-21T08:46:23.781 INFO:tasks.workunit.client.0.smithi057.stderr: 80250K .......... .......... .......... .......... .......... 59% 42.4M 2s 2024-04-21T08:46:23.782 INFO:tasks.workunit.client.0.smithi057.stderr: 80300K .......... .......... .......... .......... .......... 59% 23.9M 2s 2024-04-21T08:46:23.784 INFO:tasks.workunit.client.0.smithi057.stderr: 80350K .......... .......... .......... .......... .......... 59% 38.5M 2s 2024-04-21T08:46:23.786 INFO:tasks.workunit.client.0.smithi057.stderr: 80400K .......... .......... .......... .......... .......... 59% 24.8M 2s 2024-04-21T08:46:23.788 INFO:tasks.workunit.client.0.smithi057.stderr: 80450K .......... .......... .......... .......... .......... 59% 24.9M 2s 2024-04-21T08:46:23.788 INFO:tasks.workunit.client.0.smithi057.stderr: 80500K .......... .......... .......... .......... .......... 59% 53.4M 2s 2024-04-21T08:46:23.791 INFO:tasks.workunit.client.0.smithi057.stderr: 80550K .......... .......... .......... .......... .......... 59% 19.5M 2s 2024-04-21T08:46:23.793 INFO:tasks.workunit.client.0.smithi057.stderr: 80600K .......... .......... .......... .......... .......... 59% 23.0M 2s 2024-04-21T08:46:23.795 INFO:tasks.workunit.client.0.smithi057.stderr: 80650K .......... .......... .......... .......... .......... 59% 35.6M 2s 2024-04-21T08:46:23.797 INFO:tasks.workunit.client.0.smithi057.stderr: 80700K .......... .......... .......... .......... .......... 59% 25.1M 2s 2024-04-21T08:46:23.799 INFO:tasks.workunit.client.0.smithi057.stderr: 80750K .......... .......... .......... .......... .......... 59% 54.5M 2s 2024-04-21T08:46:23.800 INFO:tasks.workunit.client.0.smithi057.stderr: 80800K .......... .......... .......... .......... .......... 59% 22.1M 2s 2024-04-21T08:46:23.802 INFO:tasks.workunit.client.0.smithi057.stderr: 80850K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-04-21T08:46:23.803 INFO:tasks.workunit.client.0.smithi057.stderr: 80900K .......... .......... .......... .......... .......... 59% 41.3M 2s 2024-04-21T08:46:23.805 INFO:tasks.workunit.client.0.smithi057.stderr: 80950K .......... .......... .......... .......... .......... 59% 23.7M 2s 2024-04-21T08:46:23.807 INFO:tasks.workunit.client.0.smithi057.stderr: 81000K .......... .......... .......... .......... .......... 59% 21.9M 2s 2024-04-21T08:46:23.808 INFO:tasks.workunit.client.0.smithi057.stderr: 81050K .......... .......... .......... .......... .......... 59% 43.1M 2s 2024-04-21T08:46:23.810 INFO:tasks.workunit.client.0.smithi057.stderr: 81100K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-04-21T08:46:23.812 INFO:tasks.workunit.client.0.smithi057.stderr: 81150K .......... .......... .......... .......... .......... 59% 31.8M 2s 2024-04-21T08:46:23.814 INFO:tasks.workunit.client.0.smithi057.stderr: 81200K .......... .......... .......... .......... .......... 59% 22.8M 2s 2024-04-21T08:46:23.816 INFO:tasks.workunit.client.0.smithi057.stderr: 81250K .......... .......... .......... .......... .......... 59% 24.6M 2s 2024-04-21T08:46:23.817 INFO:tasks.workunit.client.0.smithi057.stderr: 81300K .......... .......... .......... .......... .......... 59% 48.8M 2s 2024-04-21T08:46:23.818 INFO:tasks.workunit.client.0.smithi057.stderr: 81350K .......... .......... .......... .......... .......... 59% 29.8M 2s 2024-04-21T08:46:23.820 INFO:tasks.workunit.client.0.smithi057.stderr: 81400K .......... .......... .......... .......... .......... 59% 28.0M 2s 2024-04-21T08:46:23.823 INFO:tasks.workunit.client.0.smithi057.stderr: 81450K .......... .......... .......... .......... .......... 60% 24.4M 2s 2024-04-21T08:46:23.825 INFO:tasks.workunit.client.0.smithi057.stderr: 81500K .......... .......... .......... .......... .......... 60% 22.8M 2s 2024-04-21T08:46:23.825 INFO:tasks.workunit.client.0.smithi057.stderr: 81550K .......... .......... .......... .......... .......... 60% 56.3M 2s 2024-04-21T08:46:23.827 INFO:tasks.workunit.client.0.smithi057.stderr: 81600K .......... .......... .......... .......... .......... 60% 23.5M 2s 2024-04-21T08:46:23.829 INFO:tasks.workunit.client.0.smithi057.stderr: 81650K .......... .......... .......... .......... .......... 60% 41.3M 2s 2024-04-21T08:46:23.831 INFO:tasks.workunit.client.0.smithi057.stderr: 81700K .......... .......... .......... .......... .......... 60% 23.6M 2s 2024-04-21T08:46:23.833 INFO:tasks.workunit.client.0.smithi057.stderr: 81750K .......... .......... .......... .......... .......... 60% 21.8M 2s 2024-04-21T08:46:23.834 INFO:tasks.workunit.client.0.smithi057.stderr: 81800K .......... .......... .......... .......... .......... 60% 60.6M 2s 2024-04-21T08:46:23.836 INFO:tasks.workunit.client.0.smithi057.stderr: 81850K .......... .......... .......... .......... .......... 60% 23.7M 2s 2024-04-21T08:46:23.837 INFO:tasks.workunit.client.0.smithi057.stderr: 81900K .......... .......... .......... .......... .......... 60% 40.7M 2s 2024-04-21T08:46:23.839 INFO:tasks.workunit.client.0.smithi057.stderr: 81950K .......... .......... .......... .......... .......... 60% 25.5M 2s 2024-04-21T08:46:23.841 INFO:tasks.workunit.client.0.smithi057.stderr: 82000K .......... .......... .......... .......... .......... 60% 20.7M 2s 2024-04-21T08:46:23.842 INFO:tasks.workunit.client.0.smithi057.stderr: 82050K .......... .......... .......... .......... .......... 60% 44.4M 2s 2024-04-21T08:46:23.844 INFO:tasks.workunit.client.0.smithi057.stderr: 82100K .......... .......... .......... .......... .......... 60% 24.2M 2s 2024-04-21T08:46:23.846 INFO:tasks.workunit.client.0.smithi057.stderr: 82150K .......... .......... .......... .......... .......... 60% 39.9M 2s 2024-04-21T08:46:23.847 INFO:tasks.workunit.client.0.smithi057.stderr: 82200K .......... .......... .......... .......... .......... 60% 30.6M 2s 2024-04-21T08:46:23.850 INFO:tasks.workunit.client.0.smithi057.stderr: 82250K .......... .......... .......... .......... .......... 60% 19.3M 2s 2024-04-21T08:46:23.850 INFO:tasks.workunit.client.0.smithi057.stderr: 82300K .......... .......... .......... .......... .......... 60% 54.0M 2s 2024-04-21T08:46:23.853 INFO:tasks.workunit.client.0.smithi057.stderr: 82350K .......... .......... .......... .......... .......... 60% 23.1M 2s 2024-04-21T08:46:23.854 INFO:tasks.workunit.client.0.smithi057.stderr: 82400K .......... .......... .......... .......... .......... 60% 36.5M 2s 2024-04-21T08:46:23.857 INFO:tasks.workunit.client.0.smithi057.stderr: 82450K .......... .......... .......... .......... .......... 60% 24.2M 2s 2024-04-21T08:46:23.858 INFO:tasks.workunit.client.0.smithi057.stderr: 82500K .......... .......... .......... .......... .......... 60% 24.4M 2s 2024-04-21T08:46:23.859 INFO:tasks.workunit.client.0.smithi057.stderr: 82550K .......... .......... .......... .......... .......... 60% 46.7M 2s 2024-04-21T08:46:23.861 INFO:tasks.workunit.client.0.smithi057.stderr: 82600K .......... .......... .......... .......... .......... 60% 24.0M 2s 2024-04-21T08:46:23.862 INFO:tasks.workunit.client.0.smithi057.stderr: 82650K .......... .......... .......... .......... .......... 60% 42.7M 2s 2024-04-21T08:46:23.864 INFO:tasks.workunit.client.0.smithi057.stderr: 82700K .......... .......... .......... .......... .......... 60% 23.3M 2s 2024-04-21T08:46:23.867 INFO:tasks.workunit.client.0.smithi057.stderr: 82750K .......... .......... .......... .......... .......... 60% 23.7M 2s 2024-04-21T08:46:23.867 INFO:tasks.workunit.client.0.smithi057.stderr: 82800K .......... .......... .......... .......... .......... 61% 72.1M 2s 2024-04-21T08:46:23.870 INFO:tasks.workunit.client.0.smithi057.stderr: 82850K .......... .......... .......... .......... .......... 61% 20.4M 2s 2024-04-21T08:46:23.871 INFO:tasks.workunit.client.0.smithi057.stderr: 82900K .......... .......... .......... .......... .......... 61% 38.4M 2s 2024-04-21T08:46:23.873 INFO:tasks.workunit.client.0.smithi057.stderr: 82950K .......... .......... .......... .......... .......... 61% 24.8M 2s 2024-04-21T08:46:23.875 INFO:tasks.workunit.client.0.smithi057.stderr: 83000K .......... .......... .......... .......... .......... 61% 21.5M 2s 2024-04-21T08:46:23.877 INFO:tasks.workunit.client.0.smithi057.stderr: 83050K .......... .......... .......... .......... .......... 61% 32.7M 2s 2024-04-21T08:46:23.879 INFO:tasks.workunit.client.0.smithi057.stderr: 83100K .......... .......... .......... .......... .......... 61% 23.6M 2s 2024-04-21T08:46:23.880 INFO:tasks.workunit.client.0.smithi057.stderr: 83150K .......... .......... .......... .......... .......... 61% 41.9M 2s 2024-04-21T08:46:23.882 INFO:tasks.workunit.client.0.smithi057.stderr: 83200K .......... .......... .......... .......... .......... 61% 23.6M 2s 2024-04-21T08:46:23.884 INFO:tasks.workunit.client.0.smithi057.stderr: 83250K .......... .......... .......... .......... .......... 61% 24.0M 2s 2024-04-21T08:46:23.885 INFO:tasks.workunit.client.0.smithi057.stderr: 83300K .......... .......... .......... .......... .......... 61% 44.6M 2s 2024-04-21T08:46:23.888 INFO:tasks.workunit.client.0.smithi057.stderr: 83350K .......... .......... .......... .......... .......... 61% 25.5M 2s 2024-04-21T08:46:23.888 INFO:tasks.workunit.client.0.smithi057.stderr: 83400K .......... .......... .......... .......... .......... 61% 38.7M 2s 2024-04-21T08:46:23.891 INFO:tasks.workunit.client.0.smithi057.stderr: 83450K .......... .......... .......... .......... .......... 61% 20.3M 2s 2024-04-21T08:46:23.893 INFO:tasks.workunit.client.0.smithi057.stderr: 83500K .......... .......... .......... .......... .......... 61% 22.0M 2s 2024-04-21T08:46:23.894 INFO:tasks.workunit.client.0.smithi057.stderr: 83550K .......... .......... .......... .......... .......... 61% 45.3M 2s 2024-04-21T08:46:23.896 INFO:tasks.workunit.client.0.smithi057.stderr: 83600K .......... .......... .......... .......... .......... 61% 23.8M 2s 2024-04-21T08:46:23.897 INFO:tasks.workunit.client.0.smithi057.stderr: 83650K .......... .......... .......... .......... .......... 61% 45.3M 2s 2024-04-21T08:46:23.899 INFO:tasks.workunit.client.0.smithi057.stderr: 83700K .......... .......... .......... .......... .......... 61% 22.6M 2s 2024-04-21T08:46:23.901 INFO:tasks.workunit.client.0.smithi057.stderr: 83750K .......... .......... .......... .......... .......... 61% 25.0M 2s 2024-04-21T08:46:23.902 INFO:tasks.workunit.client.0.smithi057.stderr: 83800K .......... .......... .......... .......... .......... 61% 48.6M 2s 2024-04-21T08:46:23.904 INFO:tasks.workunit.client.0.smithi057.stderr: 83850K .......... .......... .......... .......... .......... 61% 22.9M 2s 2024-04-21T08:46:23.906 INFO:tasks.workunit.client.0.smithi057.stderr: 83900K .......... .......... .......... .......... .......... 61% 35.7M 2s 2024-04-21T08:46:23.908 INFO:tasks.workunit.client.0.smithi057.stderr: 83950K .......... .......... .......... .......... .......... 61% 23.7M 2s 2024-04-21T08:46:23.911 INFO:tasks.workunit.client.0.smithi057.stderr: 84000K .......... .......... .......... .......... .......... 61% 25.1M 2s 2024-04-21T08:46:23.911 INFO:tasks.workunit.client.0.smithi057.stderr: 84050K .......... .......... .......... .......... .......... 61% 51.7M 2s 2024-04-21T08:46:23.913 INFO:tasks.workunit.client.0.smithi057.stderr: 84100K .......... .......... .......... .......... .......... 61% 22.2M 2s 2024-04-21T08:46:23.914 INFO:tasks.workunit.client.0.smithi057.stderr: 84150K .......... .......... .......... .......... .......... 62% 46.4M 2s 2024-04-21T08:46:23.916 INFO:tasks.workunit.client.0.smithi057.stderr: 84200K .......... .......... .......... .......... .......... 62% 22.0M 2s 2024-04-21T08:46:23.919 INFO:tasks.workunit.client.0.smithi057.stderr: 84250K .......... .......... .......... .......... .......... 62% 18.8M 2s 2024-04-21T08:46:23.920 INFO:tasks.workunit.client.0.smithi057.stderr: 84300K .......... .......... .......... .......... .......... 62% 49.8M 2s 2024-04-21T08:46:23.922 INFO:tasks.workunit.client.0.smithi057.stderr: 84350K .......... .......... .......... .......... .......... 62% 23.2M 2s 2024-04-21T08:46:23.924 INFO:tasks.workunit.client.0.smithi057.stderr: 84400K .......... .......... .......... .......... .......... 62% 43.0M 2s 2024-04-21T08:46:23.925 INFO:tasks.workunit.client.0.smithi057.stderr: 84450K .......... .......... .......... .......... .......... 62% 22.9M 2s 2024-04-21T08:46:23.927 INFO:tasks.workunit.client.0.smithi057.stderr: 84500K .......... .......... .......... .......... .......... 62% 24.8M 2s 2024-04-21T08:46:23.928 INFO:tasks.workunit.client.0.smithi057.stderr: 84550K .......... .......... .......... .......... .......... 62% 47.3M 2s 2024-04-21T08:46:23.930 INFO:tasks.workunit.client.0.smithi057.stderr: 84600K .......... .......... .......... .......... .......... 62% 23.4M 2s 2024-04-21T08:46:23.932 INFO:tasks.workunit.client.0.smithi057.stderr: 84650K .......... .......... .......... .......... .......... 62% 36.7M 2s 2024-04-21T08:46:23.934 INFO:tasks.workunit.client.0.smithi057.stderr: 84700K .......... .......... .......... .......... .......... 62% 22.6M 2s 2024-04-21T08:46:23.936 INFO:tasks.workunit.client.0.smithi057.stderr: 84750K .......... .......... .......... .......... .......... 62% 23.3M 2s 2024-04-21T08:46:23.937 INFO:tasks.workunit.client.0.smithi057.stderr: 84800K .......... .......... .......... .......... .......... 62% 56.1M 2s 2024-04-21T08:46:23.939 INFO:tasks.workunit.client.0.smithi057.stderr: 84850K .......... .......... .......... .......... .......... 62% 22.5M 2s 2024-04-21T08:46:23.941 INFO:tasks.workunit.client.0.smithi057.stderr: 84900K .......... .......... .......... .......... .......... 62% 25.0M 2s 2024-04-21T08:46:23.942 INFO:tasks.workunit.client.0.smithi057.stderr: 84950K .......... .......... .......... .......... .......... 62% 40.7M 2s 2024-04-21T08:46:23.944 INFO:tasks.workunit.client.0.smithi057.stderr: 85000K .......... .......... .......... .......... .......... 62% 22.3M 2s 2024-04-21T08:46:23.945 INFO:tasks.workunit.client.0.smithi057.stderr: 85050K .......... .......... .......... .......... .......... 62% 48.3M 2s 2024-04-21T08:46:23.947 INFO:tasks.workunit.client.0.smithi057.stderr: 85100K .......... .......... .......... .......... .......... 62% 23.4M 2s 2024-04-21T08:46:23.949 INFO:tasks.workunit.client.0.smithi057.stderr: 85150K .......... .......... .......... .......... .......... 62% 23.7M 2s 2024-04-21T08:46:23.951 INFO:tasks.workunit.client.0.smithi057.stderr: 85200K .......... .......... .......... .......... .......... 62% 36.8M 2s 2024-04-21T08:46:23.952 INFO:tasks.workunit.client.0.smithi057.stderr: 85250K .......... .......... .......... .......... .......... 62% 26.0M 2s 2024-04-21T08:46:23.953 INFO:tasks.workunit.client.0.smithi057.stderr: 85300K .......... .......... .......... .......... .......... 62% 47.8M 2s 2024-04-21T08:46:23.956 INFO:tasks.workunit.client.0.smithi057.stderr: 85350K .......... .......... .......... .......... .......... 62% 20.9M 2s 2024-04-21T08:46:23.958 INFO:tasks.workunit.client.0.smithi057.stderr: 85400K .......... .......... .......... .......... .......... 62% 24.5M 2s 2024-04-21T08:46:23.959 INFO:tasks.workunit.client.0.smithi057.stderr: 85450K .......... .......... .......... .......... .......... 62% 38.6M 2s 2024-04-21T08:46:23.961 INFO:tasks.workunit.client.0.smithi057.stderr: 85500K .......... .......... .......... .......... .......... 63% 23.9M 2s 2024-04-21T08:46:23.962 INFO:tasks.workunit.client.0.smithi057.stderr: 85550K .......... .......... .......... .......... .......... 63% 44.1M 2s 2024-04-21T08:46:23.964 INFO:tasks.workunit.client.0.smithi057.stderr: 85600K .......... .......... .......... .......... .......... 63% 25.8M 2s 2024-04-21T08:46:23.966 INFO:tasks.workunit.client.0.smithi057.stderr: 85650K .......... .......... .......... .......... .......... 63% 23.7M 2s 2024-04-21T08:46:23.967 INFO:tasks.workunit.client.0.smithi057.stderr: 85700K .......... .......... .......... .......... .......... 63% 38.3M 2s 2024-04-21T08:46:23.970 INFO:tasks.workunit.client.0.smithi057.stderr: 85750K .......... .......... .......... .......... .......... 63% 23.3M 2s 2024-04-21T08:46:23.970 INFO:tasks.workunit.client.0.smithi057.stderr: 85800K .......... .......... .......... .......... .......... 63% 46.7M 2s 2024-04-21T08:46:23.972 INFO:tasks.workunit.client.0.smithi057.stderr: 85850K .......... .......... .......... .......... .......... 63% 25.4M 2s 2024-04-21T08:46:23.975 INFO:tasks.workunit.client.0.smithi057.stderr: 85900K .......... .......... .......... .......... .......... 63% 22.8M 2s 2024-04-21T08:46:23.975 INFO:tasks.workunit.client.0.smithi057.stderr: 85950K .......... .......... .......... .......... .......... 63% 48.7M 2s 2024-04-21T08:46:23.978 INFO:tasks.workunit.client.0.smithi057.stderr: 86000K .......... .......... .......... .......... .......... 63% 24.6M 2s 2024-04-21T08:46:23.980 INFO:tasks.workunit.client.0.smithi057.stderr: 86050K .......... .......... .......... .......... .......... 63% 41.6M 2s 2024-04-21T08:46:23.981 INFO:tasks.workunit.client.0.smithi057.stderr: 86100K .......... .......... .......... .......... .......... 63% 22.7M 2s 2024-04-21T08:46:23.983 INFO:tasks.workunit.client.0.smithi057.stderr: 86150K .......... .......... .......... .......... .......... 63% 23.4M 2s 2024-04-21T08:46:23.984 INFO:tasks.workunit.client.0.smithi057.stderr: 86200K .......... .......... .......... .......... .......... 63% 39.2M 2s 2024-04-21T08:46:23.987 INFO:tasks.workunit.client.0.smithi057.stderr: 86250K .......... .......... .......... .......... .......... 63% 19.1M 2s 2024-04-21T08:46:23.988 INFO:tasks.workunit.client.0.smithi057.stderr: 86300K .......... .......... .......... .......... .......... 63% 29.7M 2s 2024-04-21T08:46:23.990 INFO:tasks.workunit.client.0.smithi057.stderr: 86350K .......... .......... .......... .......... .......... 63% 35.4M 2s 2024-04-21T08:46:23.992 INFO:tasks.workunit.client.0.smithi057.stderr: 86400K .......... .......... .......... .......... .......... 63% 24.1M 2s 2024-04-21T08:46:23.993 INFO:tasks.workunit.client.0.smithi057.stderr: 86450K .......... .......... .......... .......... .......... 63% 40.3M 2s 2024-04-21T08:46:23.995 INFO:tasks.workunit.client.0.smithi057.stderr: 86500K .......... .......... .......... .......... .......... 63% 23.1M 2s 2024-04-21T08:46:23.997 INFO:tasks.workunit.client.0.smithi057.stderr: 86550K .......... .......... .......... .......... .......... 63% 33.3M 2s 2024-04-21T08:46:23.998 INFO:tasks.workunit.client.0.smithi057.stderr: 86600K .......... .......... .......... .......... .......... 63% 42.6M 2s 2024-04-21T08:46:24.000 INFO:tasks.workunit.client.0.smithi057.stderr: 86650K .......... .......... .......... .......... .......... 63% 20.7M 2s 2024-04-21T08:46:24.001 INFO:tasks.workunit.client.0.smithi057.stderr: 86700K .......... .......... .......... .......... .......... 63% 41.4M 2s 2024-04-21T08:46:24.004 INFO:tasks.workunit.client.0.smithi057.stderr: 86750K .......... .......... .......... .......... .......... 63% 22.7M 2s 2024-04-21T08:46:24.006 INFO:tasks.workunit.client.0.smithi057.stderr: 86800K .......... .......... .......... .......... .......... 63% 24.5M 2s 2024-04-21T08:46:24.006 INFO:tasks.workunit.client.0.smithi057.stderr: 86850K .......... .......... .......... .......... .......... 64% 47.4M 2s 2024-04-21T08:46:24.009 INFO:tasks.workunit.client.0.smithi057.stderr: 86900K .......... .......... .......... .......... .......... 64% 22.9M 2s 2024-04-21T08:46:24.010 INFO:tasks.workunit.client.0.smithi057.stderr: 86950K .......... .......... .......... .......... .......... 64% 36.5M 2s 2024-04-21T08:46:24.012 INFO:tasks.workunit.client.0.smithi057.stderr: 87000K .......... .......... .......... .......... .......... 64% 22.6M 2s 2024-04-21T08:46:24.014 INFO:tasks.workunit.client.0.smithi057.stderr: 87050K .......... .......... .......... .......... .......... 64% 27.2M 2s 2024-04-21T08:46:24.015 INFO:tasks.workunit.client.0.smithi057.stderr: 87100K .......... .......... .......... .......... .......... 64% 46.4M 2s 2024-04-21T08:46:24.017 INFO:tasks.workunit.client.0.smithi057.stderr: 87150K .......... .......... .......... .......... .......... 64% 23.5M 2s 2024-04-21T08:46:24.018 INFO:tasks.workunit.client.0.smithi057.stderr: 87200K .......... .......... .......... .......... .......... 64% 39.3M 2s 2024-04-21T08:46:24.021 INFO:tasks.workunit.client.0.smithi057.stderr: 87250K .......... .......... .......... .......... .......... 64% 23.8M 2s 2024-04-21T08:46:24.023 INFO:tasks.workunit.client.0.smithi057.stderr: 87300K .......... .......... .......... .......... .......... 64% 23.6M 2s 2024-04-21T08:46:24.023 INFO:tasks.workunit.client.0.smithi057.stderr: 87350K .......... .......... .......... .......... .......... 64% 55.5M 2s 2024-04-21T08:46:24.025 INFO:tasks.workunit.client.0.smithi057.stderr: 87400K .......... .......... .......... .......... .......... 64% 22.9M 2s 2024-04-21T08:46:24.027 INFO:tasks.workunit.client.0.smithi057.stderr: 87450K .......... .......... .......... .......... .......... 64% 34.8M 2s 2024-04-21T08:46:24.029 INFO:tasks.workunit.client.0.smithi057.stderr: 87500K .......... .......... .......... .......... .......... 64% 25.1M 2s 2024-04-21T08:46:24.031 INFO:tasks.workunit.client.0.smithi057.stderr: 87550K .......... .......... .......... .......... .......... 64% 25.0M 2s 2024-04-21T08:46:24.032 INFO:tasks.workunit.client.0.smithi057.stderr: 87600K .......... .......... .......... .......... .......... 64% 45.0M 2s 2024-04-21T08:46:24.034 INFO:tasks.workunit.client.0.smithi057.stderr: 87650K .......... .......... .......... .......... .......... 64% 24.5M 2s 2024-04-21T08:46:24.036 INFO:tasks.workunit.client.0.smithi057.stderr: 87700K .......... .......... .......... .......... .......... 64% 30.3M 2s 2024-04-21T08:46:24.037 INFO:tasks.workunit.client.0.smithi057.stderr: 87750K .......... .......... .......... .......... .......... 64% 37.4M 2s 2024-04-21T08:46:24.039 INFO:tasks.workunit.client.0.smithi057.stderr: 87800K .......... .......... .......... .......... .......... 64% 18.4M 2s 2024-04-21T08:46:24.040 INFO:tasks.workunit.client.0.smithi057.stderr: 87850K .......... .......... .......... .......... .......... 64% 49.7M 2s 2024-04-21T08:46:24.042 INFO:tasks.workunit.client.0.smithi057.stderr: 87900K .......... .......... .......... .......... .......... 64% 23.2M 2s 2024-04-21T08:46:24.044 INFO:tasks.workunit.client.0.smithi057.stderr: 87950K .......... .......... .......... .......... .......... 64% 43.2M 2s 2024-04-21T08:46:24.046 INFO:tasks.workunit.client.0.smithi057.stderr: 88000K .......... .......... .......... .......... .......... 64% 22.8M 2s 2024-04-21T08:46:24.048 INFO:tasks.workunit.client.0.smithi057.stderr: 88050K .......... .......... .......... .......... .......... 64% 23.2M 2s 2024-04-21T08:46:24.049 INFO:tasks.workunit.client.0.smithi057.stderr: 88100K .......... .......... .......... .......... .......... 64% 55.6M 2s 2024-04-21T08:46:24.051 INFO:tasks.workunit.client.0.smithi057.stderr: 88150K .......... .......... .......... .......... .......... 64% 22.7M 2s 2024-04-21T08:46:24.053 INFO:tasks.workunit.client.0.smithi057.stderr: 88200K .......... .......... .......... .......... .......... 65% 31.2M 2s 2024-04-21T08:46:24.054 INFO:tasks.workunit.client.0.smithi057.stderr: 88250K .......... .......... .......... .......... .......... 65% 28.7M 2s 2024-04-21T08:46:24.056 INFO:tasks.workunit.client.0.smithi057.stderr: 88300K .......... .......... .......... .......... .......... 65% 25.1M 2s 2024-04-21T08:46:24.057 INFO:tasks.workunit.client.0.smithi057.stderr: 88350K .......... .......... .......... .......... .......... 65% 45.2M 2s 2024-04-21T08:46:24.059 INFO:tasks.workunit.client.0.smithi057.stderr: 88400K .......... .......... .......... .......... .......... 65% 24.7M 2s 2024-04-21T08:46:24.060 INFO:tasks.workunit.client.0.smithi057.stderr: 88450K .......... .......... .......... .......... .......... 65% 40.1M 2s 2024-04-21T08:46:24.062 INFO:tasks.workunit.client.0.smithi057.stderr: 88500K .......... .......... .......... .......... .......... 65% 23.4M 2s 2024-04-21T08:46:24.065 INFO:tasks.workunit.client.0.smithi057.stderr: 88550K .......... .......... .......... .......... .......... 65% 23.3M 2s 2024-04-21T08:46:24.066 INFO:tasks.workunit.client.0.smithi057.stderr: 88600K .......... .......... .......... .......... .......... 65% 48.3M 2s 2024-04-21T08:46:24.068 INFO:tasks.workunit.client.0.smithi057.stderr: 88650K .......... .......... .......... .......... .......... 65% 25.2M 2s 2024-04-21T08:46:24.069 INFO:tasks.workunit.client.0.smithi057.stderr: 88700K .......... .......... .......... .......... .......... 65% 37.1M 2s 2024-04-21T08:46:24.071 INFO:tasks.workunit.client.0.smithi057.stderr: 88750K .......... .......... .......... .......... .......... 65% 22.3M 2s 2024-04-21T08:46:24.073 INFO:tasks.workunit.client.0.smithi057.stderr: 88800K .......... .......... .......... .......... .......... 65% 23.7M 2s 2024-04-21T08:46:24.074 INFO:tasks.workunit.client.0.smithi057.stderr: 88850K .......... .......... .......... .......... .......... 65% 52.6M 2s 2024-04-21T08:46:24.076 INFO:tasks.workunit.client.0.smithi057.stderr: 88900K .......... .......... .......... .......... .......... 65% 23.4M 2s 2024-04-21T08:46:24.078 INFO:tasks.workunit.client.0.smithi057.stderr: 88950K .......... .......... .......... .......... .......... 65% 27.1M 2s 2024-04-21T08:46:24.080 INFO:tasks.workunit.client.0.smithi057.stderr: 89000K .......... .......... .......... .......... .......... 65% 31.8M 2s 2024-04-21T08:46:24.081 INFO:tasks.workunit.client.0.smithi057.stderr: 89050K .......... .......... .......... .......... .......... 65% 25.1M 2s 2024-04-21T08:46:24.082 INFO:tasks.workunit.client.0.smithi057.stderr: 89100K .......... .......... .......... .......... .......... 65% 43.5M 2s 2024-04-21T08:46:24.085 INFO:tasks.workunit.client.0.smithi057.stderr: 89150K .......... .......... .......... .......... .......... 65% 23.8M 2s 2024-04-21T08:46:24.086 INFO:tasks.workunit.client.0.smithi057.stderr: 89200K .......... .......... .......... .......... .......... 65% 31.3M 2s 2024-04-21T08:46:24.088 INFO:tasks.workunit.client.0.smithi057.stderr: 89250K .......... .......... .......... .......... .......... 65% 26.8M 2s 2024-04-21T08:46:24.090 INFO:tasks.workunit.client.0.smithi057.stderr: 89300K .......... .......... .......... .......... .......... 65% 21.4M 2s 2024-04-21T08:46:24.091 INFO:tasks.workunit.client.0.smithi057.stderr: 89350K .......... .......... .......... .......... .......... 65% 44.9M 2s 2024-04-21T08:46:24.093 INFO:tasks.workunit.client.0.smithi057.stderr: 89400K .......... .......... .......... .......... .......... 65% 24.6M 2s 2024-04-21T08:46:24.094 INFO:tasks.workunit.client.0.smithi057.stderr: 89450K .......... .......... .......... .......... .......... 65% 40.2M 2s 2024-04-21T08:46:24.097 INFO:tasks.workunit.client.0.smithi057.stderr: 89500K .......... .......... .......... .......... .......... 65% 23.8M 2s 2024-04-21T08:46:24.099 INFO:tasks.workunit.client.0.smithi057.stderr: 89550K .......... .......... .......... .......... .......... 66% 23.9M 2s 2024-04-21T08:46:24.100 INFO:tasks.workunit.client.0.smithi057.stderr: 89600K .......... .......... .......... .......... .......... 66% 53.5M 2s 2024-04-21T08:46:24.102 INFO:tasks.workunit.client.0.smithi057.stderr: 89650K .......... .......... .......... .......... .......... 66% 19.6M 2s 2024-04-21T08:46:24.104 INFO:tasks.workunit.client.0.smithi057.stderr: 89700K .......... .......... .......... .......... .......... 66% 35.6M 2s 2024-04-21T08:46:24.105 INFO:tasks.workunit.client.0.smithi057.stderr: 89750K .......... .......... .......... .......... .......... 66% 23.1M 2s 2024-04-21T08:46:24.107 INFO:tasks.workunit.client.0.smithi057.stderr: 89800K .......... .......... .......... .......... .......... 66% 26.9M 2s 2024-04-21T08:46:24.109 INFO:tasks.workunit.client.0.smithi057.stderr: 89850K .......... .......... .......... .......... .......... 66% 37.8M 2s 2024-04-21T08:46:24.111 INFO:tasks.workunit.client.0.smithi057.stderr: 89900K .......... .......... .......... .......... .......... 66% 21.9M 2s 2024-04-21T08:46:24.113 INFO:tasks.workunit.client.0.smithi057.stderr: 89950K .......... .......... .......... .......... .......... 66% 46.7M 2s 2024-04-21T08:46:24.114 INFO:tasks.workunit.client.0.smithi057.stderr: 90000K .......... .......... .......... .......... .......... 66% 22.0M 2s 2024-04-21T08:46:24.116 INFO:tasks.workunit.client.0.smithi057.stderr: 90050K .......... .......... .......... .......... .......... 66% 25.0M 2s 2024-04-21T08:46:24.117 INFO:tasks.workunit.client.0.smithi057.stderr: 90100K .......... .......... .......... .......... .......... 66% 45.3M 2s 2024-04-21T08:46:24.119 INFO:tasks.workunit.client.0.smithi057.stderr: 90150K .......... .......... .......... .......... .......... 66% 24.5M 2s 2024-04-21T08:46:24.121 INFO:tasks.workunit.client.0.smithi057.stderr: 90200K .......... .......... .......... .......... .......... 66% 24.6M 2s 2024-04-21T08:46:24.122 INFO:tasks.workunit.client.0.smithi057.stderr: 90250K .......... .......... .......... .......... .......... 66% 41.4M 2s 2024-04-21T08:46:24.124 INFO:tasks.workunit.client.0.smithi057.stderr: 90300K .......... .......... .......... .......... .......... 66% 25.6M 2s 2024-04-21T08:46:24.126 INFO:tasks.workunit.client.0.smithi057.stderr: 90350K .......... .......... .......... .......... .......... 66% 39.8M 2s 2024-04-21T08:46:24.127 INFO:tasks.workunit.client.0.smithi057.stderr: 90400K .......... .......... .......... .......... .......... 66% 24.4M 2s 2024-04-21T08:46:24.130 INFO:tasks.workunit.client.0.smithi057.stderr: 90450K .......... .......... .......... .......... .......... 66% 23.5M 2s 2024-04-21T08:46:24.131 INFO:tasks.workunit.client.0.smithi057.stderr: 90500K .......... .......... .......... .......... .......... 66% 37.9M 2s 2024-04-21T08:46:24.133 INFO:tasks.workunit.client.0.smithi057.stderr: 90550K .......... .......... .......... .......... .......... 66% 24.3M 2s 2024-04-21T08:46:24.134 INFO:tasks.workunit.client.0.smithi057.stderr: 90600K .......... .......... .......... .......... .......... 66% 53.8M 2s 2024-04-21T08:46:24.136 INFO:tasks.workunit.client.0.smithi057.stderr: 90650K .......... .......... .......... .......... .......... 66% 23.6M 2s 2024-04-21T08:46:24.138 INFO:tasks.workunit.client.0.smithi057.stderr: 90700K .......... .......... .......... .......... .......... 66% 23.9M 2s 2024-04-21T08:46:24.139 INFO:tasks.workunit.client.0.smithi057.stderr: 90750K .......... .......... .......... .......... .......... 66% 37.6M 2s 2024-04-21T08:46:24.141 INFO:tasks.workunit.client.0.smithi057.stderr: 90800K .......... .......... .......... .......... .......... 66% 25.2M 2s 2024-04-21T08:46:24.142 INFO:tasks.workunit.client.0.smithi057.stderr: 90850K .......... .......... .......... .......... .......... 66% 49.4M 2s 2024-04-21T08:46:24.144 INFO:tasks.workunit.client.0.smithi057.stderr: 90900K .......... .......... .......... .......... .......... 66% 22.9M 2s 2024-04-21T08:46:24.146 INFO:tasks.workunit.client.0.smithi057.stderr: 90950K .......... .......... .......... .......... .......... 67% 23.5M 2s 2024-04-21T08:46:24.148 INFO:tasks.workunit.client.0.smithi057.stderr: 91000K .......... .......... .......... .......... .......... 67% 36.7M 2s 2024-04-21T08:46:24.150 INFO:tasks.workunit.client.0.smithi057.stderr: 91050K .......... .......... .......... .......... .......... 67% 22.8M 2s 2024-04-21T08:46:24.152 INFO:tasks.workunit.client.0.smithi057.stderr: 91100K .......... .......... .......... .......... .......... 67% 55.1M 2s 2024-04-21T08:46:24.153 INFO:tasks.workunit.client.0.smithi057.stderr: 91150K .......... .......... .......... .......... .......... 67% 23.3M 2s 2024-04-21T08:46:24.155 INFO:tasks.workunit.client.0.smithi057.stderr: 91200K .......... .......... .......... .......... .......... 67% 24.5M 2s 2024-04-21T08:46:24.156 INFO:tasks.workunit.client.0.smithi057.stderr: 91250K .......... .......... .......... .......... .......... 67% 40.8M 2s 2024-04-21T08:46:24.158 INFO:tasks.workunit.client.0.smithi057.stderr: 91300K .......... .......... .......... .......... .......... 67% 24.6M 2s 2024-04-21T08:46:24.160 INFO:tasks.workunit.client.0.smithi057.stderr: 91350K .......... .......... .......... .......... .......... 67% 51.1M 2s 2024-04-21T08:46:24.161 INFO:tasks.workunit.client.0.smithi057.stderr: 91400K .......... .......... .......... .......... .......... 67% 22.6M 2s 2024-04-21T08:46:24.163 INFO:tasks.workunit.client.0.smithi057.stderr: 91450K .......... .......... .......... .......... .......... 67% 24.1M 2s 2024-04-21T08:46:24.164 INFO:tasks.workunit.client.0.smithi057.stderr: 91500K .......... .......... .......... .......... .......... 67% 40.4M 2s 2024-04-21T08:46:24.166 INFO:tasks.workunit.client.0.smithi057.stderr: 91550K .......... .......... .......... .......... .......... 67% 23.7M 2s 2024-04-21T08:46:24.173 INFO:tasks.workunit.client.0.smithi057.stderr: 91600K .......... .......... .......... .......... .......... 67% 9.60M 2s 2024-04-21T08:46:24.173 INFO:tasks.workunit.client.0.smithi057.stderr: 91650K .......... .......... .......... .......... .......... 67% 134M 2s 2024-04-21T08:46:24.173 INFO:tasks.workunit.client.0.smithi057.stderr: 91700K .......... .......... .......... .......... .......... 67% 124M 2s 2024-04-21T08:46:24.173 INFO:tasks.workunit.client.0.smithi057.stderr: 91750K .......... .......... .......... .......... .......... 67% 41.3M 2s 2024-04-21T08:46:24.176 INFO:tasks.workunit.client.0.smithi057.stderr: 91800K .......... .......... .......... .......... .......... 67% 21.6M 2s 2024-04-21T08:46:24.178 INFO:tasks.workunit.client.0.smithi057.stderr: 91850K .......... .......... .......... .......... .......... 67% 24.7M 2s 2024-04-21T08:46:24.178 INFO:tasks.workunit.client.0.smithi057.stderr: 91900K .......... .......... .......... .......... .......... 67% 50.1M 2s 2024-04-21T08:46:24.180 INFO:tasks.workunit.client.0.smithi057.stderr: 91950K .......... .......... .......... .......... .......... 67% 24.9M 2s 2024-04-21T08:46:24.182 INFO:tasks.workunit.client.0.smithi057.stderr: 92000K .......... .......... .......... .......... .......... 67% 43.3M 2s 2024-04-21T08:46:24.184 INFO:tasks.workunit.client.0.smithi057.stderr: 92050K .......... .......... .......... .......... .......... 67% 20.1M 2s 2024-04-21T08:46:24.186 INFO:tasks.workunit.client.0.smithi057.stderr: 92100K .......... .......... .......... .......... .......... 67% 21.8M 2s 2024-04-21T08:46:24.187 INFO:tasks.workunit.client.0.smithi057.stderr: 92150K .......... .......... .......... .......... .......... 67% 45.9M 2s 2024-04-21T08:46:24.189 INFO:tasks.workunit.client.0.smithi057.stderr: 92200K .......... .......... .......... .......... .......... 67% 25.3M 2s 2024-04-21T08:46:24.192 INFO:tasks.workunit.client.0.smithi057.stderr: 92250K .......... .......... .......... .......... .......... 67% 36.5M 2s 2024-04-21T08:46:24.193 INFO:tasks.workunit.client.0.smithi057.stderr: 92300K .......... .......... .......... .......... .......... 68% 20.8M 2s 2024-04-21T08:46:24.195 INFO:tasks.workunit.client.0.smithi057.stderr: 92350K .......... .......... .......... .......... .......... 68% 29.6M 2s 2024-04-21T08:46:24.195 INFO:tasks.workunit.client.0.smithi057.stderr: 92400K .......... .......... .......... .......... .......... 68% 49.8M 2s 2024-04-21T08:46:24.198 INFO:tasks.workunit.client.0.smithi057.stderr: 92450K .......... .......... .......... .......... .......... 68% 24.4M 2s 2024-04-21T08:46:24.199 INFO:tasks.workunit.client.0.smithi057.stderr: 92500K .......... .......... .......... .......... .......... 68% 35.3M 2s 2024-04-21T08:46:24.201 INFO:tasks.workunit.client.0.smithi057.stderr: 92550K .......... .......... .......... .......... .......... 68% 23.8M 2s 2024-04-21T08:46:24.202 INFO:tasks.workunit.client.0.smithi057.stderr: 92600K .......... .......... .......... .......... .......... 68% 45.4M 2s 2024-04-21T08:46:24.204 INFO:tasks.workunit.client.0.smithi057.stderr: 92650K .......... .......... .......... .......... .......... 68% 27.6M 2s 2024-04-21T08:46:24.206 INFO:tasks.workunit.client.0.smithi057.stderr: 92700K .......... .......... .......... .......... .......... 68% 22.8M 2s 2024-04-21T08:46:24.207 INFO:tasks.workunit.client.0.smithi057.stderr: 92750K .......... .......... .......... .......... .......... 68% 38.2M 2s 2024-04-21T08:46:24.209 INFO:tasks.workunit.client.0.smithi057.stderr: 92800K .......... .......... .......... .......... .......... 68% 25.5M 2s 2024-04-21T08:46:24.212 INFO:tasks.workunit.client.0.smithi057.stderr: 92850K .......... .......... .......... .......... .......... 68% 19.5M 2s 2024-04-21T08:46:24.213 INFO:tasks.workunit.client.0.smithi057.stderr: 92900K .......... .......... .......... .......... .......... 68% 53.3M 1s 2024-04-21T08:46:24.216 INFO:tasks.workunit.client.0.smithi057.stderr: 92950K .......... .......... .......... .......... .......... 68% 20.9M 1s 2024-04-21T08:46:24.217 INFO:tasks.workunit.client.0.smithi057.stderr: 93000K .......... .......... .......... .......... .......... 68% 25.1M 1s 2024-04-21T08:46:24.218 INFO:tasks.workunit.client.0.smithi057.stderr: 93050K .......... .......... .......... .......... .......... 68% 36.1M 1s 2024-04-21T08:46:24.221 INFO:tasks.workunit.client.0.smithi057.stderr: 93100K .......... .......... .......... .......... .......... 68% 23.9M 1s 2024-04-21T08:46:24.221 INFO:tasks.workunit.client.0.smithi057.stderr: 93150K .......... .......... .......... .......... .......... 68% 53.1M 1s 2024-04-21T08:46:24.223 INFO:tasks.workunit.client.0.smithi057.stderr: 93200K .......... .......... .......... .......... .......... 68% 22.8M 1s 2024-04-21T08:46:24.225 INFO:tasks.workunit.client.0.smithi057.stderr: 93250K .......... .......... .......... .......... .......... 68% 43.0M 1s 2024-04-21T08:46:24.227 INFO:tasks.workunit.client.0.smithi057.stderr: 93300K .......... .......... .......... .......... .......... 68% 21.7M 1s 2024-04-21T08:46:24.229 INFO:tasks.workunit.client.0.smithi057.stderr: 93350K .......... .......... .......... .......... .......... 68% 25.7M 1s 2024-04-21T08:46:24.230 INFO:tasks.workunit.client.0.smithi057.stderr: 93400K .......... .......... .......... .......... .......... 68% 45.5M 1s 2024-04-21T08:46:24.232 INFO:tasks.workunit.client.0.smithi057.stderr: 93450K .......... .......... .......... .......... .......... 68% 20.1M 1s 2024-04-21T08:46:24.233 INFO:tasks.workunit.client.0.smithi057.stderr: 93500K .......... .......... .......... .......... .......... 68% 44.7M 1s 2024-04-21T08:46:24.235 INFO:tasks.workunit.client.0.smithi057.stderr: 93550K .......... .......... .......... .......... .......... 68% 22.9M 1s 2024-04-21T08:46:24.238 INFO:tasks.workunit.client.0.smithi057.stderr: 93600K .......... .......... .......... .......... .......... 68% 23.9M 1s 2024-04-21T08:46:24.238 INFO:tasks.workunit.client.0.smithi057.stderr: 93650K .......... .......... .......... .......... .......... 69% 45.3M 1s 2024-04-21T08:46:24.241 INFO:tasks.workunit.client.0.smithi057.stderr: 93700K .......... .......... .......... .......... .......... 69% 25.6M 1s 2024-04-21T08:46:24.242 INFO:tasks.workunit.client.0.smithi057.stderr: 93750K .......... .......... .......... .......... .......... 69% 40.4M 1s 2024-04-21T08:46:24.244 INFO:tasks.workunit.client.0.smithi057.stderr: 93800K .......... .......... .......... .......... .......... 69% 23.6M 1s 2024-04-21T08:46:24.247 INFO:tasks.workunit.client.0.smithi057.stderr: 93850K .......... .......... .......... .......... .......... 69% 22.9M 1s 2024-04-21T08:46:24.247 INFO:tasks.workunit.client.0.smithi057.stderr: 93900K .......... .......... .......... .......... .......... 69% 50.0M 1s 2024-04-21T08:46:24.249 INFO:tasks.workunit.client.0.smithi057.stderr: 93950K .......... .......... .......... .......... .......... 69% 23.3M 1s 2024-04-21T08:46:24.250 INFO:tasks.workunit.client.0.smithi057.stderr: 94000K .......... .......... .......... .......... .......... 69% 41.1M 1s 2024-04-21T08:46:24.252 INFO:tasks.workunit.client.0.smithi057.stderr: 94050K .......... .......... .......... .......... .......... 69% 25.0M 1s 2024-04-21T08:46:24.254 INFO:tasks.workunit.client.0.smithi057.stderr: 94100K .......... .......... .......... .......... .......... 69% 21.9M 1s 2024-04-21T08:46:24.255 INFO:tasks.workunit.client.0.smithi057.stderr: 94150K .......... .......... .......... .......... .......... 69% 55.3M 1s 2024-04-21T08:46:24.257 INFO:tasks.workunit.client.0.smithi057.stderr: 94200K .......... .......... .......... .......... .......... 69% 22.8M 1s 2024-04-21T08:46:24.259 INFO:tasks.workunit.client.0.smithi057.stderr: 94250K .......... .......... .......... .......... .......... 69% 38.8M 1s 2024-04-21T08:46:24.261 INFO:tasks.workunit.client.0.smithi057.stderr: 94300K .......... .......... .......... .......... .......... 69% 20.8M 1s 2024-04-21T08:46:24.263 INFO:tasks.workunit.client.0.smithi057.stderr: 94350K .......... .......... .......... .......... .......... 69% 28.8M 1s 2024-04-21T08:46:24.264 INFO:tasks.workunit.client.0.smithi057.stderr: 94400K .......... .......... .......... .......... .......... 69% 44.6M 1s 2024-04-21T08:46:24.266 INFO:tasks.workunit.client.0.smithi057.stderr: 94450K .......... .......... .......... .......... .......... 69% 24.5M 1s 2024-04-21T08:46:24.268 INFO:tasks.workunit.client.0.smithi057.stderr: 94500K .......... .......... .......... .......... .......... 69% 27.7M 1s 2024-04-21T08:46:24.269 INFO:tasks.workunit.client.0.smithi057.stderr: 94550K .......... .......... .......... .......... .......... 69% 29.5M 1s 2024-04-21T08:46:24.271 INFO:tasks.workunit.client.0.smithi057.stderr: 94600K .......... .......... .......... .......... .......... 69% 27.0M 1s 2024-04-21T08:46:24.272 INFO:tasks.workunit.client.0.smithi057.stderr: 94650K .......... .......... .......... .......... .......... 69% 39.8M 1s 2024-04-21T08:46:24.274 INFO:tasks.workunit.client.0.smithi057.stderr: 94700K .......... .......... .......... .......... .......... 69% 24.0M 1s 2024-04-21T08:46:24.276 INFO:tasks.workunit.client.0.smithi057.stderr: 94750K .......... .......... .......... .......... .......... 69% 38.2M 1s 2024-04-21T08:46:24.277 INFO:tasks.workunit.client.0.smithi057.stderr: 94800K .......... .......... .......... .......... .......... 69% 24.5M 1s 2024-04-21T08:46:24.280 INFO:tasks.workunit.client.0.smithi057.stderr: 94850K .......... .......... .......... .......... .......... 69% 24.0M 1s 2024-04-21T08:46:24.280 INFO:tasks.workunit.client.0.smithi057.stderr: 94900K .......... .......... .......... .......... .......... 69% 49.0M 1s 2024-04-21T08:46:24.283 INFO:tasks.workunit.client.0.smithi057.stderr: 94950K .......... .......... .......... .......... .......... 69% 19.8M 1s 2024-04-21T08:46:24.285 INFO:tasks.workunit.client.0.smithi057.stderr: 95000K .......... .......... .......... .......... .......... 70% 44.4M 1s 2024-04-21T08:46:24.287 INFO:tasks.workunit.client.0.smithi057.stderr: 95050K .......... .......... .......... .......... .......... 70% 22.5M 1s 2024-04-21T08:46:24.289 INFO:tasks.workunit.client.0.smithi057.stderr: 95100K .......... .......... .......... .......... .......... 70% 26.6M 1s 2024-04-21T08:46:24.289 INFO:tasks.workunit.client.0.smithi057.stderr: 95150K .......... .......... .......... .......... .......... 70% 43.4M 1s 2024-04-21T08:46:24.291 INFO:tasks.workunit.client.0.smithi057.stderr: 95200K .......... .......... .......... .......... .......... 70% 24.9M 1s 2024-04-21T08:46:24.293 INFO:tasks.workunit.client.0.smithi057.stderr: 95250K .......... .......... .......... .......... .......... 70% 37.9M 1s 2024-04-21T08:46:24.294 INFO:tasks.workunit.client.0.smithi057.stderr: 95300K .......... .......... .......... .......... .......... 70% 22.9M 1s 2024-04-21T08:46:24.297 INFO:tasks.workunit.client.0.smithi057.stderr: 95350K .......... .......... .......... .......... .......... 70% 25.1M 1s 2024-04-21T08:46:24.297 INFO:tasks.workunit.client.0.smithi057.stderr: 95400K .......... .......... .......... .......... .......... 70% 49.2M 1s 2024-04-21T08:46:24.300 INFO:tasks.workunit.client.0.smithi057.stderr: 95450K .......... .......... .......... .......... .......... 70% 23.5M 1s 2024-04-21T08:46:24.302 INFO:tasks.workunit.client.0.smithi057.stderr: 95500K .......... .......... .......... .......... .......... 70% 23.3M 1s 2024-04-21T08:46:24.303 INFO:tasks.workunit.client.0.smithi057.stderr: 95550K .......... .......... .......... .......... .......... 70% 40.2M 1s 2024-04-21T08:46:24.305 INFO:tasks.workunit.client.0.smithi057.stderr: 95600K .......... .......... .......... .......... .......... 70% 23.7M 1s 2024-04-21T08:46:24.306 INFO:tasks.workunit.client.0.smithi057.stderr: 95650K .......... .......... .......... .......... .......... 70% 48.8M 1s 2024-04-21T08:46:24.308 INFO:tasks.workunit.client.0.smithi057.stderr: 95700K .......... .......... .......... .......... .......... 70% 22.3M 1s 2024-04-21T08:46:24.310 INFO:tasks.workunit.client.0.smithi057.stderr: 95750K .......... .......... .......... .......... .......... 70% 25.2M 1s 2024-04-21T08:46:24.311 INFO:tasks.workunit.client.0.smithi057.stderr: 95800K .......... .......... .......... .......... .......... 70% 41.4M 1s 2024-04-21T08:46:24.314 INFO:tasks.workunit.client.0.smithi057.stderr: 95850K .......... .......... .......... .......... .......... 70% 21.8M 1s 2024-04-21T08:46:24.314 INFO:tasks.workunit.client.0.smithi057.stderr: 95900K .......... .......... .......... .......... .......... 70% 56.2M 1s 2024-04-21T08:46:24.317 INFO:tasks.workunit.client.0.smithi057.stderr: 95950K .......... .......... .......... .......... .......... 70% 23.0M 1s 2024-04-21T08:46:24.319 INFO:tasks.workunit.client.0.smithi057.stderr: 96000K .......... .......... .......... .......... .......... 70% 24.7M 1s 2024-04-21T08:46:24.320 INFO:tasks.workunit.client.0.smithi057.stderr: 96050K .......... .......... .......... .......... .......... 70% 37.9M 1s 2024-04-21T08:46:24.322 INFO:tasks.workunit.client.0.smithi057.stderr: 96100K .......... .......... .......... .......... .......... 70% 24.0M 1s 2024-04-21T08:46:24.323 INFO:tasks.workunit.client.0.smithi057.stderr: 96150K .......... .......... .......... .......... .......... 70% 54.0M 1s 2024-04-21T08:46:24.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:24 smithi129 ceph-mon[31272]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T08:46:24.325 INFO:tasks.workunit.client.0.smithi057.stderr: 96200K .......... .......... .......... .......... .......... 70% 23.0M 1s 2024-04-21T08:46:24.327 INFO:tasks.workunit.client.0.smithi057.stderr: 96250K .......... .......... .......... .......... .......... 70% 23.8M 1s 2024-04-21T08:46:24.328 INFO:tasks.workunit.client.0.smithi057.stderr: 96300K .......... .......... .......... .......... .......... 70% 43.3M 1s 2024-04-21T08:46:24.330 INFO:tasks.workunit.client.0.smithi057.stderr: 96350K .......... .......... .......... .......... .......... 71% 24.1M 1s 2024-04-21T08:46:24.332 INFO:tasks.workunit.client.0.smithi057.stderr: 96400K .......... .......... .......... .......... .......... 71% 49.4M 1s 2024-04-21T08:46:24.333 INFO:tasks.workunit.client.0.smithi057.stderr: 96450K .......... .......... .......... .......... .......... 71% 21.6M 1s 2024-04-21T08:46:24.335 INFO:tasks.workunit.client.0.smithi057.stderr: 96500K .......... .......... .......... .......... .......... 71% 23.4M 1s 2024-04-21T08:46:24.337 INFO:tasks.workunit.client.0.smithi057.stderr: 96550K .......... .......... .......... .......... .......... 71% 35.3M 1s 2024-04-21T08:46:24.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:24 smithi057 ceph-mon[27968]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T08:46:24.339 INFO:tasks.workunit.client.0.smithi057.stderr: 96600K .......... .......... .......... .......... .......... 71% 24.6M 1s 2024-04-21T08:46:24.341 INFO:tasks.workunit.client.0.smithi057.stderr: 96650K .......... .......... .......... .......... .......... 71% 57.7M 1s 2024-04-21T08:46:24.342 INFO:tasks.workunit.client.0.smithi057.stderr: 96700K .......... .......... .......... .......... .......... 71% 23.1M 1s 2024-04-21T08:46:24.344 INFO:tasks.workunit.client.0.smithi057.stderr: 96750K .......... .......... .......... .......... .......... 71% 24.2M 1s 2024-04-21T08:46:24.345 INFO:tasks.workunit.client.0.smithi057.stderr: 96800K .......... .......... .......... .......... .......... 71% 34.3M 1s 2024-04-21T08:46:24.347 INFO:tasks.workunit.client.0.smithi057.stderr: 96850K .......... .......... .......... .......... .......... 71% 25.7M 1s 2024-04-21T08:46:24.349 INFO:tasks.workunit.client.0.smithi057.stderr: 96900K .......... .......... .......... .......... .......... 71% 24.6M 1s 2024-04-21T08:46:24.350 INFO:tasks.workunit.client.0.smithi057.stderr: 96950K .......... .......... .......... .......... .......... 71% 47.4M 1s 2024-04-21T08:46:24.352 INFO:tasks.workunit.client.0.smithi057.stderr: 97000K .......... .......... .......... .......... .......... 71% 23.5M 1s 2024-04-21T08:46:24.353 INFO:tasks.workunit.client.0.smithi057.stderr: 97050K .......... .......... .......... .......... .......... 71% 36.6M 1s 2024-04-21T08:46:24.355 INFO:tasks.workunit.client.0.smithi057.stderr: 97100K .......... .......... .......... .......... .......... 71% 25.5M 1s 2024-04-21T08:46:24.358 INFO:tasks.workunit.client.0.smithi057.stderr: 97150K .......... .......... .......... .......... .......... 71% 23.5M 1s 2024-04-21T08:46:24.358 INFO:tasks.workunit.client.0.smithi057.stderr: 97200K .......... .......... .......... .......... .......... 71% 44.2M 1s 2024-04-21T08:46:24.361 INFO:tasks.workunit.client.0.smithi057.stderr: 97250K .......... .......... .......... .......... .......... 71% 25.3M 1s 2024-04-21T08:46:24.362 INFO:tasks.workunit.client.0.smithi057.stderr: 97300K .......... .......... .......... .......... .......... 71% 42.6M 1s 2024-04-21T08:46:24.364 INFO:tasks.workunit.client.0.smithi057.stderr: 97350K .......... .......... .......... .......... .......... 71% 22.2M 1s 2024-04-21T08:46:24.366 INFO:tasks.workunit.client.0.smithi057.stderr: 97400K .......... .......... .......... .......... .......... 71% 24.5M 1s 2024-04-21T08:46:24.367 INFO:tasks.workunit.client.0.smithi057.stderr: 97450K .......... .......... .......... .......... .......... 71% 51.1M 1s 2024-04-21T08:46:24.369 INFO:tasks.workunit.client.0.smithi057.stderr: 97500K .......... .......... .......... .......... .......... 71% 23.9M 1s 2024-04-21T08:46:24.371 INFO:tasks.workunit.client.0.smithi057.stderr: 97550K .......... .......... .......... .......... .......... 71% 34.7M 1s 2024-04-21T08:46:24.372 INFO:tasks.workunit.client.0.smithi057.stderr: 97600K .......... .......... .......... .......... .......... 71% 24.7M 1s 2024-04-21T08:46:24.374 INFO:tasks.workunit.client.0.smithi057.stderr: 97650K .......... .......... .......... .......... .......... 71% 25.0M 1s 2024-04-21T08:46:24.375 INFO:tasks.workunit.client.0.smithi057.stderr: 97700K .......... .......... .......... .......... .......... 72% 42.6M 1s 2024-04-21T08:46:24.377 INFO:tasks.workunit.client.0.smithi057.stderr: 97750K .......... .......... .......... .......... .......... 72% 27.8M 1s 2024-04-21T08:46:24.379 INFO:tasks.workunit.client.0.smithi057.stderr: 97800K .......... .......... .......... .......... .......... 72% 30.2M 1s 2024-04-21T08:46:24.380 INFO:tasks.workunit.client.0.smithi057.stderr: 97850K .......... .......... .......... .......... .......... 72% 27.4M 1s 2024-04-21T08:46:24.383 INFO:tasks.workunit.client.0.smithi057.stderr: 97900K .......... .......... .......... .......... .......... 72% 21.3M 1s 2024-04-21T08:46:24.384 INFO:tasks.workunit.client.0.smithi057.stderr: 97950K .......... .......... .......... .......... .......... 72% 43.1M 1s 2024-04-21T08:46:24.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:24 smithi136 ceph-mon[31791]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T08:46:24.386 INFO:tasks.workunit.client.0.smithi057.stderr: 98000K .......... .......... .......... .......... .......... 72% 24.9M 1s 2024-04-21T08:46:24.387 INFO:tasks.workunit.client.0.smithi057.stderr: 98050K .......... .......... .......... .......... .......... 72% 38.0M 1s 2024-04-21T08:46:24.389 INFO:tasks.workunit.client.0.smithi057.stderr: 98100K .......... .......... .......... .......... .......... 72% 24.7M 1s 2024-04-21T08:46:24.391 INFO:tasks.workunit.client.0.smithi057.stderr: 98150K .......... .......... .......... .......... .......... 72% 23.8M 1s 2024-04-21T08:46:24.392 INFO:tasks.workunit.client.0.smithi057.stderr: 98200K .......... .......... .......... .......... .......... 72% 51.9M 1s 2024-04-21T08:46:24.394 INFO:tasks.workunit.client.0.smithi057.stderr: 98250K .......... .......... .......... .......... .......... 72% 23.8M 1s 2024-04-21T08:46:24.396 INFO:tasks.workunit.client.0.smithi057.stderr: 98300K .......... .......... .......... .......... .......... 72% 34.6M 1s 2024-04-21T08:46:24.398 INFO:tasks.workunit.client.0.smithi057.stderr: 98350K .......... .......... .......... .......... .......... 72% 22.9M 1s 2024-04-21T08:46:24.400 INFO:tasks.workunit.client.0.smithi057.stderr: 98400K .......... .......... .......... .......... .......... 72% 22.7M 1s 2024-04-21T08:46:24.401 INFO:tasks.workunit.client.0.smithi057.stderr: 98450K .......... .......... .......... .......... .......... 72% 50.0M 1s 2024-04-21T08:46:24.403 INFO:tasks.workunit.client.0.smithi057.stderr: 98500K .......... .......... .......... .......... .......... 72% 22.2M 1s 2024-04-21T08:46:24.404 INFO:tasks.workunit.client.0.smithi057.stderr: 98550K .......... .......... .......... .......... .......... 72% 33.6M 1s 2024-04-21T08:46:24.406 INFO:tasks.workunit.client.0.smithi057.stderr: 98600K .......... .......... .......... .......... .......... 72% 24.4M 1s 2024-04-21T08:46:24.409 INFO:tasks.workunit.client.0.smithi057.stderr: 98650K .......... .......... .......... .......... .......... 72% 24.7M 1s 2024-04-21T08:46:24.410 INFO:tasks.workunit.client.0.smithi057.stderr: 98700K .......... .......... .......... .......... .......... 72% 39.0M 1s 2024-04-21T08:46:24.411 INFO:tasks.workunit.client.0.smithi057.stderr: 98750K .......... .......... .......... .......... .......... 72% 30.3M 1s 2024-04-21T08:46:24.413 INFO:tasks.workunit.client.0.smithi057.stderr: 98800K .......... .......... .......... .......... .......... 72% 38.4M 1s 2024-04-21T08:46:24.415 INFO:tasks.workunit.client.0.smithi057.stderr: 98850K .......... .......... .......... .......... .......... 72% 23.3M 1s 2024-04-21T08:46:24.417 INFO:tasks.workunit.client.0.smithi057.stderr: 98900K .......... .......... .......... .......... .......... 72% 23.9M 1s 2024-04-21T08:46:24.418 INFO:tasks.workunit.client.0.smithi057.stderr: 98950K .......... .......... .......... .......... .......... 72% 47.6M 1s 2024-04-21T08:46:24.420 INFO:tasks.workunit.client.0.smithi057.stderr: 99000K .......... .......... .......... .......... .......... 72% 23.5M 1s 2024-04-21T08:46:24.421 INFO:tasks.workunit.client.0.smithi057.stderr: 99050K .......... .......... .......... .......... .......... 73% 40.8M 1s 2024-04-21T08:46:24.423 INFO:tasks.workunit.client.0.smithi057.stderr: 99100K .......... .......... .......... .......... .......... 73% 24.2M 1s 2024-04-21T08:46:24.425 INFO:tasks.workunit.client.0.smithi057.stderr: 99150K .......... .......... .......... .......... .......... 73% 23.8M 1s 2024-04-21T08:46:24.426 INFO:tasks.workunit.client.0.smithi057.stderr: 99200K .......... .......... .......... .......... .......... 73% 48.0M 1s 2024-04-21T08:46:24.428 INFO:tasks.workunit.client.0.smithi057.stderr: 99250K .......... .......... .......... .......... .......... 73% 24.0M 1s 2024-04-21T08:46:24.429 INFO:tasks.workunit.client.0.smithi057.stderr: 99300K .......... .......... .......... .......... .......... 73% 38.8M 1s 2024-04-21T08:46:24.431 INFO:tasks.workunit.client.0.smithi057.stderr: 99350K .......... .......... .......... .......... .......... 73% 23.7M 1s 2024-04-21T08:46:24.434 INFO:tasks.workunit.client.0.smithi057.stderr: 99400K .......... .......... .......... .......... .......... 73% 24.3M 1s 2024-04-21T08:46:24.434 INFO:tasks.workunit.client.0.smithi057.stderr: 99450K .......... .......... .......... .......... .......... 73% 49.4M 1s 2024-04-21T08:46:24.437 INFO:tasks.workunit.client.0.smithi057.stderr: 99500K .......... .......... .......... .......... .......... 73% 22.5M 1s 2024-04-21T08:46:24.438 INFO:tasks.workunit.client.0.smithi057.stderr: 99550K .......... .......... .......... .......... .......... 73% 43.5M 1s 2024-04-21T08:46:24.440 INFO:tasks.workunit.client.0.smithi057.stderr: 99600K .......... .......... .......... .......... .......... 73% 23.4M 1s 2024-04-21T08:46:24.442 INFO:tasks.workunit.client.0.smithi057.stderr: 99650K .......... .......... .......... .......... .......... 73% 24.2M 1s 2024-04-21T08:46:24.443 INFO:tasks.workunit.client.0.smithi057.stderr: 99700K .......... .......... .......... .......... .......... 73% 49.5M 1s 2024-04-21T08:46:24.445 INFO:tasks.workunit.client.0.smithi057.stderr: 99750K .......... .......... .......... .......... .......... 73% 23.1M 1s 2024-04-21T08:46:24.446 INFO:tasks.workunit.client.0.smithi057.stderr: 99800K .......... .......... .......... .......... .......... 73% 39.2M 1s 2024-04-21T08:46:24.448 INFO:tasks.workunit.client.0.smithi057.stderr: 99850K .......... .......... .......... .......... .......... 73% 23.9M 1s 2024-04-21T08:46:24.450 INFO:tasks.workunit.client.0.smithi057.stderr: 99900K .......... .......... .......... .......... .......... 73% 24.3M 1s 2024-04-21T08:46:24.451 INFO:tasks.workunit.client.0.smithi057.stderr: 99950K .......... .......... .......... .......... .......... 73% 49.5M 1s 2024-04-21T08:46:24.453 INFO:tasks.workunit.client.0.smithi057.stderr:100000K .......... .......... .......... .......... .......... 73% 23.3M 1s 2024-04-21T08:46:24.455 INFO:tasks.workunit.client.0.smithi057.stderr:100050K .......... .......... .......... .......... .......... 73% 23.6M 1s 2024-04-21T08:46:24.456 INFO:tasks.workunit.client.0.smithi057.stderr:100100K .......... .......... .......... .......... .......... 73% 46.0M 1s 2024-04-21T08:46:24.459 INFO:tasks.workunit.client.0.smithi057.stderr:100150K .......... .......... .......... .......... .......... 73% 22.3M 1s 2024-04-21T08:46:24.459 INFO:tasks.workunit.client.0.smithi057.stderr:100200K .......... .......... .......... .......... .......... 73% 52.6M 1s 2024-04-21T08:46:24.462 INFO:tasks.workunit.client.0.smithi057.stderr:100250K .......... .......... .......... .......... .......... 73% 22.9M 1s 2024-04-21T08:46:24.464 INFO:tasks.workunit.client.0.smithi057.stderr:100300K .......... .......... .......... .......... .......... 73% 26.7M 1s 2024-04-21T08:46:24.465 INFO:tasks.workunit.client.0.smithi057.stderr:100350K .......... .......... .......... .......... .......... 73% 31.2M 1s 2024-04-21T08:46:24.467 INFO:tasks.workunit.client.0.smithi057.stderr:100400K .......... .......... .......... .......... .......... 73% 21.3M 1s 2024-04-21T08:46:24.468 INFO:tasks.workunit.client.0.smithi057.stderr:100450K .......... .......... .......... .......... .......... 74% 45.9M 1s 2024-04-21T08:46:24.471 INFO:tasks.workunit.client.0.smithi057.stderr:100500K .......... .......... .......... .......... .......... 74% 23.6M 1s 2024-04-21T08:46:24.473 INFO:tasks.workunit.client.0.smithi057.stderr:100550K .......... .......... .......... .......... .......... 74% 23.3M 1s 2024-04-21T08:46:24.474 INFO:tasks.workunit.client.0.smithi057.stderr:100600K .......... .......... .......... .......... .......... 74% 31.3M 1s 2024-04-21T08:46:24.476 INFO:tasks.workunit.client.0.smithi057.stderr:100650K .......... .......... .......... .......... .......... 74% 31.4M 1s 2024-04-21T08:46:24.477 INFO:tasks.workunit.client.0.smithi057.stderr:100700K .......... .......... .......... .......... .......... 74% 43.8M 1s 2024-04-21T08:46:24.479 INFO:tasks.workunit.client.0.smithi057.stderr:100750K .......... .......... .......... .......... .......... 74% 24.0M 1s 2024-04-21T08:46:24.481 INFO:tasks.workunit.client.0.smithi057.stderr:100800K .......... .......... .......... .......... .......... 74% 21.4M 1s 2024-04-21T08:46:24.482 INFO:tasks.workunit.client.0.smithi057.stderr:100850K .......... .......... .......... .......... .......... 74% 40.0M 1s 2024-04-21T08:46:24.490 INFO:tasks.workunit.client.0.smithi057.stderr:100900K .......... .......... .......... .......... .......... 74% 36.7M 1s 2024-04-21T08:46:24.490 INFO:tasks.workunit.client.0.smithi057.stderr:100950K .......... .......... .......... .......... .......... 74% 29.2M 1s 2024-04-21T08:46:24.490 INFO:tasks.workunit.client.0.smithi057.stderr:101000K .......... .......... .......... .......... .......... 74% 23.7M 1s 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 -- 172.21.15.57:0/3068610886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9701006c0 msgr2=0x7ff970100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 --2- 172.21.15.57:0/3068610886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9701006c0 0x7ff970100aa0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff960009ca0 tx=0x7ff96002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 -- 172.21.15.57:0/3068610886 shutdown_connections 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 --2- 172.21.15.57:0/3068610886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff970107330 0x7ff97010eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 --2- 172.21.15.57:0/3068610886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff970101070 0x7ff970106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 --2- 172.21.15.57:0/3068610886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9701006c0 0x7ff970100aa0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.489+0000 7ff976198640 1 -- 172.21.15.57:0/3068610886 >> 172.21.15.57:0/3068610886 conn(0x7ff970077390 msgr2=0x7ff9700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 -- 172.21.15.57:0/3068610886 shutdown_connections 2024-04-21T08:46:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 -- 172.21.15.57:0/3068610886 wait complete. 2024-04-21T08:46:24.492 INFO:tasks.workunit.client.0.smithi057.stderr:101050K .......... .......... .......... .......... .......... 74% 29.7M 1s 2024-04-21T08:46:24.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 Processor -- start 2024-04-21T08:46:24.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 -- start start 2024-04-21T08:46:24.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9701006c0 0x7ff970198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff970107330 0x7ff970198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 0x7ff97019d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.490+0000 7ff976198640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff970112220 con 0x7ff97019cfc0 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff976198640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701120a0 con 0x7ff970107330 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff976198640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701123a0 con 0x7ff9701006c0 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 0x7ff97019d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 0x7ff97019d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34732/0 (socket says 172.21.15.57:34732) 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 -- 172.21.15.57:0/1494660929 learned_addr learned my addr 172.21.15.57:0/1494660929 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96effd640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff970107330 0x7ff970198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9701006c0 0x7ff970198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 -- 172.21.15.57:0/1494660929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9701006c0 msgr2=0x7ff970198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9701006c0 0x7ff970198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 -- 172.21.15.57:0/1494660929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff970107330 msgr2=0x7ff970198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff970107330 0x7ff970198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 -- 172.21.15.57:0/1494660929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97019dab0 con 0x7ff97019cfc0 2024-04-21T08:46:24.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9701006c0 0x7ff970198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:24.496 INFO:tasks.workunit.client.0.smithi057.stderr:101100K .......... .......... .......... .......... .......... 74% 32.7M 1s 2024-04-21T08:46:24.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96ffff640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 0x7ff97019d470 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff96800d750 tx=0x7ff96800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:24.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96cff9640 1 -- 172.21.15.57:0/1494660929 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff968017070 con 0x7ff97019cfc0 2024-04-21T08:46:24.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96cff9640 1 -- 172.21.15.57:0/1494660929 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9680054d0 con 0x7ff97019cfc0 2024-04-21T08:46:24.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97019dda0 con 0x7ff97019cfc0 2024-04-21T08:46:24.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff96cff9640 1 -- 172.21.15.57:0/1494660929 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff968004c60 con 0x7ff97019cfc0 2024-04-21T08:46:24.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.491+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9701a5540 con 0x7ff97019cfc0 2024-04-21T08:46:24.499 INFO:tasks.workunit.client.0.smithi057.stderr:101150K .......... .......... .......... .......... .......... 74% 23.0M 1s 2024-04-21T08:46:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96cff9640 1 -- 172.21.15.57:0/1494660929 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7ff968009bd0 con 0x7ff97019cfc0 2024-04-21T08:46:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 Processor -- start 2024-04-21T08:46:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 -- start start 2024-04-21T08:46:24.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650c5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c5b60 0x5564650c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55646502a8d0 con 0x5564650cf0e0 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556464fc7e80 con 0x5564650c68d0 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff978d81740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556465026890 con 0x5564650c5b60 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96cff9640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff9580a3ee0 0x7ff9580a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34736/0 (socket says 172.21.15.57:34736) 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96effd640 1 -- 172.21.15.57:0/2023953783 learned_addr learned my addr 172.21.15.57:0/2023953783 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c5b60 0x5564650c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96effd640 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.494+0000 7ff96ffff640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff9580a3ee0 0x7ff9580a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.502 INFO:tasks.workunit.client.0.smithi057.stderr:101200K .......... .......... .......... .......... .......... 74% 43.1M 1s 2024-04-21T08:46:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96effd640 1 -- 172.21.15.57:0/2023953783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c5b60 msgr2=0x5564650c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.503 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96effd640 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c5b60 0x5564650c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96effd640 1 -- 172.21.15.57:0/2023953783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 msgr2=0x5564650c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96effd640 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96effd640 1 -- 172.21.15.57:0/2023953783 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556464fe9320 con 0x5564650cf0e0 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c5b60 0x5564650c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96cff9640 1 -- 172.21.15.57:0/1494660929 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff9680d0820 con 0x7ff97019cfc0 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96effd640 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d14d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff9640138f0 tx=0x7ff96400e410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe40145fc9304bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:24.504 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff96ffff640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff9580a3ee0 0x7ff9580a63a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9680cdaf0 tx=0x7ff968097e10 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff955ffb640 1 -- 172.21.15.57:0/2023953783 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96400b8d0 con 0x5564650cf0e0 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff955ffb640 1 -- 172.21.15.57:0/2023953783 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff964005020 con 0x5564650cf0e0 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.495+0000 7ff955ffb640 1 -- 172.21.15.57:0/2023953783 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9640049e0 con 0x5564650cf0e0 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 -- 172.21.15.57:0/2023953783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 msgr2=0x5564650d14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d14d0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff9640138f0 tx=0x7ff96400e410 comp rx=0 tx=0).stop 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 -- 172.21.15.57:0/2023953783 shutdown_connections 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d14d0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c5b60 0x5564650c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 --2- 172.21.15.57:0/2023953783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 -- 172.21.15.57:0/2023953783 >> 172.21.15.57:0/2023953783 conn(0x55646508cc90 msgr2=0x5564650c0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 -- 172.21.15.57:0/2023953783 shutdown_connections 2024-04-21T08:46:24.506 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 -- 172.21.15.57:0/2023953783 wait complete. 2024-04-21T08:46:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 -- start start 2024-04-21T08:46:24.507 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 Processor -- start 2024-04-21T08:46:24.508 INFO:tasks.workunit.client.0.smithi057.stderr:101250K .......... .......... .......... .......... .......... 74% 25.4M 1s 2024-04-21T08:46:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.496+0000 7ff978d81740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 0x5564650cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.509 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5564650d1e10 con 0x5564650c68d0 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556464fb2eb0 con 0x5564650c5b60 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5564650d1f90 con 0x5564650cf0e0 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 10 client.0 _ll_register_callbacks cb 0x556464fdfb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 0x5564650cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 0x5564650cbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34746/0 (socket says 172.21.15.57:34746) 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 -- 172.21.15.57:0/2243595141 learned_addr learned my addr 172.21.15.57:0/2243595141 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96effd640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96effd640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 -- 172.21.15.57:0/2243595141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 msgr2=0x5564650cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 -- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 msgr2=0x5564650ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 -- 172.21.15.57:0/2243595141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556464d3d210 con 0x5564650c68d0 2024-04-21T08:46:24.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96effd640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:24.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 0x5564650cbe90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff960009ca0 tx=0x7ff9600326f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2552c639a55d62b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:24.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff94effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:24.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff94effd640 1 -- 172.21.15.57:0/2243595141 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff960045020 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff964005020 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.497+0000 7ff94effd640 1 -- 172.21.15.57:0/2243595141 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff960003b70 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff94effd640 1 -- 172.21.15.57:0/2243595141 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff960050050 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 10 client.15918 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556464e09620 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff94effd640 1 -- 172.21.15.57:0/2243595141 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff964005020 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff94effd640 1 -- 172.21.15.57:0/2243595141 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff960040480 con 0x5564650c68d0 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 10 client.15918 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 20 client.15918 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5564650df1c0 0x5564650e15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 4 client.15918 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55646503e880 con 0x5564650df1c0 2024-04-21T08:46:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.498+0000 7ff96effd640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5564650df1c0 0x5564650e15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.515 INFO:tasks.workunit.client.0.smithi057.stderr:101300K .......... .......... .......... .......... .......... 74% 23.2M 1s 2024-04-21T08:46:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.499+0000 7ff96effd640 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5564650df1c0 0x5564650e15f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8baf73fa2912cf17 server_cookie=f68d561817ecf9f9 in_seq=0 out_seq=0 2024-04-21T08:46:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.499+0000 7ff94effd640 10 client.15918 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.499+0000 7ff94effd640 1 -- 172.21.15.57:0/2243595141 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55646503e880 con 0x5564650df1c0 2024-04-21T08:46:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.499+0000 7ff94effd640 10 client.15918 handle_command_reply: tid=0 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.499+0000 7ff978d81740 1 client.15918 shutdown 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 msgr2=0x5564650cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 0x5564650cbe90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff960009ca0 tx=0x7ff9600326f0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 shutdown_connections 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5564650df1c0 0x5564650e15f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c68d0 0x5564650cbe90 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 --2- 172.21.15.57:0/2243595141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff94effd640 0 client.15918 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 >> 172.21.15.57:0/2243595141 conn(0x55646508cc90 msgr2=0x5564650c25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 shutdown_connections 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 1 -- 172.21.15.57:0/2243595141 wait complete. 2024-04-21T08:46:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.500+0000 7ff978d81740 20 client.15918 trim_cache size 0 max 16384 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101350K .......... .......... .......... .......... .......... 74% 31.5M 1s 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101400K .......... .......... .......... .......... .......... 74% 45.0M 1s 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101450K .......... .......... .......... .......... .......... 74% 30.2M 1s 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101500K .......... .......... .......... .......... .......... 74% 23.6M 1s 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101550K .......... .......... .......... .......... .......... 74% 32.4M 1s 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101600K .......... .......... .......... .......... .......... 74% 27.1M 1s 2024-04-21T08:46:24.525 INFO:tasks.workunit.client.0.smithi057.stderr:101650K .......... .......... .......... .......... .......... 74% 20.4M 1s 2024-04-21T08:46:24.526 INFO:tasks.workunit.client.0.smithi057.stderr:101700K .......... .......... .......... .......... .......... 74% 41.7M 1s 2024-04-21T08:46:24.526 INFO:tasks.workunit.client.0.smithi057.stderr:101750K .......... .......... .......... .......... .......... 74% 23.6M 1s 2024-04-21T08:46:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 Processor -- start 2024-04-21T08:46:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 -- start start 2024-04-21T08:46:24.526 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 0x5564650dad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.527 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.527 INFO:tasks.workunit.client.0.smithi057.stderr:101800K .......... .......... .......... .......... .......... 75% 27.8M 1s 2024-04-21T08:46:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701ad190 con 0x5564650c5b60 2024-04-21T08:46:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701af9f0 con 0x5564650c68d0 2024-04-21T08:46:24.528 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.515+0000 7ff978d81740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701ad7e0 con 0x5564650cf0e0 2024-04-21T08:46:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 0x5564650dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 0x5564650dad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34754/0 (socket says 172.21.15.57:34754) 2024-04-21T08:46:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 -- 172.21.15.57:0/1748618424 learned_addr learned my addr 172.21.15.57:0/1748618424 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 -- 172.21.15.57:0/1748618424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 msgr2=0x5564650ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 -- 172.21.15.57:0/1748618424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 msgr2=0x5564650ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 -- 172.21.15.57:0/1748618424 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564650c4e30 con 0x5564650c5b60 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff96effd640 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 0x5564650dad10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff96400b690 tx=0x7ff96400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5f595f6a4f2497 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff9557fa640 1 -- 172.21.15.57:0/1748618424 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff964004220 con 0x5564650c5b60 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff9557fa640 1 -- 172.21.15.57:0/1748618424 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9640298e0 con 0x5564650c5b60 2024-04-21T08:46:24.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.516+0000 7ff9557fa640 1 -- 172.21.15.57:0/1748618424 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96400be70 con 0x5564650c5b60 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 -- 172.21.15.57:0/1748618424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 msgr2=0x5564650dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 0x5564650dad10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff96400b690 tx=0x7ff96400eb50 comp rx=0 tx=0).stop 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 -- 172.21.15.57:0/1748618424 shutdown_connections 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650cf0e0 0x5564650ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c68d0 0x5564650ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 --2- 172.21.15.57:0/1748618424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650c5b60 0x5564650dad10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 -- 172.21.15.57:0/1748618424 >> 172.21.15.57:0/1748618424 conn(0x55646508cc90 msgr2=0x5564650c0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 -- 172.21.15.57:0/1748618424 shutdown_connections 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 -- 172.21.15.57:0/1748618424 wait complete. 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 -- start start 2024-04-21T08:46:24.533 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.517+0000 7ff978d81740 1 Processor -- start 2024-04-21T08:46:24.534 INFO:tasks.workunit.client.0.smithi057.stderr:101850K .......... .......... .......... .......... .......... 75% 25.1M 1s 2024-04-21T08:46:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650d7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c68d0 0x5564650cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701adeb0 con 0x5564650cf0e0 2024-04-21T08:46:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701af320 con 0x5564650c5b60 2024-04-21T08:46:24.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff9701ae580 con 0x5564650c68d0 2024-04-21T08:46:24.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 10 client.0 _ll_register_callbacks cb 0x7ff960009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:24.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34758/0 (socket says 172.21.15.57:34758) 2024-04-21T08:46:24.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 -- 172.21.15.57:0/801986293 learned_addr learned my addr 172.21.15.57:0/801986293 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c68d0 0x5564650cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650d7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 -- 172.21.15.57:0/801986293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c68d0 msgr2=0x5564650cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c68d0 0x5564650cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 -- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 msgr2=0x5564650d7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650d7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 -- 172.21.15.57:0/801986293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564650d0930 con 0x5564650cf0e0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff96effd640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d0100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff9640130e0 tx=0x7ff96400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c8f8c745675baf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff94effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff94effd640 1 -- 172.21.15.57:0/801986293 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff964029630 con 0x5564650cf0e0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff94effd640 1 -- 172.21.15.57:0/801986293 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff96400baf0 con 0x5564650cf0e0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.518+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9640298e0 con 0x5564650cf0e0 2024-04-21T08:46:24.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff94effd640 1 -- 172.21.15.57:0/801986293 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff96403e050 con 0x5564650cf0e0 2024-04-21T08:46:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 10 client.15924 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:24.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556465149c70 con 0x5564650cf0e0 2024-04-21T08:46:24.539 INFO:tasks.workunit.client.0.smithi057.stderr:101900K .......... .......... .......... .......... .......... 75% 39.6M 1s 2024-04-21T08:46:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff94effd640 1 -- 172.21.15.57:0/801986293 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9640298e0 con 0x5564650cf0e0 2024-04-21T08:46:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff94effd640 1 -- 172.21.15.57:0/801986293 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff9640046d0 con 0x5564650cf0e0 2024-04-21T08:46:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 10 client.15924 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:24.540 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 20 client.15924 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55646514a0f0 0x55646514c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 4 client.15924 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556464fca3d0 con 0x55646514a0f0 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.519+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55646514a0f0 0x55646514c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.520+0000 7ff96f7fe640 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55646514a0f0 0x55646514c4e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e7900733a3c07 server_cookie=4d0c1c9778231fec in_seq=0 out_seq=0 2024-04-21T08:46:24.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.520+0000 7ff94effd640 10 client.15924 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:24.546 INFO:tasks.workunit.client.0.smithi057.stderr:101950K .......... .......... .......... .......... .......... 75% 38.2M 1s 2024-04-21T08:46:24.546 INFO:tasks.workunit.client.0.smithi057.stderr:102000K .......... .......... .......... .......... .......... 75% 22.2M 1s 2024-04-21T08:46:24.546 INFO:tasks.workunit.client.0.smithi057.stderr:102050K .......... .......... .......... .......... .......... 75% 23.0M 1s 2024-04-21T08:46:24.546 INFO:tasks.workunit.client.0.smithi057.stderr:102100K .......... .......... .......... .......... .......... 75% 38.1M 1s 2024-04-21T08:46:24.546 INFO:tasks.workunit.client.0.smithi057.stderr:102150K .......... .......... .......... .......... .......... 75% 23.2M 1s 2024-04-21T08:46:24.546 INFO:tasks.workunit.client.0.smithi057.stderr:102200K .......... .......... .......... .......... .......... 75% 22.4M 1s 2024-04-21T08:46:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff94effd640 1 -- 172.21.15.57:0/801986293 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556464fca3d0 con 0x55646514a0f0 2024-04-21T08:46:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff94effd640 10 client.15924 handle_command_reply: tid=0 2024-04-21T08:46:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 client.15924 shutdown 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 msgr2=0x5564650d0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d0100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff9640130e0 tx=0x7ff96400ebc0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 shutdown_connections 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55646514a0f0 0x55646514c4e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5564650cf0e0 0x5564650d0100 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5564650c68d0 0x5564650cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 --2- 172.21.15.57:0/801986293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5564650c5b60 0x5564650d7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff94effd640 0 client.15924 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 >> 172.21.15.57:0/801986293 conn(0x55646508cc90 msgr2=0x5564650c6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 shutdown_connections 2024-04-21T08:46:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 1 -- 172.21.15.57:0/801986293 wait complete. 2024-04-21T08:46:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.530+0000 7ff978d81740 20 client.15924 trim_cache size 0 max 16384 2024-04-21T08:46:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:46:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:46:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:46:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:46:24.550 INFO:tasks.workunit.client.0.smithi057.stderr:102250K .......... .......... .......... .......... .......... 75% 48.6M 1s 2024-04-21T08:46:24.550 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff9580a3ee0 msgr2=0x7ff9580a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff9580a3ee0 0x7ff9580a63a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9680cdaf0 tx=0x7ff968097e10 comp rx=0 tx=0).stop 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 msgr2=0x7ff97019d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 0x7ff97019d470 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff96800d750 tx=0x7ff96800dc10 comp rx=0 tx=0).stop 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 shutdown_connections 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff9580a3ee0 0x7ff9580a63a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff97019cfc0 0x7ff97019d470 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff970107330 0x7ff970198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 --2- 172.21.15.57:0/1494660929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9701006c0 0x7ff970198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 >> 172.21.15.57:0/1494660929 conn(0x7ff970077390 msgr2=0x7ff9700fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 shutdown_connections 2024-04-21T08:46:24.551 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:24.531+0000 7ff976198640 1 -- 172.21.15.57:0/1494660929 wait complete. 2024-04-21T08:46:24.568 INFO:tasks.workunit.client.0.smithi057.stderr:102300K .......... .......... .......... .......... .......... 75% 23.3M 1s 2024-04-21T08:46:24.568 INFO:tasks.workunit.client.0.smithi057.stderr:102350K .......... .......... .......... .......... .......... 75% 31.5M 1s 2024-04-21T08:46:24.568 INFO:tasks.workunit.client.0.smithi057.stderr:102400K .......... .......... .......... .......... .......... 75% 29.5M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102450K .......... .......... .......... .......... .......... 75% 35.4M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102500K .......... .......... .......... .......... .......... 75% 29.3M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102550K .......... .......... .......... .......... .......... 75% 23.2M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102600K .......... .......... .......... .......... .......... 75% 35.5M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102650K .......... .......... .......... .......... .......... 75% 24.4M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102700K .......... .......... .......... .......... .......... 75% 26.6M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102750K .......... .......... .......... .......... .......... 75% 41.5M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102800K .......... .......... .......... .......... .......... 75% 23.8M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102850K .......... .......... .......... .......... .......... 75% 40.2M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102900K .......... .......... .......... .......... .......... 75% 31.9M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:102950K .......... .......... .......... .......... .......... 75% 25.3M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:103000K .......... .......... .......... .......... .......... 75% 31.1M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:103050K .......... .......... .......... .......... .......... 75% 22.4M 1s 2024-04-21T08:46:24.569 INFO:tasks.workunit.client.0.smithi057.stderr:103100K .......... .......... .......... .......... .......... 75% 35.8M 1s 2024-04-21T08:46:24.570 INFO:tasks.workunit.client.0.smithi057.stderr:103150K .......... .......... .......... .......... .......... 76% 23.6M 1s 2024-04-21T08:46:24.570 INFO:tasks.workunit.client.0.smithi057.stderr:103200K .......... .......... .......... .......... .......... 76% 25.4M 1s 2024-04-21T08:46:24.570 INFO:tasks.workunit.client.0.smithi057.stderr:103250K .......... .......... .......... .......... .......... 76% 41.6M 1s 2024-04-21T08:46:24.570 INFO:tasks.workunit.client.0.smithi057.stderr:103300K .......... .......... .......... .......... .......... 76% 27.5M 1s 2024-04-21T08:46:24.570 INFO:tasks.workunit.client.0.smithi057.stderr:103350K .......... .......... .......... .......... .......... 76% 22.3M 1s 2024-04-21T08:46:24.571 INFO:tasks.workunit.client.0.smithi057.stderr:103400K .......... .......... .......... .......... .......... 76% 38.5M 1s 2024-04-21T08:46:24.571 INFO:tasks.workunit.client.0.smithi057.stderr:103450K .......... .......... .......... .......... .......... 76% 27.2M 1s 2024-04-21T08:46:24.573 INFO:tasks.workunit.client.0.smithi057.stderr:103500K .......... .......... .......... .......... .......... 76% 37.8M 1s 2024-04-21T08:46:24.575 INFO:tasks.workunit.client.0.smithi057.stderr:103550K .......... .......... .......... .......... .......... 76% 23.3M 1s 2024-04-21T08:46:24.577 INFO:tasks.workunit.client.0.smithi057.stderr:103600K .......... .......... .......... .......... .......... 76% 28.3M 1s 2024-04-21T08:46:24.578 INFO:tasks.workunit.client.0.smithi057.stderr:103650K .......... .......... .......... .......... .......... 76% 28.0M 1s 2024-04-21T08:46:24.579 INFO:tasks.workunit.client.0.smithi057.stderr:103700K .......... .......... .......... .......... .......... 76% 33.4M 1s 2024-04-21T08:46:24.581 INFO:tasks.workunit.client.0.smithi057.stderr:103750K .......... .......... .......... .......... .......... 76% 33.9M 1s 2024-04-21T08:46:24.583 INFO:tasks.workunit.client.0.smithi057.stderr:103800K .......... .......... .......... .......... .......... 76% 24.8M 1s 2024-04-21T08:46:24.585 INFO:tasks.workunit.client.0.smithi057.stderr:103850K .......... .......... .......... .......... .......... 76% 22.4M 1s 2024-04-21T08:46:24.586 INFO:tasks.workunit.client.0.smithi057.stderr:103900K .......... .......... .......... .......... .......... 76% 39.0M 1s 2024-04-21T08:46:24.589 INFO:tasks.workunit.client.0.smithi057.stderr:103950K .......... .......... .......... .......... .......... 76% 23.8M 1s 2024-04-21T08:46:24.589 INFO:tasks.workunit.client.0.smithi057.stderr:104000K .......... .......... .......... .......... .......... 76% 50.2M 1s 2024-04-21T08:46:24.591 INFO:tasks.workunit.client.0.smithi057.stderr:104050K .......... .......... .......... .......... .......... 76% 29.4M 1s 2024-04-21T08:46:24.593 INFO:tasks.workunit.client.0.smithi057.stderr:104100K .......... .......... .......... .......... .......... 76% 21.0M 1s 2024-04-21T08:46:24.594 INFO:tasks.workunit.client.0.smithi057.stderr:104150K .......... .......... .......... .......... .......... 76% 51.0M 1s 2024-04-21T08:46:24.596 INFO:tasks.workunit.client.0.smithi057.stderr:104200K .......... .......... .......... .......... .......... 76% 25.0M 1s 2024-04-21T08:46:24.598 INFO:tasks.workunit.client.0.smithi057.stderr:104250K .......... .......... .......... .......... .......... 76% 35.2M 1s 2024-04-21T08:46:24.600 INFO:tasks.workunit.client.0.smithi057.stderr:104300K .......... .......... .......... .......... .......... 76% 22.8M 1s 2024-04-21T08:46:24.601 INFO:tasks.workunit.client.0.smithi057.stderr:104350K .......... .......... .......... .......... .......... 76% 34.1M 1s 2024-04-21T08:46:24.604 INFO:tasks.workunit.client.0.smithi057.stderr:104400K .......... .......... .......... .......... .......... 76% 24.5M 1s 2024-04-21T08:46:24.605 INFO:tasks.workunit.client.0.smithi057.stderr:104450K .......... .......... .......... .......... .......... 76% 24.5M 1s 2024-04-21T08:46:24.607 INFO:tasks.workunit.client.0.smithi057.stderr:104500K .......... .......... .......... .......... .......... 77% 48.4M 1s 2024-04-21T08:46:24.608 INFO:tasks.workunit.client.0.smithi057.stderr:104550K .......... .......... .......... .......... .......... 77% 26.2M 1s 2024-04-21T08:46:24.609 INFO:tasks.workunit.client.0.smithi057.stderr:104600K .......... .......... .......... .......... .......... 77% 36.4M 1s 2024-04-21T08:46:24.611 INFO:tasks.workunit.client.0.smithi057.stderr:104650K .......... .......... .......... .......... .......... 77% 29.7M 1s 2024-04-21T08:46:24.614 INFO:tasks.workunit.client.0.smithi057.stderr:104700K .......... .......... .......... .......... .......... 77% 18.8M 1s 2024-04-21T08:46:24.614 INFO:tasks.workunit.client.0.smithi057.stderr:104750K .......... .......... .......... .......... .......... 77% 45.0M 1s 2024-04-21T08:46:24.617 INFO:tasks.workunit.client.0.smithi057.stderr:104800K .......... .......... .......... .......... .......... 77% 24.3M 1s 2024-04-21T08:46:24.618 INFO:tasks.workunit.client.0.smithi057.stderr:104850K .......... .......... .......... .......... .......... 77% 37.6M 1s 2024-04-21T08:46:24.620 INFO:tasks.workunit.client.0.smithi057.stderr:104900K .......... .......... .......... .......... .......... 77% 23.9M 1s 2024-04-21T08:46:24.622 INFO:tasks.workunit.client.0.smithi057.stderr:104950K .......... .......... .......... .......... .......... 77% 24.7M 1s 2024-04-21T08:46:24.623 INFO:tasks.workunit.client.0.smithi057.stderr:105000K .......... .......... .......... .......... .......... 77% 50.4M 1s 2024-04-21T08:46:24.625 INFO:tasks.workunit.client.0.smithi057.stderr:105050K .......... .......... .......... .......... .......... 77% 23.6M 1s 2024-04-21T08:46:24.626 INFO:tasks.workunit.client.0.smithi057.stderr:105100K .......... .......... .......... .......... .......... 77% 34.8M 1s 2024-04-21T08:46:24.629 INFO:tasks.workunit.client.0.smithi057.stderr:105150K .......... .......... .......... .......... .......... 77% 26.6M 1s 2024-04-21T08:46:24.630 INFO:tasks.workunit.client.0.smithi057.stderr:105200K .......... .......... .......... .......... .......... 77% 22.6M 1s 2024-04-21T08:46:24.631 INFO:tasks.workunit.client.0.smithi057.stderr:105250K .......... .......... .......... .......... .......... 77% 50.7M 1s 2024-04-21T08:46:24.634 INFO:tasks.workunit.client.0.smithi057.stderr:105300K .......... .......... .......... .......... .......... 77% 22.6M 1s 2024-04-21T08:46:24.635 INFO:tasks.workunit.client.0.smithi057.stderr:105350K .......... .......... .......... .......... .......... 77% 36.3M 1s 2024-04-21T08:46:24.637 INFO:tasks.workunit.client.0.smithi057.stderr:105400K .......... .......... .......... .......... .......... 77% 26.5M 1s 2024-04-21T08:46:24.638 INFO:tasks.workunit.client.0.smithi057.stderr:105450K .......... .......... .......... .......... .......... 77% 27.4M 1s 2024-04-21T08:46:24.640 INFO:tasks.workunit.client.0.smithi057.stderr:105500K .......... .......... .......... .......... .......... 77% 42.5M 1s 2024-04-21T08:46:24.642 INFO:tasks.workunit.client.0.smithi057.stderr:105550K .......... .......... .......... .......... .......... 77% 22.7M 1s 2024-04-21T08:46:24.644 INFO:tasks.workunit.client.0.smithi057.stderr:105600K .......... .......... .......... .......... .......... 77% 38.6M 1s 2024-04-21T08:46:24.645 INFO:tasks.workunit.client.0.smithi057.stderr:105650K .......... .......... .......... .......... .......... 77% 24.5M 1s 2024-04-21T08:46:24.647 INFO:tasks.workunit.client.0.smithi057.stderr:105700K .......... .......... .......... .......... .......... 77% 22.4M 1s 2024-04-21T08:46:24.648 INFO:tasks.workunit.client.0.smithi057.stderr:105750K .......... .......... .......... .......... .......... 77% 54.0M 1s 2024-04-21T08:46:24.650 INFO:tasks.workunit.client.0.smithi057.stderr:105800K .......... .......... .......... .......... .......... 77% 22.0M 1s 2024-04-21T08:46:24.652 INFO:tasks.workunit.client.0.smithi057.stderr:105850K .......... .......... .......... .......... .......... 78% 24.4M 1s 2024-04-21T08:46:24.654 INFO:tasks.workunit.client.0.smithi057.stderr:105900K .......... .......... .......... .......... .......... 78% 41.8M 1s 2024-04-21T08:46:24.656 INFO:tasks.workunit.client.0.smithi057.stderr:105950K .......... .......... .......... .......... .......... 78% 23.1M 1s 2024-04-21T08:46:24.657 INFO:tasks.workunit.client.0.smithi057.stderr:106000K .......... .......... .......... .......... .......... 78% 49.9M 1s 2024-04-21T08:46:24.659 INFO:tasks.workunit.client.0.smithi057.stderr:106050K .......... .......... .......... .......... .......... 78% 22.6M 1s 2024-04-21T08:46:24.661 INFO:tasks.workunit.client.0.smithi057.stderr:106100K .......... .......... .......... .......... .......... 78% 25.5M 1s 2024-04-21T08:46:24.662 INFO:tasks.workunit.client.0.smithi057.stderr:106150K .......... .......... .......... .......... .......... 78% 39.1M 1s 2024-04-21T08:46:24.664 INFO:tasks.workunit.client.0.smithi057.stderr:106200K .......... .......... .......... .......... .......... 78% 24.4M 1s 2024-04-21T08:46:24.665 INFO:tasks.workunit.client.0.smithi057.stderr:106250K .......... .......... .......... .......... .......... 78% 49.1M 1s 2024-04-21T08:46:24.667 INFO:tasks.workunit.client.0.smithi057.stderr:106300K .......... .......... .......... .......... .......... 78% 28.5M 1s 2024-04-21T08:46:24.669 INFO:tasks.workunit.client.0.smithi057.stderr:106350K .......... .......... .......... .......... .......... 78% 19.8M 1s 2024-04-21T08:46:24.670 INFO:tasks.workunit.client.0.smithi057.stderr:106400K .......... .......... .......... .......... .......... 78% 40.7M 1s 2024-04-21T08:46:24.673 INFO:tasks.workunit.client.0.smithi057.stderr:106450K .......... .......... .......... .......... .......... 78% 22.5M 1s 2024-04-21T08:46:24.673 INFO:tasks.workunit.client.0.smithi057.stderr:106500K .......... .......... .......... .......... .......... 78% 53.7M 1s 2024-04-21T08:46:24.676 INFO:tasks.workunit.client.0.smithi057.stderr:106550K .......... .......... .......... .......... .......... 78% 22.8M 1s 2024-04-21T08:46:24.678 INFO:tasks.workunit.client.0.smithi057.stderr:106600K .......... .......... .......... .......... .......... 78% 21.0M 1s 2024-04-21T08:46:24.679 INFO:tasks.workunit.client.0.smithi057.stderr:106650K .......... .......... .......... .......... .......... 78% 37.5M 1s 2024-04-21T08:46:24.681 INFO:tasks.workunit.client.0.smithi057.stderr:106700K .......... .......... .......... .......... .......... 78% 24.9M 1s 2024-04-21T08:46:24.682 INFO:tasks.workunit.client.0.smithi057.stderr:106750K .......... .......... .......... .......... .......... 78% 45.7M 1s 2024-04-21T08:46:24.684 INFO:tasks.workunit.client.0.smithi057.stderr:106800K .......... .......... .......... .......... .......... 78% 23.8M 1s 2024-04-21T08:46:24.686 INFO:tasks.workunit.client.0.smithi057.stderr:106850K .......... .......... .......... .......... .......... 78% 24.0M 1s 2024-04-21T08:46:24.693 INFO:tasks.workunit.client.0.smithi057.stderr:106900K .......... .......... .......... .......... .......... 78% 38.1M 1s 2024-04-21T08:46:24.693 INFO:tasks.workunit.client.0.smithi057.stderr:106950K .......... .......... .......... .......... .......... 78% 24.4M 1s 2024-04-21T08:46:24.693 INFO:tasks.workunit.client.0.smithi057.stderr:107000K .......... .......... .......... .......... .......... 78% 44.4M 1s 2024-04-21T08:46:24.693 INFO:tasks.workunit.client.0.smithi057.stderr:107050K .......... .......... .......... .......... .......... 78% 23.7M 1s 2024-04-21T08:46:24.695 INFO:tasks.workunit.client.0.smithi057.stderr:107100K .......... .......... .......... .......... .......... 78% 22.0M 1s 2024-04-21T08:46:24.696 INFO:tasks.workunit.client.0.smithi057.stderr:107150K .......... .......... .......... .......... .......... 78% 44.1M 1s 2024-04-21T08:46:24.698 INFO:tasks.workunit.client.0.smithi057.stderr:107200K .......... .......... .......... .......... .......... 79% 22.1M 1s 2024-04-21T08:46:24.701 INFO:tasks.workunit.client.0.smithi057.stderr:107250K .......... .......... .......... .......... .......... 79% 20.7M 1s 2024-04-21T08:46:24.702 INFO:tasks.workunit.client.0.smithi057.stderr:107300K .......... .......... .......... .......... .......... 79% 47.4M 1s 2024-04-21T08:46:24.704 INFO:tasks.workunit.client.0.smithi057.stderr:107350K .......... .......... .......... .......... .......... 79% 39.8M 1s 2024-04-21T08:46:24.705 INFO:tasks.workunit.client.0.smithi057.stderr:107400K .......... .......... .......... .......... .......... 79% 24.8M 1s 2024-04-21T08:46:24.707 INFO:tasks.workunit.client.0.smithi057.stderr:107450K .......... .......... .......... .......... .......... 79% 30.5M 1s 2024-04-21T08:46:24.709 INFO:tasks.workunit.client.0.smithi057.stderr:107500K .......... .......... .......... .......... .......... 79% 19.8M 1s 2024-04-21T08:46:24.710 INFO:tasks.workunit.client.0.smithi057.stderr:107550K .......... .......... .......... .......... .......... 79% 44.8M 1s 2024-04-21T08:46:24.712 INFO:tasks.workunit.client.0.smithi057.stderr:107600K .......... .......... .......... .......... .......... 79% 25.2M 1s 2024-04-21T08:46:24.714 INFO:tasks.workunit.client.0.smithi057.stderr:107650K .......... .......... .......... .......... .......... 79% 44.5M 1s 2024-04-21T08:46:24.715 INFO:tasks.workunit.client.0.smithi057.stderr:107700K .......... .......... .......... .......... .......... 79% 23.1M 1s 2024-04-21T08:46:24.718 INFO:tasks.workunit.client.0.smithi057.stderr:107750K .......... .......... .......... .......... .......... 79% 22.2M 1s 2024-04-21T08:46:24.718 INFO:tasks.workunit.client.0.smithi057.stderr:107800K .......... .......... .......... .......... .......... 79% 55.0M 1s 2024-04-21T08:46:24.720 INFO:tasks.workunit.client.0.smithi057.stderr:107850K .......... .......... .......... .......... .......... 79% 22.5M 1s 2024-04-21T08:46:24.722 INFO:tasks.workunit.client.0.smithi057.stderr:107900K .......... .......... .......... .......... .......... 79% 38.1M 1s 2024-04-21T08:46:24.724 INFO:tasks.workunit.client.0.smithi057.stderr:107950K .......... .......... .......... .......... .......... 79% 25.5M 1s 2024-04-21T08:46:24.726 INFO:tasks.workunit.client.0.smithi057.stderr:108000K .......... .......... .......... .......... .......... 79% 23.4M 1s 2024-04-21T08:46:24.727 INFO:tasks.workunit.client.0.smithi057.stderr:108050K .......... .......... .......... .......... .......... 79% 46.9M 1s 2024-04-21T08:46:24.729 INFO:tasks.workunit.client.0.smithi057.stderr:108100K .......... .......... .......... .......... .......... 79% 24.8M 1s 2024-04-21T08:46:24.730 INFO:tasks.workunit.client.0.smithi057.stderr:108150K .......... .......... .......... .......... .......... 79% 36.6M 1s 2024-04-21T08:46:24.732 INFO:tasks.workunit.client.0.smithi057.stderr:108200K .......... .......... .......... .......... .......... 79% 26.5M 1s 2024-04-21T08:46:24.734 INFO:tasks.workunit.client.0.smithi057.stderr:108250K .......... .......... .......... .......... .......... 79% 22.8M 1s 2024-04-21T08:46:24.735 INFO:tasks.workunit.client.0.smithi057.stderr:108300K .......... .......... .......... .......... .......... 79% 52.2M 1s 2024-04-21T08:46:24.737 INFO:tasks.workunit.client.0.smithi057.stderr:108350K .......... .......... .......... .......... .......... 79% 23.0M 1s 2024-04-21T08:46:24.739 INFO:tasks.workunit.client.0.smithi057.stderr:108400K .......... .......... .......... .......... .......... 79% 33.3M 1s 2024-04-21T08:46:24.741 INFO:tasks.workunit.client.0.smithi057.stderr:108450K .......... .......... .......... .......... .......... 79% 24.4M 1s 2024-04-21T08:46:24.743 INFO:tasks.workunit.client.0.smithi057.stderr:108500K .......... .......... .......... .......... .......... 79% 24.5M 1s 2024-04-21T08:46:24.743 INFO:tasks.workunit.client.0.smithi057.stderr:108550K .......... .......... .......... .......... .......... 79% 55.8M 1s 2024-04-21T08:46:24.745 INFO:tasks.workunit.client.0.smithi057.stderr:108600K .......... .......... .......... .......... .......... 80% 23.7M 1s 2024-04-21T08:46:24.747 INFO:tasks.workunit.client.0.smithi057.stderr:108650K .......... .......... .......... .......... .......... 80% 34.3M 1s 2024-04-21T08:46:24.749 INFO:tasks.workunit.client.0.smithi057.stderr:108700K .......... .......... .......... .......... .......... 80% 25.0M 1s 2024-04-21T08:46:24.751 INFO:tasks.workunit.client.0.smithi057.stderr:108750K .......... .......... .......... .......... .......... 80% 25.4M 1s 2024-04-21T08:46:24.752 INFO:tasks.workunit.client.0.smithi057.stderr:108800K .......... .......... .......... .......... .......... 80% 46.8M 1s 2024-04-21T08:46:24.754 INFO:tasks.workunit.client.0.smithi057.stderr:108850K .......... .......... .......... .......... .......... 80% 24.0M 1s 2024-04-21T08:46:24.755 INFO:tasks.workunit.client.0.smithi057.stderr:108900K .......... .......... .......... .......... .......... 80% 38.6M 1s 2024-04-21T08:46:24.757 INFO:tasks.workunit.client.0.smithi057.stderr:108950K .......... .......... .......... .......... .......... 80% 23.1M 1s 2024-04-21T08:46:24.759 INFO:tasks.workunit.client.0.smithi057.stderr:109000K .......... .......... .......... .......... .......... 80% 23.5M 1s 2024-04-21T08:46:24.760 INFO:tasks.workunit.client.0.smithi057.stderr:109050K .......... .......... .......... .......... .......... 80% 53.4M 1s 2024-04-21T08:46:24.762 INFO:tasks.workunit.client.0.smithi057.stderr:109100K .......... .......... .......... .......... .......... 80% 26.0M 1s 2024-04-21T08:46:24.764 INFO:tasks.workunit.client.0.smithi057.stderr:109150K .......... .......... .......... .......... .......... 80% 32.4M 1s 2024-04-21T08:46:24.766 INFO:tasks.workunit.client.0.smithi057.stderr:109200K .......... .......... .......... .......... .......... 80% 19.2M 1s 2024-04-21T08:46:24.768 INFO:tasks.workunit.client.0.smithi057.stderr:109250K .......... .......... .......... .......... .......... 80% 22.1M 1s 2024-04-21T08:46:24.769 INFO:tasks.workunit.client.0.smithi057.stderr:109300K .......... .......... .......... .......... .......... 80% 57.7M 1s 2024-04-21T08:46:24.771 INFO:tasks.workunit.client.0.smithi057.stderr:109350K .......... .......... .......... .......... .......... 80% 28.0M 1s 2024-04-21T08:46:24.773 INFO:tasks.workunit.client.0.smithi057.stderr:109400K .......... .......... .......... .......... .......... 80% 29.1M 1s 2024-04-21T08:46:24.774 INFO:tasks.workunit.client.0.smithi057.stderr:109450K .......... .......... .......... .......... .......... 80% 25.4M 1s 2024-04-21T08:46:24.777 INFO:tasks.workunit.client.0.smithi057.stderr:109500K .......... .......... .......... .......... .......... 80% 22.6M 1s 2024-04-21T08:46:24.778 INFO:tasks.workunit.client.0.smithi057.stderr:109550K .......... .......... .......... .......... .......... 80% 37.6M 1s 2024-04-21T08:46:24.780 INFO:tasks.workunit.client.0.smithi057.stderr:109600K .......... .......... .......... .......... .......... 80% 23.9M 1s 2024-04-21T08:46:24.781 INFO:tasks.workunit.client.0.smithi057.stderr:109650K .......... .......... .......... .......... .......... 80% 43.2M 1s 2024-04-21T08:46:24.784 INFO:tasks.workunit.client.0.smithi057.stderr:109700K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-04-21T08:46:24.785 INFO:tasks.workunit.client.0.smithi057.stderr:109750K .......... .......... .......... .......... .......... 80% 27.1M 1s 2024-04-21T08:46:24.786 INFO:tasks.workunit.client.0.smithi057.stderr:109800K .......... .......... .......... .......... .......... 80% 37.3M 1s 2024-04-21T08:46:24.788 INFO:tasks.workunit.client.0.smithi057.stderr:109850K .......... .......... .......... .......... .......... 80% 30.2M 1s 2024-04-21T08:46:24.791 INFO:tasks.workunit.client.0.smithi057.stderr:109900K .......... .......... .......... .......... .......... 80% 20.7M 1s 2024-04-21T08:46:24.792 INFO:tasks.workunit.client.0.smithi057.stderr:109950K .......... .......... .......... .......... .......... 81% 34.3M 1s 2024-04-21T08:46:24.794 INFO:tasks.workunit.client.0.smithi057.stderr:110000K .......... .......... .......... .......... .......... 81% 25.1M 1s 2024-04-21T08:46:24.794 INFO:tasks.workunit.client.0.smithi057.stderr:110050K .......... .......... .......... .......... .......... 81% 49.1M 1s 2024-04-21T08:46:24.797 INFO:tasks.workunit.client.0.smithi057.stderr:110100K .......... .......... .......... .......... .......... 81% 22.6M 1s 2024-04-21T08:46:24.798 INFO:tasks.workunit.client.0.smithi057.stderr:110150K .......... .......... .......... .......... .......... 81% 45.3M 1s 2024-04-21T08:46:24.800 INFO:tasks.workunit.client.0.smithi057.stderr:110200K .......... .......... .......... .......... .......... 81% 22.2M 1s 2024-04-21T08:46:24.802 INFO:tasks.workunit.client.0.smithi057.stderr:110250K .......... .......... .......... .......... .......... 81% 25.9M 1s 2024-04-21T08:46:24.803 INFO:tasks.workunit.client.0.smithi057.stderr:110300K .......... .......... .......... .......... .......... 81% 46.9M 1s 2024-04-21T08:46:24.805 INFO:tasks.workunit.client.0.smithi057.stderr:110350K .......... .......... .......... .......... .......... 81% 23.8M 1s 2024-04-21T08:46:24.806 INFO:tasks.workunit.client.0.smithi057.stderr:110400K .......... .......... .......... .......... .......... 81% 31.7M 1s 2024-04-21T08:46:24.809 INFO:tasks.workunit.client.0.smithi057.stderr:110450K .......... .......... .......... .......... .......... 81% 20.3M 1s 2024-04-21T08:46:24.811 INFO:tasks.workunit.client.0.smithi057.stderr:110500K .......... .......... .......... .......... .......... 81% 25.7M 1s 2024-04-21T08:46:24.812 INFO:tasks.workunit.client.0.smithi057.stderr:110550K .......... .......... .......... .......... .......... 81% 48.7M 1s 2024-04-21T08:46:24.814 INFO:tasks.workunit.client.0.smithi057.stderr:110600K .......... .......... .......... .......... .......... 81% 21.0M 1s 2024-04-21T08:46:24.816 INFO:tasks.workunit.client.0.smithi057.stderr:110650K .......... .......... .......... .......... .......... 81% 43.9M 1s 2024-04-21T08:46:24.817 INFO:tasks.workunit.client.0.smithi057.stderr:110700K .......... .......... .......... .......... .......... 81% 23.7M 1s 2024-04-21T08:46:24.820 INFO:tasks.workunit.client.0.smithi057.stderr:110750K .......... .......... .......... .......... .......... 81% 22.9M 1s 2024-04-21T08:46:24.821 INFO:tasks.workunit.client.0.smithi057.stderr:110800K .......... .......... .......... .......... .......... 81% 51.9M 1s 2024-04-21T08:46:24.822 INFO:tasks.workunit.client.0.smithi057.stderr:110850K .......... .......... .......... .......... .......... 81% 23.9M 1s 2024-04-21T08:46:24.824 INFO:tasks.workunit.client.0.smithi057.stderr:110900K .......... .......... .......... .......... .......... 81% 35.6M 1s 2024-04-21T08:46:24.826 INFO:tasks.workunit.client.0.smithi057.stderr:110950K .......... .......... .......... .......... .......... 81% 23.7M 1s 2024-04-21T08:46:24.828 INFO:tasks.workunit.client.0.smithi057.stderr:111000K .......... .......... .......... .......... .......... 81% 22.5M 1s 2024-04-21T08:46:24.829 INFO:tasks.workunit.client.0.smithi057.stderr:111050K .......... .......... .......... .......... .......... 81% 61.3M 1s 2024-04-21T08:46:24.831 INFO:tasks.workunit.client.0.smithi057.stderr:111100K .......... .......... .......... .......... .......... 81% 23.3M 1s 2024-04-21T08:46:24.833 INFO:tasks.workunit.client.0.smithi057.stderr:111150K .......... .......... .......... .......... .......... 81% 22.9M 1s 2024-04-21T08:46:24.834 INFO:tasks.workunit.client.0.smithi057.stderr:111200K .......... .......... .......... .......... .......... 81% 39.7M 1s 2024-04-21T08:46:24.836 INFO:tasks.workunit.client.0.smithi057.stderr:111250K .......... .......... .......... .......... .......... 81% 23.4M 1s 2024-04-21T08:46:24.837 INFO:tasks.workunit.client.0.smithi057.stderr:111300K .......... .......... .......... .......... .......... 82% 53.8M 1s 2024-04-21T08:46:24.840 INFO:tasks.workunit.client.0.smithi057.stderr:111350K .......... .......... .......... .......... .......... 82% 22.6M 1s 2024-04-21T08:46:24.842 INFO:tasks.workunit.client.0.smithi057.stderr:111400K .......... .......... .......... .......... .......... 82% 21.5M 1s 2024-04-21T08:46:24.843 INFO:tasks.workunit.client.0.smithi057.stderr:111450K .......... .......... .......... .......... .......... 82% 40.5M 1s 2024-04-21T08:46:24.845 INFO:tasks.workunit.client.0.smithi057.stderr:111500K .......... .......... .......... .......... .......... 82% 24.3M 1s 2024-04-21T08:46:24.846 INFO:tasks.workunit.client.0.smithi057.stderr:111550K .......... .......... .......... .......... .......... 82% 46.6M 1s 2024-04-21T08:46:24.848 INFO:tasks.workunit.client.0.smithi057.stderr:111600K .......... .......... .......... .......... .......... 82% 23.7M 1s 2024-04-21T08:46:24.850 INFO:tasks.workunit.client.0.smithi057.stderr:111650K .......... .......... .......... .......... .......... 82% 23.9M 1s 2024-04-21T08:46:24.851 INFO:tasks.workunit.client.0.smithi057.stderr:111700K .......... .......... .......... .......... .......... 82% 41.6M 1s 2024-04-21T08:46:24.853 INFO:tasks.workunit.client.0.smithi057.stderr:111750K .......... .......... .......... .......... .......... 82% 23.1M 1s 2024-04-21T08:46:24.854 INFO:tasks.workunit.client.0.smithi057.stderr:111800K .......... .......... .......... .......... .......... 82% 57.7M 1s 2024-04-21T08:46:24.856 INFO:tasks.workunit.client.0.smithi057.stderr:111850K .......... .......... .......... .......... .......... 82% 21.6M 1s 2024-04-21T08:46:24.859 INFO:tasks.workunit.client.0.smithi057.stderr:111900K .......... .......... .......... .......... .......... 82% 23.4M 1s 2024-04-21T08:46:24.860 INFO:tasks.workunit.client.0.smithi057.stderr:111950K .......... .......... .......... .......... .......... 82% 38.2M 1s 2024-04-21T08:46:24.862 INFO:tasks.workunit.client.0.smithi057.stderr:112000K .......... .......... .......... .......... .......... 82% 24.6M 1s 2024-04-21T08:46:24.863 INFO:tasks.workunit.client.0.smithi057.stderr:112050K .......... .......... .......... .......... .......... 82% 48.5M 1s 2024-04-21T08:46:24.865 INFO:tasks.workunit.client.0.smithi057.stderr:112100K .......... .......... .......... .......... .......... 82% 24.7M 1s 2024-04-21T08:46:24.867 INFO:tasks.workunit.client.0.smithi057.stderr:112150K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-04-21T08:46:24.868 INFO:tasks.workunit.client.0.smithi057.stderr:112200K .......... .......... .......... .......... .......... 82% 42.2M 1s 2024-04-21T08:46:24.870 INFO:tasks.workunit.client.0.smithi057.stderr:112250K .......... .......... .......... .......... .......... 82% 23.5M 1s 2024-04-21T08:46:24.872 INFO:tasks.workunit.client.0.smithi057.stderr:112300K .......... .......... .......... .......... .......... 82% 45.9M 1s 2024-04-21T08:46:24.873 INFO:tasks.workunit.client.0.smithi057.stderr:112350K .......... .......... .......... .......... .......... 82% 22.3M 1s 2024-04-21T08:46:24.875 INFO:tasks.workunit.client.0.smithi057.stderr:112400K .......... .......... .......... .......... .......... 82% 25.3M 1s 2024-04-21T08:46:24.877 INFO:tasks.workunit.client.0.smithi057.stderr:112450K .......... .......... .......... .......... .......... 82% 31.0M 1s 2024-04-21T08:46:24.879 INFO:tasks.workunit.client.0.smithi057.stderr:112500K .......... .......... .......... .......... .......... 82% 23.7M 1s 2024-04-21T08:46:24.881 INFO:tasks.workunit.client.0.smithi057.stderr:112550K .......... .......... .......... .......... .......... 82% 27.0M 1s 2024-04-21T08:46:24.882 INFO:tasks.workunit.client.0.smithi057.stderr:112600K .......... .......... .......... .......... .......... 82% 40.0M 1s 2024-04-21T08:46:24.884 INFO:tasks.workunit.client.0.smithi057.stderr:112650K .......... .......... .......... .......... .......... 83% 26.6M 1s 2024-04-21T08:46:24.885 INFO:tasks.workunit.client.0.smithi057.stderr:112700K .......... .......... .......... .......... .......... 83% 31.6M 1s 2024-04-21T08:46:24.887 INFO:tasks.workunit.client.0.smithi057.stderr:112750K .......... .......... .......... .......... .......... 83% 25.5M 1s 2024-04-21T08:46:24.889 INFO:tasks.workunit.client.0.smithi057.stderr:112800K .......... .......... .......... .......... .......... 83% 23.9M 1s 2024-04-21T08:46:24.897 INFO:tasks.workunit.client.0.smithi057.stderr:112850K .......... .......... .......... .......... .......... 83% 41.6M 1s 2024-04-21T08:46:24.897 INFO:tasks.workunit.client.0.smithi057.stderr:112900K .......... .......... .......... .......... .......... 83% 23.1M 1s 2024-04-21T08:46:24.897 INFO:tasks.workunit.client.0.smithi057.stderr:112950K .......... .......... .......... .......... .......... 83% 39.3M 1s 2024-04-21T08:46:24.897 INFO:tasks.workunit.client.0.smithi057.stderr:113000K .......... .......... .......... .......... .......... 83% 25.4M 1s 2024-04-21T08:46:24.898 INFO:tasks.workunit.client.0.smithi057.stderr:113050K .......... .......... .......... .......... .......... 83% 23.0M 1s 2024-04-21T08:46:24.899 INFO:tasks.workunit.client.0.smithi057.stderr:113100K .......... .......... .......... .......... .......... 83% 44.4M 1s 2024-04-21T08:46:24.901 INFO:tasks.workunit.client.0.smithi057.stderr:113150K .......... .......... .......... .......... .......... 83% 24.6M 1s 2024-04-21T08:46:24.902 INFO:tasks.workunit.client.0.smithi057.stderr:113200K .......... .......... .......... .......... .......... 83% 39.9M 1s 2024-04-21T08:46:24.904 INFO:tasks.workunit.client.0.smithi057.stderr:113250K .......... .......... .......... .......... .......... 83% 23.1M 1s 2024-04-21T08:46:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:113300K .......... .......... .......... .......... .......... 83% 23.5M 1s 2024-04-21T08:46:24.907 INFO:tasks.workunit.client.0.smithi057.stderr:113350K .......... .......... .......... .......... .......... 83% 51.5M 1s 2024-04-21T08:46:24.909 INFO:tasks.workunit.client.0.smithi057.stderr:113400K .......... .......... .......... .......... .......... 83% 23.4M 1s 2024-04-21T08:46:24.911 INFO:tasks.workunit.client.0.smithi057.stderr:113450K .......... .......... .......... .......... .......... 83% 37.8M 1s 2024-04-21T08:46:24.912 INFO:tasks.workunit.client.0.smithi057.stderr:113500K .......... .......... .......... .......... .......... 83% 29.2M 1s 2024-04-21T08:46:24.915 INFO:tasks.workunit.client.0.smithi057.stderr:113550K .......... .......... .......... .......... .......... 83% 20.5M 1s 2024-04-21T08:46:24.915 INFO:tasks.workunit.client.0.smithi057.stderr:113600K .......... .......... .......... .......... .......... 83% 65.5M 1s 2024-04-21T08:46:24.918 INFO:tasks.workunit.client.0.smithi057.stderr:113650K .......... .......... .......... .......... .......... 83% 22.2M 1s 2024-04-21T08:46:24.919 INFO:tasks.workunit.client.0.smithi057.stderr:113700K .......... .......... .......... .......... .......... 83% 37.3M 1s 2024-04-21T08:46:24.921 INFO:tasks.workunit.client.0.smithi057.stderr:113750K .......... .......... .......... .......... .......... 83% 24.1M 1s 2024-04-21T08:46:24.923 INFO:tasks.workunit.client.0.smithi057.stderr:113800K .......... .......... .......... .......... .......... 83% 23.2M 1s 2024-04-21T08:46:24.924 INFO:tasks.workunit.client.0.smithi057.stderr:113850K .......... .......... .......... .......... .......... 83% 50.2M 1s 2024-04-21T08:46:24.926 INFO:tasks.workunit.client.0.smithi057.stderr:113900K .......... .......... .......... .......... .......... 83% 22.5M 1s 2024-04-21T08:46:24.928 INFO:tasks.workunit.client.0.smithi057.stderr:113950K .......... .......... .......... .......... .......... 83% 37.7M 1s 2024-04-21T08:46:24.930 INFO:tasks.workunit.client.0.smithi057.stderr:114000K .......... .......... .......... .......... .......... 84% 24.7M 1s 2024-04-21T08:46:24.932 INFO:tasks.workunit.client.0.smithi057.stderr:114050K .......... .......... .......... .......... .......... 84% 23.9M 1s 2024-04-21T08:46:24.933 INFO:tasks.workunit.client.0.smithi057.stderr:114100K .......... .......... .......... .......... .......... 84% 45.6M 1s 2024-04-21T08:46:24.935 INFO:tasks.workunit.client.0.smithi057.stderr:114150K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-04-21T08:46:24.936 INFO:tasks.workunit.client.0.smithi057.stderr:114200K .......... .......... .......... .......... .......... 84% 37.8M 1s 2024-04-21T08:46:24.938 INFO:tasks.workunit.client.0.smithi057.stderr:114250K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-04-21T08:46:24.940 INFO:tasks.workunit.client.0.smithi057.stderr:114300K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-04-21T08:46:24.941 INFO:tasks.workunit.client.0.smithi057.stderr:114350K .......... .......... .......... .......... .......... 84% 47.5M 1s 2024-04-21T08:46:24.943 INFO:tasks.workunit.client.0.smithi057.stderr:114400K .......... .......... .......... .......... .......... 84% 23.0M 1s 2024-04-21T08:46:24.944 INFO:tasks.workunit.client.0.smithi057.stderr:114450K .......... .......... .......... .......... .......... 84% 42.8M 1s 2024-04-21T08:46:24.946 INFO:tasks.workunit.client.0.smithi057.stderr:114500K .......... .......... .......... .......... .......... 84% 24.2M 1s 2024-04-21T08:46:24.948 INFO:tasks.workunit.client.0.smithi057.stderr:114550K .......... .......... .......... .......... .......... 84% 23.9M 1s 2024-04-21T08:46:24.949 INFO:tasks.workunit.client.0.smithi057.stderr:114600K .......... .......... .......... .......... .......... 84% 47.8M 1s 2024-04-21T08:46:24.951 INFO:tasks.workunit.client.0.smithi057.stderr:114650K .......... .......... .......... .......... .......... 84% 24.0M 1s 2024-04-21T08:46:24.953 INFO:tasks.workunit.client.0.smithi057.stderr:114700K .......... .......... .......... .......... .......... 84% 39.2M 1s 2024-04-21T08:46:24.954 INFO:tasks.workunit.client.0.smithi057.stderr:114750K .......... .......... .......... .......... .......... 84% 26.4M 1s 2024-04-21T08:46:24.957 INFO:tasks.workunit.client.0.smithi057.stderr:114800K .......... .......... .......... .......... .......... 84% 21.9M 1s 2024-04-21T08:46:24.958 INFO:tasks.workunit.client.0.smithi057.stderr:114850K .......... .......... .......... .......... .......... 84% 47.9M 1s 2024-04-21T08:46:24.960 INFO:tasks.workunit.client.0.smithi057.stderr:114900K .......... .......... .......... .......... .......... 84% 23.8M 1s 2024-04-21T08:46:24.961 INFO:tasks.workunit.client.0.smithi057.stderr:114950K .......... .......... .......... .......... .......... 84% 38.4M 1s 2024-04-21T08:46:24.963 INFO:tasks.workunit.client.0.smithi057.stderr:115000K .......... .......... .......... .......... .......... 84% 23.2M 1s 2024-04-21T08:46:24.965 INFO:tasks.workunit.client.0.smithi057.stderr:115050K .......... .......... .......... .......... .......... 84% 22.7M 1s 2024-04-21T08:46:24.967 INFO:tasks.workunit.client.0.smithi057.stderr:115100K .......... .......... .......... .......... .......... 84% 48.7M 1s 2024-04-21T08:46:24.968 INFO:tasks.workunit.client.0.smithi057.stderr:115150K .......... .......... .......... .......... .......... 84% 23.1M 1s 2024-04-21T08:46:24.970 INFO:tasks.workunit.client.0.smithi057.stderr:115200K .......... .......... .......... .......... .......... 84% 37.1M 1s 2024-04-21T08:46:24.972 INFO:tasks.workunit.client.0.smithi057.stderr:115250K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-04-21T08:46:24.974 INFO:tasks.workunit.client.0.smithi057.stderr:115300K .......... .......... .......... .......... .......... 84% 21.4M 1s 2024-04-21T08:46:24.975 INFO:tasks.workunit.client.0.smithi057.stderr:115350K .......... .......... .......... .......... .......... 85% 43.8M 1s 2024-04-21T08:46:24.977 INFO:tasks.workunit.client.0.smithi057.stderr:115400K .......... .......... .......... .......... .......... 85% 23.6M 1s 2024-04-21T08:46:24.978 INFO:tasks.workunit.client.0.smithi057.stderr:115450K .......... .......... .......... .......... .......... 85% 36.5M 1s 2024-04-21T08:46:24.981 INFO:tasks.workunit.client.0.smithi057.stderr:115500K .......... .......... .......... .......... .......... 85% 25.8M 1s 2024-04-21T08:46:24.982 INFO:tasks.workunit.client.0.smithi057.stderr:115550K .......... .......... .......... .......... .......... 85% 23.2M 1s 2024-04-21T08:46:24.984 INFO:tasks.workunit.client.0.smithi057.stderr:115600K .......... .......... .......... .......... .......... 85% 44.6M 1s 2024-04-21T08:46:24.985 INFO:tasks.workunit.client.0.smithi057.stderr:115650K .......... .......... .......... .......... .......... 85% 24.5M 1s 2024-04-21T08:46:24.987 INFO:tasks.workunit.client.0.smithi057.stderr:115700K .......... .......... .......... .......... .......... 85% 43.9M 1s 2024-04-21T08:46:24.989 INFO:tasks.workunit.client.0.smithi057.stderr:115750K .......... .......... .......... .......... .......... 85% 21.1M 1s 2024-04-21T08:46:24.991 INFO:tasks.workunit.client.0.smithi057.stderr:115800K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-04-21T08:46:24.992 INFO:tasks.workunit.client.0.smithi057.stderr:115850K .......... .......... .......... .......... .......... 85% 44.8M 1s 2024-04-21T08:46:24.994 INFO:tasks.workunit.client.0.smithi057.stderr:115900K .......... .......... .......... .......... .......... 85% 24.1M 1s 2024-04-21T08:46:24.996 INFO:tasks.workunit.client.0.smithi057.stderr:115950K .......... .......... .......... .......... .......... 85% 35.0M 1s 2024-04-21T08:46:24.998 INFO:tasks.workunit.client.0.smithi057.stderr:116000K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-04-21T08:46:25.000 INFO:tasks.workunit.client.0.smithi057.stderr:116050K .......... .......... .......... .......... .......... 85% 25.2M 1s 2024-04-21T08:46:25.001 INFO:tasks.workunit.client.0.smithi057.stderr:116100K .......... .......... .......... .......... .......... 85% 43.1M 1s 2024-04-21T08:46:25.003 INFO:tasks.workunit.client.0.smithi057.stderr:116150K .......... .......... .......... .......... .......... 85% 21.0M 1s 2024-04-21T08:46:25.005 INFO:tasks.workunit.client.0.smithi057.stderr:116200K .......... .......... .......... .......... .......... 85% 28.2M 1s 2024-04-21T08:46:25.006 INFO:tasks.workunit.client.0.smithi057.stderr:116250K .......... .......... .......... .......... .......... 85% 36.8M 1s 2024-04-21T08:46:25.008 INFO:tasks.workunit.client.0.smithi057.stderr:116300K .......... .......... .......... .......... .......... 85% 24.7M 1s 2024-04-21T08:46:25.009 INFO:tasks.workunit.client.0.smithi057.stderr:116350K .......... .......... .......... .......... .......... 85% 35.3M 1s 2024-04-21T08:46:25.011 INFO:tasks.workunit.client.0.smithi057.stderr:116400K .......... .......... .......... .......... .......... 85% 24.5M 1s 2024-04-21T08:46:25.014 INFO:tasks.workunit.client.0.smithi057.stderr:116450K .......... .......... .......... .......... .......... 85% 22.5M 1s 2024-04-21T08:46:25.015 INFO:tasks.workunit.client.0.smithi057.stderr:116500K .......... .......... .......... .......... .......... 85% 44.9M 1s 2024-04-21T08:46:25.017 INFO:tasks.workunit.client.0.smithi057.stderr:116550K .......... .......... .......... .......... .......... 85% 22.9M 1s 2024-04-21T08:46:25.018 INFO:tasks.workunit.client.0.smithi057.stderr:116600K .......... .......... .......... .......... .......... 85% 45.0M 1s 2024-04-21T08:46:25.020 INFO:tasks.workunit.client.0.smithi057.stderr:116650K .......... .......... .......... .......... .......... 85% 25.1M 1s 2024-04-21T08:46:25.021 INFO:tasks.workunit.client.0.smithi057.stderr:116700K .......... .......... .......... .......... .......... 86% 38.1M 1s 2024-04-21T08:46:25.023 INFO:tasks.workunit.client.0.smithi057.stderr:116750K .......... .......... .......... .......... .......... 86% 25.3M 1s 2024-04-21T08:46:25.025 INFO:tasks.workunit.client.0.smithi057.stderr:116800K .......... .......... .......... .......... .......... 86% 32.3M 1s 2024-04-21T08:46:25.026 INFO:tasks.workunit.client.0.smithi057.stderr:116850K .......... .......... .......... .......... .......... 86% 32.7M 1s 2024-04-21T08:46:25.028 INFO:tasks.workunit.client.0.smithi057.stderr:116900K .......... .......... .......... .......... .......... 86% 23.6M 1s 2024-04-21T08:46:25.031 INFO:tasks.workunit.client.0.smithi057.stderr:116950K .......... .......... .......... .......... .......... 86% 22.7M 1s 2024-04-21T08:46:25.032 INFO:tasks.workunit.client.0.smithi057.stderr:117000K .......... .......... .......... .......... .......... 86% 32.6M 1s 2024-04-21T08:46:25.033 INFO:tasks.workunit.client.0.smithi057.stderr:117050K .......... .......... .......... .......... .......... 86% 40.3M 1s 2024-04-21T08:46:25.034 INFO:tasks.workunit.client.0.smithi057.stderr:117100K .......... .......... .......... .......... .......... 86% 33.8M 1s 2024-04-21T08:46:25.037 INFO:tasks.workunit.client.0.smithi057.stderr:117150K .......... .......... .......... .......... .......... 86% 22.7M 1s 2024-04-21T08:46:25.039 INFO:tasks.workunit.client.0.smithi057.stderr:117200K .......... .......... .......... .......... .......... 86% 23.2M 1s 2024-04-21T08:46:25.040 INFO:tasks.workunit.client.0.smithi057.stderr:117250K .......... .......... .......... .......... .......... 86% 48.7M 1s 2024-04-21T08:46:25.042 INFO:tasks.workunit.client.0.smithi057.stderr:117300K .......... .......... .......... .......... .......... 86% 23.2M 1s 2024-04-21T08:46:25.044 INFO:tasks.workunit.client.0.smithi057.stderr:117350K .......... .......... .......... .......... .......... 86% 38.4M 1s 2024-04-21T08:46:25.045 INFO:tasks.workunit.client.0.smithi057.stderr:117400K .......... .......... .......... .......... .......... 86% 26.8M 1s 2024-04-21T08:46:25.047 INFO:tasks.workunit.client.0.smithi057.stderr:117450K .......... .......... .......... .......... .......... 86% 23.0M 1s 2024-04-21T08:46:25.048 INFO:tasks.workunit.client.0.smithi057.stderr:117500K .......... .......... .......... .......... .......... 86% 54.2M 1s 2024-04-21T08:46:25.050 INFO:tasks.workunit.client.0.smithi057.stderr:117550K .......... .......... .......... .......... .......... 86% 22.1M 1s 2024-04-21T08:46:25.052 INFO:tasks.workunit.client.0.smithi057.stderr:117600K .......... .......... .......... .......... .......... 86% 44.7M 1s 2024-04-21T08:46:25.053 INFO:tasks.workunit.client.0.smithi057.stderr:117650K .......... .......... .......... .......... .......... 86% 23.7M 1s 2024-04-21T08:46:25.056 INFO:tasks.workunit.client.0.smithi057.stderr:117700K .......... .......... .......... .......... .......... 86% 23.3M 1s 2024-04-21T08:46:25.056 INFO:tasks.workunit.client.0.smithi057.stderr:117750K .......... .......... .......... .......... .......... 86% 40.9M 1s 2024-04-21T08:46:25.059 INFO:tasks.workunit.client.0.smithi057.stderr:117800K .......... .......... .......... .......... .......... 86% 23.1M 1s 2024-04-21T08:46:25.060 INFO:tasks.workunit.client.0.smithi057.stderr:117850K .......... .......... .......... .......... .......... 86% 38.8M 1s 2024-04-21T08:46:25.062 INFO:tasks.workunit.client.0.smithi057.stderr:117900K .......... .......... .......... .......... .......... 86% 28.3M 1s 2024-04-21T08:46:25.064 INFO:tasks.workunit.client.0.smithi057.stderr:117950K .......... .......... .......... .......... .......... 86% 18.0M 1s 2024-04-21T08:46:25.067 INFO:tasks.workunit.client.0.smithi057.stderr:118000K .......... .......... .......... .......... .......... 86% 27.8M 1s 2024-04-21T08:46:25.068 INFO:tasks.workunit.client.0.smithi057.stderr:118050K .......... .......... .......... .......... .......... 86% 64.4M 1s 2024-04-21T08:46:25.070 INFO:tasks.workunit.client.0.smithi057.stderr:118100K .......... .......... .......... .......... .......... 87% 17.3M 1s 2024-04-21T08:46:25.071 INFO:tasks.workunit.client.0.smithi057.stderr:118150K .......... .......... .......... .......... .......... 87% 46.8M 1s 2024-04-21T08:46:25.073 INFO:tasks.workunit.client.0.smithi057.stderr:118200K .......... .......... .......... .......... .......... 87% 32.6M 1s 2024-04-21T08:46:25.074 INFO:tasks.workunit.client.0.smithi057.stderr:118250K .......... .......... .......... .......... .......... 87% 27.4M 1s 2024-04-21T08:46:25.077 INFO:tasks.workunit.client.0.smithi057.stderr:118300K .......... .......... .......... .......... .......... 87% 17.4M 1s 2024-04-21T08:46:25.078 INFO:tasks.workunit.client.0.smithi057.stderr:118350K .......... .......... .......... .......... .......... 87% 31.5M 1s 2024-04-21T08:46:25.080 INFO:tasks.workunit.client.0.smithi057.stderr:118400K .......... .......... .......... .......... .......... 87% 44.1M 1s 2024-04-21T08:46:25.081 INFO:tasks.workunit.client.0.smithi057.stderr:118450K .......... .......... .......... .......... .......... 87% 31.1M 1s 2024-04-21T08:46:25.083 INFO:tasks.workunit.client.0.smithi057.stderr:118500K .......... .......... .......... .......... .......... 87% 27.5M 1s 2024-04-21T08:46:25.085 INFO:tasks.workunit.client.0.smithi057.stderr:118550K .......... .......... .......... .......... .......... 87% 24.2M 1s 2024-04-21T08:46:25.087 INFO:tasks.workunit.client.0.smithi057.stderr:118600K .......... .......... .......... .......... .......... 87% 24.3M 1s 2024-04-21T08:46:25.088 INFO:tasks.workunit.client.0.smithi057.stderr:118650K .......... .......... .......... .......... .......... 87% 53.5M 1s 2024-04-21T08:46:25.090 INFO:tasks.workunit.client.0.smithi057.stderr:118700K .......... .......... .......... .......... .......... 87% 24.1M 1s 2024-04-21T08:46:25.091 INFO:tasks.workunit.client.0.smithi057.stderr:118750K .......... .......... .......... .......... .......... 87% 38.7M 1s 2024-04-21T08:46:25.094 INFO:tasks.workunit.client.0.smithi057.stderr:118800K .......... .......... .......... .......... .......... 87% 24.6M 1s 2024-04-21T08:46:25.099 INFO:tasks.workunit.client.0.smithi057.stderr:118850K .......... .......... .......... .......... .......... 87% 27.2M 1s 2024-04-21T08:46:25.100 INFO:tasks.workunit.client.0.smithi057.stderr:118900K .......... .......... .......... .......... .......... 87% 37.4M 1s 2024-04-21T08:46:25.100 INFO:tasks.workunit.client.0.smithi057.stderr:118950K .......... .......... .......... .......... .......... 87% 22.7M 1s 2024-04-21T08:46:25.100 INFO:tasks.workunit.client.0.smithi057.stderr:119000K .......... .......... .......... .......... .......... 87% 44.4M 1s 2024-04-21T08:46:25.101 INFO:tasks.workunit.client.0.smithi057.stderr:119050K .......... .......... .......... .......... .......... 87% 22.9M 1s 2024-04-21T08:46:25.104 INFO:tasks.workunit.client.0.smithi057.stderr:119100K .......... .......... .......... .......... .......... 87% 21.5M 1s 2024-04-21T08:46:25.105 INFO:tasks.workunit.client.0.smithi057.stderr:119150K .......... .......... .......... .......... .......... 87% 43.8M 1s 2024-04-21T08:46:25.107 INFO:tasks.workunit.client.0.smithi057.stderr:119200K .......... .......... .......... .......... .......... 87% 20.3M 1s 2024-04-21T08:46:25.109 INFO:tasks.workunit.client.0.smithi057.stderr:119250K .......... .......... .......... .......... .......... 87% 40.1M 1s 2024-04-21T08:46:25.111 INFO:tasks.workunit.client.0.smithi057.stderr:119300K .......... .......... .......... .......... .......... 87% 23.1M 1s 2024-04-21T08:46:25.113 INFO:tasks.workunit.client.0.smithi057.stderr:119350K .......... .......... .......... .......... .......... 87% 23.6M 1s 2024-04-21T08:46:25.113 INFO:tasks.workunit.client.0.smithi057.stderr:119400K .......... .......... .......... .......... .......... 87% 55.3M 1s 2024-04-21T08:46:25.116 INFO:tasks.workunit.client.0.smithi057.stderr:119450K .......... .......... .......... .......... .......... 88% 25.5M 1s 2024-04-21T08:46:25.117 INFO:tasks.workunit.client.0.smithi057.stderr:119500K .......... .......... .......... .......... .......... 88% 34.2M 1s 2024-04-21T08:46:25.120 INFO:tasks.workunit.client.0.smithi057.stderr:119550K .......... .......... .......... .......... .......... 88% 22.8M 1s 2024-04-21T08:46:25.121 INFO:tasks.workunit.client.0.smithi057.stderr:119600K .......... .......... .......... .......... .......... 88% 24.8M 1s 2024-04-21T08:46:25.122 INFO:tasks.workunit.client.0.smithi057.stderr:119650K .......... .......... .......... .......... .......... 88% 44.7M 1s 2024-04-21T08:46:25.124 INFO:tasks.workunit.client.0.smithi057.stderr:119700K .......... .......... .......... .......... .......... 88% 25.9M 1s 2024-04-21T08:46:25.125 INFO:tasks.workunit.client.0.smithi057.stderr:119750K .......... .......... .......... .......... .......... 88% 31.9M 1s 2024-04-21T08:46:25.128 INFO:tasks.workunit.client.0.smithi057.stderr:119800K .......... .......... .......... .......... .......... 88% 21.6M 1s 2024-04-21T08:46:25.130 INFO:tasks.workunit.client.0.smithi057.stderr:119850K .......... .......... .......... .......... .......... 88% 25.5M 1s 2024-04-21T08:46:25.131 INFO:tasks.workunit.client.0.smithi057.stderr:119900K .......... .......... .......... .......... .......... 88% 48.0M 1s 2024-04-21T08:46:25.133 INFO:tasks.workunit.client.0.smithi057.stderr:119950K .......... .......... .......... .......... .......... 88% 22.9M 1s 2024-04-21T08:46:25.134 INFO:tasks.workunit.client.0.smithi057.stderr:120000K .......... .......... .......... .......... .......... 88% 40.4M 1s 2024-04-21T08:46:25.136 INFO:tasks.workunit.client.0.smithi057.stderr:120050K .......... .......... .......... .......... .......... 88% 24.1M 1s 2024-04-21T08:46:25.138 INFO:tasks.workunit.client.0.smithi057.stderr:120100K .......... .......... .......... .......... .......... 88% 24.1M 1s 2024-04-21T08:46:25.139 INFO:tasks.workunit.client.0.smithi057.stderr:120150K .......... .......... .......... .......... .......... 88% 51.7M 1s 2024-04-21T08:46:25.141 INFO:tasks.workunit.client.0.smithi057.stderr:120200K .......... .......... .......... .......... .......... 88% 22.4M 1s 2024-04-21T08:46:25.142 INFO:tasks.workunit.client.0.smithi057.stderr:120250K .......... .......... .......... .......... .......... 88% 42.2M 1s 2024-04-21T08:46:25.145 INFO:tasks.workunit.client.0.smithi057.stderr:120300K .......... .......... .......... .......... .......... 88% 22.3M 1s 2024-04-21T08:46:25.147 INFO:tasks.workunit.client.0.smithi057.stderr:120350K .......... .......... .......... .......... .......... 88% 23.3M 1s 2024-04-21T08:46:25.147 INFO:tasks.workunit.client.0.smithi057.stderr:120400K .......... .......... .......... .......... .......... 88% 53.8M 1s 2024-04-21T08:46:25.150 INFO:tasks.workunit.client.0.smithi057.stderr:120450K .......... .......... .......... .......... .......... 88% 24.3M 1s 2024-04-21T08:46:25.151 INFO:tasks.workunit.client.0.smithi057.stderr:120500K .......... .......... .......... .......... .......... 88% 37.0M 1s 2024-04-21T08:46:25.153 INFO:tasks.workunit.client.0.smithi057.stderr:120550K .......... .......... .......... .......... .......... 88% 21.5M 1s 2024-04-21T08:46:25.155 INFO:tasks.workunit.client.0.smithi057.stderr:120600K .......... .......... .......... .......... .......... 88% 26.1M 1s 2024-04-21T08:46:25.156 INFO:tasks.workunit.client.0.smithi057.stderr:120650K .......... .......... .......... .......... .......... 88% 49.4M 1s 2024-04-21T08:46:25.158 INFO:tasks.workunit.client.0.smithi057.stderr:120700K .......... .......... .......... .......... .......... 88% 22.7M 1s 2024-04-21T08:46:25.159 INFO:tasks.workunit.client.0.smithi057.stderr:120750K .......... .......... .......... .......... .......... 88% 42.6M 1s 2024-04-21T08:46:25.162 INFO:tasks.workunit.client.0.smithi057.stderr:120800K .......... .......... .......... .......... .......... 89% 23.0M 1s 2024-04-21T08:46:25.164 INFO:tasks.workunit.client.0.smithi057.stderr:120850K .......... .......... .......... .......... .......... 89% 25.0M 1s 2024-04-21T08:46:25.164 INFO:tasks.workunit.client.0.smithi057.stderr:120900K .......... .......... .......... .......... .......... 89% 45.0M 1s 2024-04-21T08:46:25.166 INFO:tasks.workunit.client.0.smithi057.stderr:120950K .......... .......... .......... .......... .......... 89% 27.2M 1s 2024-04-21T08:46:25.171 INFO:tasks.workunit.client.0.smithi057.stderr:121000K .......... .......... .......... .......... .......... 89% 11.1M 1s 2024-04-21T08:46:25.173 INFO:tasks.workunit.client.0.smithi057.stderr:121050K .......... .......... .......... .......... .......... 89% 55.3M 1s 2024-04-21T08:46:25.173 INFO:tasks.workunit.client.0.smithi057.stderr:121100K .......... .......... .......... .......... .......... 89% 62.7M 1s 2024-04-21T08:46:25.173 INFO:tasks.workunit.client.0.smithi057.stderr:121150K .......... .......... .......... .......... .......... 89% 71.2M 1s 2024-04-21T08:46:25.176 INFO:tasks.workunit.client.0.smithi057.stderr:121200K .......... .......... .......... .......... .......... 89% 20.7M 1s 2024-04-21T08:46:25.177 INFO:tasks.workunit.client.0.smithi057.stderr:121250K .......... .......... .......... .......... .......... 89% 41.0M 1s 2024-04-21T08:46:25.179 INFO:tasks.workunit.client.0.smithi057.stderr:121300K .......... .......... .......... .......... .......... 89% 25.6M 1s 2024-04-21T08:46:25.181 INFO:tasks.workunit.client.0.smithi057.stderr:121350K .......... .......... .......... .......... .......... 89% 22.2M 0s 2024-04-21T08:46:25.182 INFO:tasks.workunit.client.0.smithi057.stderr:121400K .......... .......... .......... .......... .......... 89% 46.1M 0s 2024-04-21T08:46:25.184 INFO:tasks.workunit.client.0.smithi057.stderr:121450K .......... .......... .......... .......... .......... 89% 26.1M 0s 2024-04-21T08:46:25.186 INFO:tasks.workunit.client.0.smithi057.stderr:121500K .......... .......... .......... .......... .......... 89% 35.6M 0s 2024-04-21T08:46:25.187 INFO:tasks.workunit.client.0.smithi057.stderr:121550K .......... .......... .......... .......... .......... 89% 25.2M 0s 2024-04-21T08:46:25.189 INFO:tasks.workunit.client.0.smithi057.stderr:121600K .......... .......... .......... .......... .......... 89% 23.6M 0s 2024-04-21T08:46:25.190 INFO:tasks.workunit.client.0.smithi057.stderr:121650K .......... .......... .......... .......... .......... 89% 54.5M 0s 2024-04-21T08:46:25.192 INFO:tasks.workunit.client.0.smithi057.stderr:121700K .......... .......... .......... .......... .......... 89% 22.5M 0s 2024-04-21T08:46:25.194 INFO:tasks.workunit.client.0.smithi057.stderr:121750K .......... .......... .......... .......... .......... 89% 25.0M 0s 2024-04-21T08:46:25.196 INFO:tasks.workunit.client.0.smithi057.stderr:121800K .......... .......... .......... .......... .......... 89% 34.1M 0s 2024-04-21T08:46:25.198 INFO:tasks.workunit.client.0.smithi057.stderr:121850K .......... .......... .......... .......... .......... 89% 24.2M 0s 2024-04-21T08:46:25.199 INFO:tasks.workunit.client.0.smithi057.stderr:121900K .......... .......... .......... .......... .......... 89% 51.8M 0s 2024-04-21T08:46:25.201 INFO:tasks.workunit.client.0.smithi057.stderr:121950K .......... .......... .......... .......... .......... 89% 23.4M 0s 2024-04-21T08:46:25.203 INFO:tasks.workunit.client.0.smithi057.stderr:122000K .......... .......... .......... .......... .......... 89% 23.3M 0s 2024-04-21T08:46:25.204 INFO:tasks.workunit.client.0.smithi057.stderr:122050K .......... .......... .......... .......... .......... 89% 48.2M 0s 2024-04-21T08:46:25.206 INFO:tasks.workunit.client.0.smithi057.stderr:122100K .......... .......... .......... .......... .......... 89% 22.0M 0s 2024-04-21T08:46:25.207 INFO:tasks.workunit.client.0.smithi057.stderr:122150K .......... .......... .......... .......... .......... 90% 45.2M 0s 2024-04-21T08:46:25.209 INFO:tasks.workunit.client.0.smithi057.stderr:122200K .......... .......... .......... .......... .......... 90% 25.8M 0s 2024-04-21T08:46:25.211 INFO:tasks.workunit.client.0.smithi057.stderr:122250K .......... .......... .......... .......... .......... 90% 23.9M 0s 2024-04-21T08:46:25.212 INFO:tasks.workunit.client.0.smithi057.stderr:122300K .......... .......... .......... .......... .......... 90% 40.5M 0s 2024-04-21T08:46:25.214 INFO:tasks.workunit.client.0.smithi057.stderr:122350K .......... .......... .......... .......... .......... 90% 26.2M 0s 2024-04-21T08:46:25.215 INFO:tasks.workunit.client.0.smithi057.stderr:122400K .......... .......... .......... .......... .......... 90% 42.5M 0s 2024-04-21T08:46:25.217 INFO:tasks.workunit.client.0.smithi057.stderr:122450K .......... .......... .......... .......... .......... 90% 22.1M 0s 2024-04-21T08:46:25.220 INFO:tasks.workunit.client.0.smithi057.stderr:122500K .......... .......... .......... .......... .......... 90% 24.5M 0s 2024-04-21T08:46:25.221 INFO:tasks.workunit.client.0.smithi057.stderr:122550K .......... .......... .......... .......... .......... 90% 41.0M 0s 2024-04-21T08:46:25.223 INFO:tasks.workunit.client.0.smithi057.stderr:122600K .......... .......... .......... .......... .......... 90% 24.2M 0s 2024-04-21T08:46:25.225 INFO:tasks.workunit.client.0.smithi057.stderr:122650K .......... .......... .......... .......... .......... 90% 42.8M 0s 2024-04-21T08:46:25.226 INFO:tasks.workunit.client.0.smithi057.stderr:122700K .......... .......... .......... .......... .......... 90% 24.4M 0s 2024-04-21T08:46:25.228 INFO:tasks.workunit.client.0.smithi057.stderr:122750K .......... .......... .......... .......... .......... 90% 23.3M 0s 2024-04-21T08:46:25.229 INFO:tasks.workunit.client.0.smithi057.stderr:122800K .......... .......... .......... .......... .......... 90% 39.9M 0s 2024-04-21T08:46:25.231 INFO:tasks.workunit.client.0.smithi057.stderr:122850K .......... .......... .......... .......... .......... 90% 24.2M 0s 2024-04-21T08:46:25.233 INFO:tasks.workunit.client.0.smithi057.stderr:122900K .......... .......... .......... .......... .......... 90% 47.4M 0s 2024-04-21T08:46:25.234 INFO:tasks.workunit.client.0.smithi057.stderr:122950K .......... .......... .......... .......... .......... 90% 24.4M 0s 2024-04-21T08:46:25.236 INFO:tasks.workunit.client.0.smithi057.stderr:123000K .......... .......... .......... .......... .......... 90% 22.6M 0s 2024-04-21T08:46:25.237 INFO:tasks.workunit.client.0.smithi057.stderr:123050K .......... .......... .......... .......... .......... 90% 42.3M 0s 2024-04-21T08:46:25.240 INFO:tasks.workunit.client.0.smithi057.stderr:123100K .......... .......... .......... .......... .......... 90% 22.6M 0s 2024-04-21T08:46:25.241 INFO:tasks.workunit.client.0.smithi057.stderr:123150K .......... .......... .......... .......... .......... 90% 25.6M 0s 2024-04-21T08:46:25.243 INFO:tasks.workunit.client.0.smithi057.stderr:123200K .......... .......... .......... .......... .......... 90% 39.5M 0s 2024-04-21T08:46:25.245 INFO:tasks.workunit.client.0.smithi057.stderr:123250K .......... .......... .......... .......... .......... 90% 23.5M 0s 2024-04-21T08:46:25.246 INFO:tasks.workunit.client.0.smithi057.stderr:123300K .......... .......... .......... .......... .......... 90% 45.2M 0s 2024-04-21T08:46:25.248 INFO:tasks.workunit.client.0.smithi057.stderr:123350K .......... .......... .......... .......... .......... 90% 22.1M 0s 2024-04-21T08:46:25.250 INFO:tasks.workunit.client.0.smithi057.stderr:123400K .......... .......... .......... .......... .......... 90% 23.4M 0s 2024-04-21T08:46:25.251 INFO:tasks.workunit.client.0.smithi057.stderr:123450K .......... .......... .......... .......... .......... 90% 43.7M 0s 2024-04-21T08:46:25.253 INFO:tasks.workunit.client.0.smithi057.stderr:123500K .......... .......... .......... .......... .......... 91% 25.0M 0s 2024-04-21T08:46:25.255 INFO:tasks.workunit.client.0.smithi057.stderr:123550K .......... .......... .......... .......... .......... 91% 41.0M 0s 2024-04-21T08:46:25.256 INFO:tasks.workunit.client.0.smithi057.stderr:123600K .......... .......... .......... .......... .......... 91% 22.9M 0s 2024-04-21T08:46:25.259 INFO:tasks.workunit.client.0.smithi057.stderr:123650K .......... .......... .......... .......... .......... 91% 24.2M 0s 2024-04-21T08:46:25.259 INFO:tasks.workunit.client.0.smithi057.stderr:123700K .......... .......... .......... .......... .......... 91% 54.0M 0s 2024-04-21T08:46:25.261 INFO:tasks.workunit.client.0.smithi057.stderr:123750K .......... .......... .......... .......... .......... 91% 22.8M 0s 2024-04-21T08:46:25.263 INFO:tasks.workunit.client.0.smithi057.stderr:123800K .......... .......... .......... .......... .......... 91% 40.3M 0s 2024-04-21T08:46:25.265 INFO:tasks.workunit.client.0.smithi057.stderr:123850K .......... .......... .......... .......... .......... 91% 22.3M 0s 2024-04-21T08:46:25.267 INFO:tasks.workunit.client.0.smithi057.stderr:123900K .......... .......... .......... .......... .......... 91% 23.9M 0s 2024-04-21T08:46:25.268 INFO:tasks.workunit.client.0.smithi057.stderr:123950K .......... .......... .......... .......... .......... 91% 43.2M 0s 2024-04-21T08:46:25.270 INFO:tasks.workunit.client.0.smithi057.stderr:124000K .......... .......... .......... .......... .......... 91% 24.1M 0s 2024-04-21T08:46:25.272 INFO:tasks.workunit.client.0.smithi057.stderr:124050K .......... .......... .......... .......... .......... 91% 29.3M 0s 2024-04-21T08:46:25.274 INFO:tasks.workunit.client.0.smithi057.stderr:124100K .......... .......... .......... .......... .......... 91% 23.4M 0s 2024-04-21T08:46:25.276 INFO:tasks.workunit.client.0.smithi057.stderr:124150K .......... .......... .......... .......... .......... 91% 24.3M 0s 2024-04-21T08:46:25.277 INFO:tasks.workunit.client.0.smithi057.stderr:124200K .......... .......... .......... .......... .......... 91% 48.6M 0s 2024-04-21T08:46:25.279 INFO:tasks.workunit.client.0.smithi057.stderr:124250K .......... .......... .......... .......... .......... 91% 24.1M 0s 2024-04-21T08:46:25.280 INFO:tasks.workunit.client.0.smithi057.stderr:124300K .......... .......... .......... .......... .......... 91% 42.4M 0s 2024-04-21T08:46:25.282 INFO:tasks.workunit.client.0.smithi057.stderr:124350K .......... .......... .......... .......... .......... 91% 23.7M 0s 2024-04-21T08:46:25.284 INFO:tasks.workunit.client.0.smithi057.stderr:124400K .......... .......... .......... .......... .......... 91% 22.4M 0s 2024-04-21T08:46:25.285 INFO:tasks.workunit.client.0.smithi057.stderr:124450K .......... .......... .......... .......... .......... 91% 43.2M 0s 2024-04-21T08:46:25.287 INFO:tasks.workunit.client.0.smithi057.stderr:124500K .......... .......... .......... .......... .......... 91% 25.9M 0s 2024-04-21T08:46:25.289 INFO:tasks.workunit.client.0.smithi057.stderr:124550K .......... .......... .......... .......... .......... 91% 33.0M 0s 2024-04-21T08:46:25.291 INFO:tasks.workunit.client.0.smithi057.stderr:124600K .......... .......... .......... .......... .......... 91% 24.8M 0s 2024-04-21T08:46:25.293 INFO:tasks.workunit.client.0.smithi057.stderr:124650K .......... .......... .......... .......... .......... 91% 24.6M 0s 2024-04-21T08:46:25.294 INFO:tasks.workunit.client.0.smithi057.stderr:124700K .......... .......... .......... .......... .......... 91% 46.2M 0s 2024-04-21T08:46:25.296 INFO:tasks.workunit.client.0.smithi057.stderr:124750K .......... .......... .......... .......... .......... 91% 22.1M 0s 2024-04-21T08:46:25.297 INFO:tasks.workunit.client.0.smithi057.stderr:124800K .......... .......... .......... .......... .......... 91% 36.5M 0s 2024-04-21T08:46:25.303 INFO:tasks.workunit.client.0.smithi057.stderr:124850K .......... .......... .......... .......... .......... 92% 25.5M 0s 2024-04-21T08:46:25.303 INFO:tasks.workunit.client.0.smithi057.stderr:124900K .......... .......... .......... .......... .......... 92% 21.4M 0s 2024-04-21T08:46:25.303 INFO:tasks.workunit.client.0.smithi057.stderr:124950K .......... .......... .......... .......... .......... 92% 52.6M 0s 2024-04-21T08:46:25.304 INFO:tasks.workunit.client.0.smithi057.stderr:125000K .......... .......... .......... .......... .......... 92% 23.0M 0s 2024-04-21T08:46:25.306 INFO:tasks.workunit.client.0.smithi057.stderr:125050K .......... .......... .......... .......... .......... 92% 45.4M 0s 2024-04-21T08:46:25.307 INFO:tasks.workunit.client.0.smithi057.stderr:125100K .......... .......... .......... .......... .......... 92% 27.1M 0s 2024-04-21T08:46:25.310 INFO:tasks.workunit.client.0.smithi057.stderr:125150K .......... .......... .......... .......... .......... 92% 20.1M 0s 2024-04-21T08:46:25.311 INFO:tasks.workunit.client.0.smithi057.stderr:125200K .......... .......... .......... .......... .......... 92% 52.2M 0s 2024-04-21T08:46:25.313 INFO:tasks.workunit.client.0.smithi057.stderr:125250K .......... .......... .......... .......... .......... 92% 23.2M 0s 2024-04-21T08:46:25.314 INFO:tasks.workunit.client.0.smithi057.stderr:125300K .......... .......... .......... .......... .......... 92% 43.4M 0s 2024-04-21T08:46:25.316 INFO:tasks.workunit.client.0.smithi057.stderr:125350K .......... .......... .......... .......... .......... 92% 22.4M 0s 2024-04-21T08:46:25.318 INFO:tasks.workunit.client.0.smithi057.stderr:125400K .......... .......... .......... .......... .......... 92% 23.4M 0s 2024-04-21T08:46:25.319 INFO:tasks.workunit.client.0.smithi057.stderr:125450K .......... .......... .......... .......... .......... 92% 55.6M 0s 2024-04-21T08:46:25.321 INFO:tasks.workunit.client.0.smithi057.stderr:125500K .......... .......... .......... .......... .......... 92% 25.7M 0s 2024-04-21T08:46:25.324 INFO:tasks.workunit.client.0.smithi057.stderr:125550K .......... .......... .......... .......... .......... 92% 21.3M 0s 2024-04-21T08:46:25.324 INFO:tasks.workunit.client.0.smithi057.stderr:125600K .......... .......... .......... .......... .......... 92% 73.6M 0s 2024-04-21T08:46:25.327 INFO:tasks.workunit.client.0.smithi057.stderr:125650K .......... .......... .......... .......... .......... 92% 18.8M 0s 2024-04-21T08:46:25.327 INFO:tasks.workunit.client.0.smithi057.stderr:125700K .......... .......... .......... .......... .......... 92% 55.8M 0s 2024-04-21T08:46:25.330 INFO:tasks.workunit.client.0.smithi057.stderr:125750K .......... .......... .......... .......... .......... 92% 23.4M 0s 2024-04-21T08:46:25.331 INFO:tasks.workunit.client.0.smithi057.stderr:125800K .......... .......... .......... .......... .......... 92% 33.5M 0s 2024-04-21T08:46:25.333 INFO:tasks.workunit.client.0.smithi057.stderr:125850K .......... .......... .......... .......... .......... 92% 27.1M 0s 2024-04-21T08:46:25.335 INFO:tasks.workunit.client.0.smithi057.stderr:125900K .......... .......... .......... .......... .......... 92% 23.7M 0s 2024-04-21T08:46:25.336 INFO:tasks.workunit.client.0.smithi057.stderr:125950K .......... .......... .......... .......... .......... 92% 52.1M 0s 2024-04-21T08:46:25.338 INFO:tasks.workunit.client.0.smithi057.stderr:126000K .......... .......... .......... .......... .......... 92% 22.4M 0s 2024-04-21T08:46:25.339 INFO:tasks.workunit.client.0.smithi057.stderr:126050K .......... .......... .......... .......... .......... 92% 36.6M 0s 2024-04-21T08:46:25.341 INFO:tasks.workunit.client.0.smithi057.stderr:126100K .......... .......... .......... .......... .......... 92% 24.8M 0s 2024-04-21T08:46:25.344 INFO:tasks.workunit.client.0.smithi057.stderr:126150K .......... .......... .......... .......... .......... 92% 24.4M 0s 2024-04-21T08:46:25.344 INFO:tasks.workunit.client.0.smithi057.stderr:126200K .......... .......... .......... .......... .......... 92% 58.5M 0s 2024-04-21T08:46:25.346 INFO:tasks.workunit.client.0.smithi057.stderr:126250K .......... .......... .......... .......... .......... 93% 21.5M 0s 2024-04-21T08:46:25.348 INFO:tasks.workunit.client.0.smithi057.stderr:126300K .......... .......... .......... .......... .......... 93% 42.3M 0s 2024-04-21T08:46:25.350 INFO:tasks.workunit.client.0.smithi057.stderr:126350K .......... .......... .......... .......... .......... 93% 23.2M 0s 2024-04-21T08:46:25.352 INFO:tasks.workunit.client.0.smithi057.stderr:126400K .......... .......... .......... .......... .......... 93% 24.9M 0s 2024-04-21T08:46:25.353 INFO:tasks.workunit.client.0.smithi057.stderr:126450K .......... .......... .......... .......... .......... 93% 43.9M 0s 2024-04-21T08:46:25.355 INFO:tasks.workunit.client.0.smithi057.stderr:126500K .......... .......... .......... .......... .......... 93% 25.3M 0s 2024-04-21T08:46:25.356 INFO:tasks.workunit.client.0.smithi057.stderr:126550K .......... .......... .......... .......... .......... 93% 41.8M 0s 2024-04-21T08:46:25.358 INFO:tasks.workunit.client.0.smithi057.stderr:126600K .......... .......... .......... .......... .......... 93% 22.6M 0s 2024-04-21T08:46:25.360 INFO:tasks.workunit.client.0.smithi057.stderr:126650K .......... .......... .......... .......... .......... 93% 22.7M 0s 2024-04-21T08:46:25.361 INFO:tasks.workunit.client.0.smithi057.stderr:126700K .......... .......... .......... .......... .......... 93% 37.5M 0s 2024-04-21T08:46:25.364 INFO:tasks.workunit.client.0.smithi057.stderr:126750K .......... .......... .......... .......... .......... 93% 20.4M 0s 2024-04-21T08:46:25.365 INFO:tasks.workunit.client.0.smithi057.stderr:126800K .......... .......... .......... .......... .......... 93% 35.6M 0s 2024-04-21T08:46:25.367 INFO:tasks.workunit.client.0.smithi057.stderr:126850K .......... .......... .......... .......... .......... 93% 27.4M 0s 2024-04-21T08:46:25.368 INFO:tasks.workunit.client.0.smithi057.stderr:126900K .......... .......... .......... .......... .......... 93% 38.4M 0s 2024-04-21T08:46:25.370 INFO:tasks.workunit.client.0.smithi057.stderr:126950K .......... .......... .......... .......... .......... 93% 28.5M 0s 2024-04-21T08:46:25.372 INFO:tasks.workunit.client.0.smithi057.stderr:127000K .......... .......... .......... .......... .......... 93% 22.7M 0s 2024-04-21T08:46:25.374 INFO:tasks.workunit.client.0.smithi057.stderr:127050K .......... .......... .......... .......... .......... 93% 25.1M 0s 2024-04-21T08:46:25.376 INFO:tasks.workunit.client.0.smithi057.stderr:127100K .......... .......... .......... .......... .......... 93% 32.0M 0s 2024-04-21T08:46:25.378 INFO:tasks.workunit.client.0.smithi057.stderr:127150K .......... .......... .......... .......... .......... 93% 22.2M 0s 2024-04-21T08:46:25.379 INFO:tasks.workunit.client.0.smithi057.stderr:127200K .......... .......... .......... .......... .......... 93% 44.7M 0s 2024-04-21T08:46:25.381 INFO:tasks.workunit.client.0.smithi057.stderr:127250K .......... .......... .......... .......... .......... 93% 24.7M 0s 2024-04-21T08:46:25.383 INFO:tasks.workunit.client.0.smithi057.stderr:127300K .......... .......... .......... .......... .......... 93% 25.2M 0s 2024-04-21T08:46:25.384 INFO:tasks.workunit.client.0.smithi057.stderr:127350K .......... .......... .......... .......... .......... 93% 39.2M 0s 2024-04-21T08:46:25.386 INFO:tasks.workunit.client.0.smithi057.stderr:127400K .......... .......... .......... .......... .......... 93% 23.9M 0s 2024-04-21T08:46:25.387 INFO:tasks.workunit.client.0.smithi057.stderr:127450K .......... .......... .......... .......... .......... 93% 48.5M 0s 2024-04-21T08:46:25.389 INFO:tasks.workunit.client.0.smithi057.stderr:127500K .......... .......... .......... .......... .......... 93% 24.6M 0s 2024-04-21T08:46:25.391 INFO:tasks.workunit.client.0.smithi057.stderr:127550K .......... .......... .......... .......... .......... 93% 22.4M 0s 2024-04-21T08:46:25.393 INFO:tasks.workunit.client.0.smithi057.stderr:127600K .......... .......... .......... .......... .......... 94% 42.9M 0s 2024-04-21T08:46:25.395 INFO:tasks.workunit.client.0.smithi057.stderr:127650K .......... .......... .......... .......... .......... 94% 23.0M 0s 2024-04-21T08:46:25.396 INFO:tasks.workunit.client.0.smithi057.stderr:127700K .......... .......... .......... .......... .......... 94% 46.4M 0s 2024-04-21T08:46:25.398 INFO:tasks.workunit.client.0.smithi057.stderr:127750K .......... .......... .......... .......... .......... 94% 23.1M 0s 2024-04-21T08:46:25.400 INFO:tasks.workunit.client.0.smithi057.stderr:127800K .......... .......... .......... .......... .......... 94% 24.4M 0s 2024-04-21T08:46:25.401 INFO:tasks.workunit.client.0.smithi057.stderr:127850K .......... .......... .......... .......... .......... 94% 40.6M 0s 2024-04-21T08:46:25.403 INFO:tasks.workunit.client.0.smithi057.stderr:127900K .......... .......... .......... .......... .......... 94% 23.6M 0s 2024-04-21T08:46:25.406 INFO:tasks.workunit.client.0.smithi057.stderr:127950K .......... .......... .......... .......... .......... 94% 39.2M 0s 2024-04-21T08:46:25.407 INFO:tasks.workunit.client.0.smithi057.stderr:128000K .......... .......... .......... .......... .......... 94% 20.9M 0s 2024-04-21T08:46:25.408 INFO:tasks.workunit.client.0.smithi057.stderr:128050K .......... .......... .......... .......... .......... 94% 32.1M 0s 2024-04-21T08:46:25.411 INFO:tasks.workunit.client.0.smithi057.stderr:128100K .......... .......... .......... .......... .......... 94% 26.8M 0s 2024-04-21T08:46:25.412 INFO:tasks.workunit.client.0.smithi057.stderr:128150K .......... .......... .......... .......... .......... 94% 24.8M 0s 2024-04-21T08:46:25.414 INFO:tasks.workunit.client.0.smithi057.stderr:128200K .......... .......... .......... .......... .......... 94% 50.0M 0s 2024-04-21T08:46:25.416 INFO:tasks.workunit.client.0.smithi057.stderr:128250K .......... .......... .......... .......... .......... 94% 23.3M 0s 2024-04-21T08:46:25.417 INFO:tasks.workunit.client.0.smithi057.stderr:128300K .......... .......... .......... .......... .......... 94% 26.6M 0s 2024-04-21T08:46:25.418 INFO:tasks.workunit.client.0.smithi057.stderr:128350K .......... .......... .......... .......... .......... 94% 30.1M 0s 2024-04-21T08:46:25.421 INFO:tasks.workunit.client.0.smithi057.stderr:128400K .......... .......... .......... .......... .......... 94% 24.9M 0s 2024-04-21T08:46:25.423 INFO:tasks.workunit.client.0.smithi057.stderr:128450K .......... .......... .......... .......... .......... 94% 25.0M 0s 2024-04-21T08:46:25.423 INFO:tasks.workunit.client.0.smithi057.stderr:128500K .......... .......... .......... .......... .......... 94% 49.4M 0s 2024-04-21T08:46:25.426 INFO:tasks.workunit.client.0.smithi057.stderr:128550K .......... .......... .......... .......... .......... 94% 23.1M 0s 2024-04-21T08:46:25.428 INFO:tasks.workunit.client.0.smithi057.stderr:128600K .......... .......... .......... .......... .......... 94% 30.6M 0s 2024-04-21T08:46:25.429 INFO:tasks.workunit.client.0.smithi057.stderr:128650K .......... .......... .......... .......... .......... 94% 25.3M 0s 2024-04-21T08:46:25.431 INFO:tasks.workunit.client.0.smithi057.stderr:128700K .......... .......... .......... .......... .......... 94% 24.6M 0s 2024-04-21T08:46:25.432 INFO:tasks.workunit.client.0.smithi057.stderr:128750K .......... .......... .......... .......... .......... 94% 34.8M 0s 2024-04-21T08:46:25.436 INFO:tasks.workunit.client.0.smithi057.stderr:128800K .......... .......... .......... .......... .......... 94% 22.1M 0s 2024-04-21T08:46:25.436 INFO:tasks.workunit.client.0.smithi057.stderr:128850K .......... .......... .......... .......... .......... 94% 52.7M 0s 2024-04-21T08:46:25.440 INFO:tasks.workunit.client.0.smithi057.stderr:128900K .......... .......... .......... .......... .......... 94% 14.9M 0s 2024-04-21T08:46:25.440 INFO:tasks.workunit.client.0.smithi057.stderr:128950K .......... .......... .......... .......... .......... 95% 66.1M 0s 2024-04-21T08:46:25.441 INFO:tasks.workunit.client.0.smithi057.stderr:129000K .......... .......... .......... .......... .......... 95% 35.7M 0s 2024-04-21T08:46:25.443 INFO:tasks.workunit.client.0.smithi057.stderr:129050K .......... .......... .......... .......... .......... 95% 27.1M 0s 2024-04-21T08:46:25.444 INFO:tasks.workunit.client.0.smithi057.stderr:129100K .......... .......... .......... .......... .......... 95% 52.7M 0s 2024-04-21T08:46:25.446 INFO:tasks.workunit.client.0.smithi057.stderr:129150K .......... .......... .......... .......... .......... 95% 22.2M 0s 2024-04-21T08:46:25.448 INFO:tasks.workunit.client.0.smithi057.stderr:129200K .......... .......... .......... .......... .......... 95% 24.6M 0s 2024-04-21T08:46:25.449 INFO:tasks.workunit.client.0.smithi057.stderr:129250K .......... .......... .......... .......... .......... 95% 46.4M 0s 2024-04-21T08:46:25.451 INFO:tasks.workunit.client.0.smithi057.stderr:129300K .......... .......... .......... .......... .......... 95% 28.7M 0s 2024-04-21T08:46:25.452 INFO:tasks.workunit.client.0.smithi057.stderr:129350K .......... .......... .......... .......... .......... 95% 34.1M 0s 2024-04-21T08:46:25.454 INFO:tasks.workunit.client.0.smithi057.stderr:129400K .......... .......... .......... .......... .......... 95% 23.1M 0s 2024-04-21T08:46:25.457 INFO:tasks.workunit.client.0.smithi057.stderr:129450K .......... .......... .......... .......... .......... 95% 24.3M 0s 2024-04-21T08:46:25.458 INFO:tasks.workunit.client.0.smithi057.stderr:129500K .......... .......... .......... .......... .......... 95% 50.3M 0s 2024-04-21T08:46:25.459 INFO:tasks.workunit.client.0.smithi057.stderr:129550K .......... .......... .......... .......... .......... 95% 26.5M 0s 2024-04-21T08:46:25.461 INFO:tasks.workunit.client.0.smithi057.stderr:129600K .......... .......... .......... .......... .......... 95% 25.2M 0s 2024-04-21T08:46:25.463 INFO:tasks.workunit.client.0.smithi057.stderr:129650K .......... .......... .......... .......... .......... 95% 31.6M 0s 2024-04-21T08:46:25.465 INFO:tasks.workunit.client.0.smithi057.stderr:129700K .......... .......... .......... .......... .......... 95% 22.5M 0s 2024-04-21T08:46:25.465 INFO:tasks.workunit.client.0.smithi057.stderr:129750K .......... .......... .......... .......... .......... 95% 52.2M 0s 2024-04-21T08:46:25.468 INFO:tasks.workunit.client.0.smithi057.stderr:129800K .......... .......... .......... .......... .......... 95% 27.6M 0s 2024-04-21T08:46:25.469 INFO:tasks.workunit.client.0.smithi057.stderr:129850K .......... .......... .......... .......... .......... 95% 29.7M 0s 2024-04-21T08:46:25.471 INFO:tasks.workunit.client.0.smithi057.stderr:129900K .......... .......... .......... .......... .......... 95% 24.0M 0s 2024-04-21T08:46:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:129950K .......... .......... .......... .......... .......... 95% 20.7M 0s 2024-04-21T08:46:25.474 INFO:tasks.workunit.client.0.smithi057.stderr:130000K .......... .......... .......... .......... .......... 95% 53.5M 0s 2024-04-21T08:46:25.476 INFO:tasks.workunit.client.0.smithi057.stderr:130050K .......... .......... .......... .......... .......... 95% 29.1M 0s 2024-04-21T08:46:25.478 INFO:tasks.workunit.client.0.smithi057.stderr:130100K .......... .......... .......... .......... .......... 95% 28.7M 0s 2024-04-21T08:46:25.480 INFO:tasks.workunit.client.0.smithi057.stderr:130150K .......... .......... .......... .......... .......... 95% 23.9M 0s 2024-04-21T08:46:25.481 INFO:tasks.workunit.client.0.smithi057.stderr:130200K .......... .......... .......... .......... .......... 95% 31.2M 0s 2024-04-21T08:46:25.483 INFO:tasks.workunit.client.0.smithi057.stderr:130250K .......... .......... .......... .......... .......... 95% 37.6M 0s 2024-04-21T08:46:25.485 INFO:tasks.workunit.client.0.smithi057.stderr:130300K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-21T08:46:25.486 INFO:tasks.workunit.client.0.smithi057.stderr:130350K .......... .......... .......... .......... .......... 96% 36.2M 0s 2024-04-21T08:46:25.489 INFO:tasks.workunit.client.0.smithi057.stderr:130400K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-04-21T08:46:25.491 INFO:tasks.workunit.client.0.smithi057.stderr:130450K .......... .......... .......... .......... .......... 96% 24.7M 0s 2024-04-21T08:46:25.491 INFO:tasks.workunit.client.0.smithi057.stderr:130500K .......... .......... .......... .......... .......... 96% 47.6M 0s 2024-04-21T08:46:25.493 INFO:tasks.workunit.client.0.smithi057.stderr:130550K .......... .......... .......... .......... .......... 96% 24.5M 0s 2024-04-21T08:46:25.494 INFO:tasks.workunit.client.0.smithi057.stderr:130600K .......... .......... .......... .......... .......... 96% 41.1M 0s 2024-04-21T08:46:25.497 INFO:tasks.workunit.client.0.smithi057.stderr:130650K .......... .......... .......... .......... .......... 96% 22.2M 0s 2024-04-21T08:46:25.498 INFO:tasks.workunit.client.0.smithi057.stderr:130700K .......... .......... .......... .......... .......... 96% 25.2M 0s 2024-04-21T08:46:25.500 INFO:tasks.workunit.client.0.smithi057.stderr:130750K .......... .......... .......... .......... .......... 96% 40.6M 0s 2024-04-21T08:46:25.502 INFO:tasks.workunit.client.0.smithi057.stderr:130800K .......... .......... .......... .......... .......... 96% 24.0M 0s 2024-04-21T08:46:25.509 INFO:tasks.workunit.client.0.smithi057.stderr:130850K .......... .......... .......... .......... .......... 96% 43.9M 0s 2024-04-21T08:46:25.509 INFO:tasks.workunit.client.0.smithi057.stderr:130900K .......... .......... .......... .......... .......... 96% 22.0M 0s 2024-04-21T08:46:25.509 INFO:tasks.workunit.client.0.smithi057.stderr:130950K .......... .......... .......... .......... .......... 96% 26.2M 0s 2024-04-21T08:46:25.510 INFO:tasks.workunit.client.0.smithi057.stderr:131000K .......... .......... .......... .......... .......... 96% 45.1M 0s 2024-04-21T08:46:25.510 INFO:tasks.workunit.client.0.smithi057.stderr:131050K .......... .......... .......... .......... .......... 96% 23.2M 0s 2024-04-21T08:46:25.511 INFO:tasks.workunit.client.0.smithi057.stderr:131100K .......... .......... .......... .......... .......... 96% 43.2M 0s 2024-04-21T08:46:25.513 INFO:tasks.workunit.client.0.smithi057.stderr:131150K .......... .......... .......... .......... .......... 96% 22.9M 0s 2024-04-21T08:46:25.515 INFO:tasks.workunit.client.0.smithi057.stderr:131200K .......... .......... .......... .......... .......... 96% 24.7M 0s 2024-04-21T08:46:25.516 INFO:tasks.workunit.client.0.smithi057.stderr:131250K .......... .......... .......... .......... .......... 96% 46.8M 0s 2024-04-21T08:46:25.519 INFO:tasks.workunit.client.0.smithi057.stderr:131300K .......... .......... .......... .......... .......... 96% 23.7M 0s 2024-04-21T08:46:25.520 INFO:tasks.workunit.client.0.smithi057.stderr:131350K .......... .......... .......... .......... .......... 96% 38.9M 0s 2024-04-21T08:46:25.522 INFO:tasks.workunit.client.0.smithi057.stderr:131400K .......... .......... .......... .......... .......... 96% 24.4M 0s 2024-04-21T08:46:25.524 INFO:tasks.workunit.client.0.smithi057.stderr:131450K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-21T08:46:25.525 INFO:tasks.workunit.client.0.smithi057.stderr:131500K .......... .......... .......... .......... .......... 96% 53.0M 0s 2024-04-21T08:46:25.527 INFO:tasks.workunit.client.0.smithi057.stderr:131550K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-21T08:46:25.529 INFO:tasks.workunit.client.0.smithi057.stderr:131600K .......... .......... .......... .......... .......... 96% 42.7M 0s 2024-04-21T08:46:25.530 INFO:tasks.workunit.client.0.smithi057.stderr:131650K .......... .......... .......... .......... .......... 97% 23.6M 0s 2024-04-21T08:46:25.532 INFO:tasks.workunit.client.0.smithi057.stderr:131700K .......... .......... .......... .......... .......... 97% 22.6M 0s 2024-04-21T08:46:25.533 INFO:tasks.workunit.client.0.smithi057.stderr:131750K .......... .......... .......... .......... .......... 97% 45.5M 0s 2024-04-21T08:46:25.535 INFO:tasks.workunit.client.0.smithi057.stderr:131800K .......... .......... .......... .......... .......... 97% 24.3M 0s 2024-04-21T08:46:25.537 INFO:tasks.workunit.client.0.smithi057.stderr:131850K .......... .......... .......... .......... .......... 97% 36.4M 0s 2024-04-21T08:46:25.539 INFO:tasks.workunit.client.0.smithi057.stderr:131900K .......... .......... .......... .......... .......... 97% 23.1M 0s 2024-04-21T08:46:25.541 INFO:tasks.workunit.client.0.smithi057.stderr:131950K .......... .......... .......... .......... .......... 97% 23.9M 0s 2024-04-21T08:46:25.542 INFO:tasks.workunit.client.0.smithi057.stderr:132000K .......... .......... .......... .......... .......... 97% 48.8M 0s 2024-04-21T08:46:25.544 INFO:tasks.workunit.client.0.smithi057.stderr:132050K .......... .......... .......... .......... .......... 97% 23.8M 0s 2024-04-21T08:46:25.546 INFO:tasks.workunit.client.0.smithi057.stderr:132100K .......... .......... .......... .......... .......... 97% 24.5M 0s 2024-04-21T08:46:25.547 INFO:tasks.workunit.client.0.smithi057.stderr:132150K .......... .......... .......... .......... .......... 97% 37.7M 0s 2024-04-21T08:46:25.549 INFO:tasks.workunit.client.0.smithi057.stderr:132200K .......... .......... .......... .......... .......... 97% 24.1M 0s 2024-04-21T08:46:25.550 INFO:tasks.workunit.client.0.smithi057.stderr:132250K .......... .......... .......... .......... .......... 97% 58.3M 0s 2024-04-21T08:46:25.552 INFO:tasks.workunit.client.0.smithi057.stderr:132300K .......... .......... .......... .......... .......... 97% 22.1M 0s 2024-04-21T08:46:25.554 INFO:tasks.workunit.client.0.smithi057.stderr:132350K .......... .......... .......... .......... .......... 97% 25.6M 0s 2024-04-21T08:46:25.556 INFO:tasks.workunit.client.0.smithi057.stderr:132400K .......... .......... .......... .......... .......... 97% 34.6M 0s 2024-04-21T08:46:25.558 INFO:tasks.workunit.client.0.smithi057.stderr:132450K .......... .......... .......... .......... .......... 97% 25.0M 0s 2024-04-21T08:46:25.558 INFO:tasks.workunit.client.0.smithi057.stderr:132500K .......... .......... .......... .......... .......... 97% 52.7M 0s 2024-04-21T08:46:25.561 INFO:tasks.workunit.client.0.smithi057.stderr:132550K .......... .......... .......... .......... .......... 97% 22.8M 0s 2024-04-21T08:46:25.563 INFO:tasks.workunit.client.0.smithi057.stderr:132600K .......... .......... .......... .......... .......... 97% 24.8M 0s 2024-04-21T08:46:25.564 INFO:tasks.workunit.client.0.smithi057.stderr:132650K .......... .......... .......... .......... .......... 97% 37.1M 0s 2024-04-21T08:46:25.566 INFO:tasks.workunit.client.0.smithi057.stderr:132700K .......... .......... .......... .......... .......... 97% 22.4M 0s 2024-04-21T08:46:25.567 INFO:tasks.workunit.client.0.smithi057.stderr:132750K .......... .......... .......... .......... .......... 97% 49.4M 0s 2024-04-21T08:46:25.569 INFO:tasks.workunit.client.0.smithi057.stderr:132800K .......... .......... .......... .......... .......... 97% 24.1M 0s 2024-04-21T08:46:25.572 INFO:tasks.workunit.client.0.smithi057.stderr:132850K .......... .......... .......... .......... .......... 97% 19.3M 0s 2024-04-21T08:46:25.573 INFO:tasks.workunit.client.0.smithi057.stderr:132900K .......... .......... .......... .......... .......... 97% 40.7M 0s 2024-04-21T08:46:25.575 INFO:tasks.workunit.client.0.smithi057.stderr:132950K .......... .......... .......... .......... .......... 97% 21.2M 0s 2024-04-21T08:46:25.576 INFO:tasks.workunit.client.0.smithi057.stderr:133000K .......... .......... .......... .......... .......... 98% 60.6M 0s 2024-04-21T08:46:25.578 INFO:tasks.workunit.client.0.smithi057.stderr:133050K .......... .......... .......... .......... .......... 98% 22.3M 0s 2024-04-21T08:46:25.580 INFO:tasks.workunit.client.0.smithi057.stderr:133100K .......... .......... .......... .......... .......... 98% 24.8M 0s 2024-04-21T08:46:25.581 INFO:tasks.workunit.client.0.smithi057.stderr:133150K .......... .......... .......... .......... .......... 98% 37.3M 0s 2024-04-21T08:46:25.583 INFO:tasks.workunit.client.0.smithi057.stderr:133200K .......... .......... .......... .......... .......... 98% 23.8M 0s 2024-04-21T08:46:25.585 INFO:tasks.workunit.client.0.smithi057.stderr:133250K .......... .......... .......... .......... .......... 98% 48.6M 0s 2024-04-21T08:46:25.586 INFO:tasks.workunit.client.0.smithi057.stderr:133300K .......... .......... .......... .......... .......... 98% 24.2M 0s 2024-04-21T08:46:25.588 INFO:tasks.workunit.client.0.smithi057.stderr:133350K .......... .......... .......... .......... .......... 98% 21.8M 0s 2024-04-21T08:46:25.590 INFO:tasks.workunit.client.0.smithi057.stderr:133400K .......... .......... .......... .......... .......... 98% 36.9M 0s 2024-04-21T08:46:25.592 INFO:tasks.workunit.client.0.smithi057.stderr:133450K .......... .......... .......... .......... .......... 98% 24.8M 0s 2024-04-21T08:46:25.594 INFO:tasks.workunit.client.0.smithi057.stderr:133500K .......... .......... .......... .......... .......... 98% 45.2M 0s 2024-04-21T08:46:25.595 INFO:tasks.workunit.client.0.smithi057.stderr:133550K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-04-21T08:46:25.597 INFO:tasks.workunit.client.0.smithi057.stderr:133600K .......... .......... .......... .......... .......... 98% 21.4M 0s 2024-04-21T08:46:25.599 INFO:tasks.workunit.client.0.smithi057.stderr:133650K .......... .......... .......... .......... .......... 98% 37.3M 0s 2024-04-21T08:46:25.601 INFO:tasks.workunit.client.0.smithi057.stderr:133700K .......... .......... .......... .......... .......... 98% 23.6M 0s 2024-04-21T08:46:25.603 INFO:tasks.workunit.client.0.smithi057.stderr:133750K .......... .......... .......... .......... .......... 98% 24.6M 0s 2024-04-21T08:46:25.604 INFO:tasks.workunit.client.0.smithi057.stderr:133800K .......... .......... .......... .......... .......... 98% 47.0M 0s 2024-04-21T08:46:25.606 INFO:tasks.workunit.client.0.smithi057.stderr:133850K .......... .......... .......... .......... .......... 98% 24.0M 0s 2024-04-21T08:46:25.607 INFO:tasks.workunit.client.0.smithi057.stderr:133900K .......... .......... .......... .......... .......... 98% 35.2M 0s 2024-04-21T08:46:25.609 INFO:tasks.workunit.client.0.smithi057.stderr:133950K .......... .......... .......... .......... .......... 98% 24.4M 0s 2024-04-21T08:46:25.611 INFO:tasks.workunit.client.0.smithi057.stderr:134000K .......... .......... .......... .......... .......... 98% 24.6M 0s 2024-04-21T08:46:25.612 INFO:tasks.workunit.client.0.smithi057.stderr:134050K .......... .......... .......... .......... .......... 98% 51.3M 0s 2024-04-21T08:46:25.614 INFO:tasks.workunit.client.0.smithi057.stderr:134100K .......... .......... .......... .......... .......... 98% 23.4M 0s 2024-04-21T08:46:25.615 INFO:tasks.workunit.client.0.smithi057.stderr:134150K .......... .......... .......... .......... .......... 98% 42.2M 0s 2024-04-21T08:46:25.617 INFO:tasks.workunit.client.0.smithi057.stderr:134200K .......... .......... .......... .......... .......... 98% 23.4M 0s 2024-04-21T08:46:25.619 INFO:tasks.workunit.client.0.smithi057.stderr:134250K .......... .......... .......... .......... .......... 98% 23.5M 0s 2024-04-21T08:46:25.620 INFO:tasks.workunit.client.0.smithi057.stderr:134300K .......... .......... .......... .......... .......... 98% 50.9M 0s 2024-04-21T08:46:25.623 INFO:tasks.workunit.client.0.smithi057.stderr:134350K .......... .......... .......... .......... .......... 99% 23.3M 0s 2024-04-21T08:46:25.624 INFO:tasks.workunit.client.0.smithi057.stderr:134400K .......... .......... .......... .......... .......... 99% 43.2M 0s 2024-04-21T08:46:25.626 INFO:tasks.workunit.client.0.smithi057.stderr:134450K .......... .......... .......... .......... .......... 99% 23.3M 0s 2024-04-21T08:46:25.628 INFO:tasks.workunit.client.0.smithi057.stderr:134500K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-04-21T08:46:25.629 INFO:tasks.workunit.client.0.smithi057.stderr:134550K .......... .......... .......... .......... .......... 99% 43.4M 0s 2024-04-21T08:46:25.631 INFO:tasks.workunit.client.0.smithi057.stderr:134600K .......... .......... .......... .......... .......... 99% 24.6M 0s 2024-04-21T08:46:25.632 INFO:tasks.workunit.client.0.smithi057.stderr:134650K .......... .......... .......... .......... .......... 99% 43.8M 0s 2024-04-21T08:46:25.634 INFO:tasks.workunit.client.0.smithi057.stderr:134700K .......... .......... .......... .......... .......... 99% 20.6M 0s 2024-04-21T08:46:25.636 INFO:tasks.workunit.client.0.smithi057.stderr:134750K .......... .......... .......... .......... .......... 99% 26.8M 0s 2024-04-21T08:46:25.637 INFO:tasks.workunit.client.0.smithi057.stderr:134800K .......... .......... .......... .......... .......... 99% 49.4M 0s 2024-04-21T08:46:25.639 INFO:tasks.workunit.client.0.smithi057.stderr:134850K .......... .......... .......... .......... .......... 99% 24.9M 0s 2024-04-21T08:46:25.642 INFO:tasks.workunit.client.0.smithi057.stderr:134900K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-04-21T08:46:25.643 INFO:tasks.workunit.client.0.smithi057.stderr:134950K .......... .......... .......... .......... .......... 99% 41.5M 0s 2024-04-21T08:46:25.644 INFO:tasks.workunit.client.0.smithi057.stderr:135000K .......... .......... .......... .......... .......... 99% 36.9M 0s 2024-04-21T08:46:25.646 INFO:tasks.workunit.client.0.smithi057.stderr:135050K .......... .......... .......... .......... .......... 99% 28.2M 0s 2024-04-21T08:46:25.648 INFO:tasks.workunit.client.0.smithi057.stderr:135100K .......... .......... .......... .......... .......... 99% 22.4M 0s 2024-04-21T08:46:25.649 INFO:tasks.workunit.client.0.smithi057.stderr:135150K .......... .......... .......... .......... .......... 99% 46.4M 0s 2024-04-21T08:46:25.651 INFO:tasks.workunit.client.0.smithi057.stderr:135200K .......... .......... .......... .......... .......... 99% 20.1M 0s 2024-04-21T08:46:25.652 INFO:tasks.workunit.client.0.smithi057.stderr:135250K .......... .......... .......... .......... .......... 99% 52.6M 0s 2024-04-21T08:46:25.654 INFO:tasks.workunit.client.0.smithi057.stderr:135300K .......... .......... .......... .......... .......... 99% 26.7M 0s 2024-04-21T08:46:25.656 INFO:tasks.workunit.client.0.smithi057.stderr:135350K .......... .......... .......... .......... .......... 99% 26.3M 0s 2024-04-21T08:46:25.657 INFO:tasks.workunit.client.0.smithi057.stderr:135400K .......... .......... .......... .......... .......... 99% 28.2M 0s 2024-04-21T08:46:25.659 INFO:tasks.workunit.client.0.smithi057.stderr:135450K .......... .......... .......... .......... .......... 99% 26.0M 0s 2024-04-21T08:46:25.661 INFO:tasks.workunit.client.0.smithi057.stderr:135500K .......... .......... .......... .......... .......... 99% 23.6M 0s 2024-04-21T08:46:25.663 INFO:tasks.workunit.client.0.smithi057.stderr:135550K .......... .......... .......... .......... .......... 99% 31.0M 0s 2024-04-21T08:46:25.665 INFO:tasks.workunit.client.0.smithi057.stderr:135600K .......... .......... .......... .......... .......... 99% 23.9M 0s 2024-04-21T08:46:25.666 INFO:tasks.workunit.client.0.smithi057.stderr:135650K .......... .......... .......... .......... .......... 99% 38.1M 0s 2024-04-21T08:46:25.669 INFO:tasks.workunit.client.0.smithi057.stderr:135700K .......... .......... .......... .......... .......... 99% 31.8M 0s 2024-04-21T08:46:25.669 INFO:tasks.workunit.client.0.smithi057.stderr:135750K .. 100% 102M=4.7s 2024-04-21T08:46:25.669 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:25.669 INFO:tasks.workunit.client.0.smithi057.stderr:2024-04-21 08:46:25 (28.1 MB/s) - ‘linux.tar.xz’ saved [139010916/139010916] 2024-04-21T08:46:25.669 INFO:tasks.workunit.client.0.smithi057.stderr: 2024-04-21T08:46:25.669 INFO:tasks.workunit.client.0.smithi057.stderr:+ mkdir t 2024-04-21T08:46:25.671 INFO:tasks.workunit.client.0.smithi057.stderr:+ cd t 2024-04-21T08:46:25.671 INFO:tasks.workunit.client.0.smithi057.stderr:+ tar xJf ../linux.tar.xz 2024-04-21T08:46:26.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:26 smithi129 ceph-mon[31272]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T08:46:26.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:26 smithi057 ceph-mon[27968]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T08:46:26.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:26 smithi136 ceph-mon[31791]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 2.5 KiB/s wr, 3 op/s 2024-04-21T08:46:28.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:28 smithi129 ceph-mon[31272]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.1 KiB/s wr, 8 op/s 2024-04-21T08:46:28.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:28 smithi057 ceph-mon[27968]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.1 KiB/s wr, 8 op/s 2024-04-21T08:46:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:28 smithi136 ceph-mon[31791]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.1 KiB/s wr, 8 op/s 2024-04-21T08:46:30.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:30 smithi129 ceph-mon[31272]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.6 KiB/s wr, 5 op/s 2024-04-21T08:46:30.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:30 smithi057 ceph-mon[27968]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.6 KiB/s wr, 5 op/s 2024-04-21T08:46:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:30 smithi136 ceph-mon[31791]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.6 KiB/s wr, 5 op/s 2024-04-21T08:46:31.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:31.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:31 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:31 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:31.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:31 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:32.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:32 smithi129 ceph-mon[31272]: pgmap v36: 129 pgs: 129 active+clean; 58 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 4.7 MiB/s wr, 19 op/s 2024-04-21T08:46:32.328 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:32.328 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:32.329 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:32.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:32 smithi057 ceph-mon[27968]: pgmap v36: 129 pgs: 129 active+clean; 58 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 4.7 MiB/s wr, 19 op/s 2024-04-21T08:46:32.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:32.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:32.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:32 smithi136 ceph-mon[31791]: pgmap v36: 129 pgs: 129 active+clean; 58 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 4.7 MiB/s wr, 19 op/s 2024-04-21T08:46:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:46:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:46:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:46:32.648 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:32.649 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 -- 172.21.15.57:0/2532261997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538101a40 msgr2=0x7f353810dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 --2- 172.21.15.57:0/2532261997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538101a40 0x7f353810dca0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3530009ca0 tx=0x7f353002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 -- 172.21.15.57:0/2532261997 shutdown_connections 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 --2- 172.21.15.57:0/2532261997 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538101a40 0x7f353810dca0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 --2- 172.21.15.57:0/2532261997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f3538101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 --2- 172.21.15.57:0/2532261997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3538105dd0 0x7f35381061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 -- 172.21.15.57:0/2532261997 >> 172.21.15.57:0/2532261997 conn(0x7f3538077c40 msgr2=0x7f3538078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 -- 172.21.15.57:0/2532261997 shutdown_connections 2024-04-21T08:46:32.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.760+0000 7f353cbd5640 1 -- 172.21.15.57:0/2532261997 wait complete. 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 Processor -- start 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 -- start start 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f35381103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3538101a40 0x7f353810e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538105dd0 0x7f353810edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f35381122d0 con 0x7f3538105dd0 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3538112150 con 0x7f35381010a0 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353cbd5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3538112450 con 0x7f3538101a40 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538105dd0 0x7f353810edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f35381103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f35381103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57612/0 (socket says 172.21.15.57:57612) 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.761+0000 7f353659c640 1 -- 172.21.15.57:0/186340212 learned_addr learned my addr 172.21.15.57:0/186340212 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353659c640 1 -- 172.21.15.57:0/186340212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3538101a40 msgr2=0x7f353810e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353659c640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3538101a40 0x7f353810e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353659c640 1 -- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538105dd0 msgr2=0x7f353810edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353659c640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538105dd0 0x7f353810edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353659c640 1 -- 172.21.15.57:0/186340212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35381ab0f0 con 0x7f35381010a0 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f3536d9d640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538105dd0 0x7f353810edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353659c640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f35381103b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f352800d770 tx=0x7f352800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f352ffff640 1 -- 172.21.15.57:0/186340212 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528017070 con 0x7f35381010a0 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35381ab320 con 0x7f35381010a0 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35381ab840 con 0x7f35381010a0 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f352ffff640 1 -- 172.21.15.57:0/186340212 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35280054d0 con 0x7f35381010a0 2024-04-21T08:46:32.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.762+0000 7f352ffff640 1 -- 172.21.15.57:0/186340212 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528004c30 con 0x7f35381010a0 2024-04-21T08:46:32.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f352ffff640 1 -- 172.21.15.57:0/186340212 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f3528004df0 con 0x7f35381010a0 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 Processor -- start 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 -- start start 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 0x55fa7e887960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887ea0 0x55fa7e890540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e892e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fa7e7ec1e0 con 0x55fa7e887580 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fa7e789390 con 0x55fa7e887ea0 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f353f7c7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fa7e7e81a0 con 0x55fa7e890a80 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.765+0000 7f352ffff640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f35240a3e40 0x7f35240a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 0x55fa7e887960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f353659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887ea0 0x55fa7e890540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 0x55fa7e887960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36692/0 (socket says 172.21.15.57:36692) 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 -- 172.21.15.57:0/2892438683 learned_addr learned my addr 172.21.15.57:0/2892438683 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f35240a3e40 0x7f35240a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f352ffff640 1 -- 172.21.15.57:0/186340212 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f35280d0900 con 0x7f35381010a0 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f352dd9b640 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e892e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 -- 172.21.15.57:0/2892438683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 msgr2=0x55fa7e892e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e892e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 -- 172.21.15.57:0/2892438683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887ea0 msgr2=0x55fa7e890540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887ea0 0x55fa7e890540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 -- 172.21.15.57:0/2892438683 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa7e5c87d0 con 0x55fa7e887580 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f352dd9b640 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e892e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f353659c640 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887ea0 0x55fa7e890540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 0x55fa7e887960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3530009430 tx=0x7f35300327f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aef389a1357ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f3536d9d640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f35240a3e40 0x7f35240a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3530006010 tx=0x7f3530042000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f352cd99640 1 -- 172.21.15.57:0/2892438683 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530051070 con 0x55fa7e887580 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f352cd99640 1 -- 172.21.15.57:0/2892438683 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3530004d40 con 0x55fa7e887580 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.766+0000 7f352cd99640 1 -- 172.21.15.57:0/2892438683 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530039040 con 0x55fa7e887580 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 -- 172.21.15.57:0/2892438683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 msgr2=0x55fa7e887960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 0x55fa7e887960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3530009430 tx=0x7f35300327f0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 -- 172.21.15.57:0/2892438683 shutdown_connections 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e892e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887ea0 0x55fa7e890540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2892438683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887580 0x55fa7e887960 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 -- 172.21.15.57:0/2892438683 >> 172.21.15.57:0/2892438683 conn(0x55fa7e881cb0 msgr2=0x55fa7e882090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 -- 172.21.15.57:0/2892438683 shutdown_connections 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.767+0000 7f353f7c7740 1 -- 172.21.15.57:0/2892438683 wait complete. 2024-04-21T08:46:32.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 -- start start 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 Processor -- start 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e88dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e88e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fa7e893810 con 0x55fa7e887ea0 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fa7e7a52f0 con 0x55fa7e887580 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fa7e893990 con 0x55fa7e890a80 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353f7c7740 10 client.0 _ll_register_callbacks cb 0x55fa7e7b2580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36704/0 (socket says 172.21.15.57:36704) 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e88dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 -- 172.21.15.57:0/3095682199 learned_addr learned my addr 172.21.15.57:0/3095682199 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:32.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f3536d9d640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e88e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 -- 172.21.15.57:0/3095682199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 msgr2=0x55fa7e88e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e88e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 -- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 msgr2=0x55fa7e88dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e88dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f352dd9b640 1 -- 172.21.15.57:0/3095682199 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa7e4fe3f0 con 0x55fa7e887ea0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f3536d9d640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e88e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.768+0000 7f353659c640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e88dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f352dd9b640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e010 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3520010650 tx=0x7f3520010b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2723aad690bbd0f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f351dffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f351dffb640 1 -- 172.21.15.57:0/3095682199 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520004250 con 0x55fa7e887ea0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f351dffb640 1 -- 172.21.15.57:0/3095682199 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f35200043f0 con 0x55fa7e887ea0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3530004d40 con 0x55fa7e887ea0 2024-04-21T08:46:32.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f351dffb640 1 -- 172.21.15.57:0/3095682199 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3520010e50 con 0x55fa7e887ea0 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 10 client.15936 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fa7e4be0f0 con 0x55fa7e887ea0 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f351dffb640 1 -- 172.21.15.57:0/3095682199 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3530004d40 con 0x55fa7e887ea0 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f351dffb640 1 -- 172.21.15.57:0/3095682199 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f352000f040 con 0x55fa7e887ea0 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 10 client.15936 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 20 client.15936 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e8a0a50 0x55fa7e8a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 4 client.15936 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.769+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa7e7f9c00 con 0x55fa7e8a0a50 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.770+0000 7f353659c640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e8a0a50 0x55fa7e8a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.770+0000 7f353659c640 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e8a0a50 0x55fa7e8a2e90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14baa2e87f50679f server_cookie=6e6d84e976ada183 in_seq=0 out_seq=0 2024-04-21T08:46:32.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.770+0000 7f351dffb640 10 client.15936 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.771+0000 7f351dffb640 1 -- 172.21.15.57:0/3095682199 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa7e7f9c00 con 0x55fa7e8a0a50 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.771+0000 7f351dffb640 10 client.15936 handle_command_reply: tid=0 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.771+0000 7f353f7c7740 1 client.15936 shutdown 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 msgr2=0x55fa7e88e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e010 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3520010650 tx=0x7f3520010b10 comp rx=0 tx=0).stop 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 shutdown_connections 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e8a0a50 0x55fa7e8a2e90 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e890a80 0x55fa7e88e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e010 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f351dffb640 0 client.15936 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 --2- 172.21.15.57:0/3095682199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e88dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 >> 172.21.15.57:0/3095682199 conn(0x55fa7e881cb0 msgr2=0x55fa7e883a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 shutdown_connections 2024-04-21T08:46:32.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 1 -- 172.21.15.57:0/3095682199 wait complete. 2024-04-21T08:46:32.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.772+0000 7f353f7c7740 20 client.15936 trim_cache size 0 max 16384 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 Processor -- start 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 -- start start 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887580 0x55fa7e89c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e890a80 0x55fa7e880bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b3880 con 0x55fa7e887ea0 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b60e0 con 0x55fa7e890a80 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353f7c7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b3ed0 con 0x55fa7e887580 2024-04-21T08:46:32.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36716/0 (socket says 172.21.15.57:36716) 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 -- 172.21.15.57:0/2886178067 learned_addr learned my addr 172.21.15.57:0/2886178067 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f3536d9d640 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e890a80 0x55fa7e880bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f353659c640 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887580 0x55fa7e89c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 -- 172.21.15.57:0/2886178067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887580 msgr2=0x55fa7e89c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887580 0x55fa7e89c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 -- 172.21.15.57:0/2886178067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e890a80 msgr2=0x55fa7e880bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.790+0000 7f352dd9b640 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e890a80 0x55fa7e880bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f352dd9b640 1 -- 172.21.15.57:0/2886178067 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa7e887cd0 con 0x55fa7e887ea0 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f352dd9b640 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3520003ec0 tx=0x7f352000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe989c9d3045b551 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f351ffff640 1 -- 172.21.15.57:0/2886178067 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352001a450 con 0x55fa7e887ea0 2024-04-21T08:46:32.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f351ffff640 1 -- 172.21.15.57:0/2886178067 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3520003aa0 con 0x55fa7e887ea0 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f351ffff640 1 -- 172.21.15.57:0/2886178067 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35200041a0 con 0x55fa7e887ea0 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f353f7c7740 1 -- 172.21.15.57:0/2886178067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 msgr2=0x55fa7e88e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e370 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3520003ec0 tx=0x7f352000e6c0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.791+0000 7f353f7c7740 1 -- 172.21.15.57:0/2886178067 shutdown_connections 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e890a80 0x55fa7e880bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e887ea0 0x55fa7e88e370 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 --2- 172.21.15.57:0/2886178067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887580 0x55fa7e89c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- 172.21.15.57:0/2886178067 >> 172.21.15.57:0/2886178067 conn(0x55fa7e881cb0 msgr2=0x55fa7e8804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- 172.21.15.57:0/2886178067 shutdown_connections 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- 172.21.15.57:0/2886178067 wait complete. 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- start start 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 Processor -- start 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e898be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887ea0 0x55fa7e899120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 0x55fa7e88a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b45a0 con 0x55fa7e890a80 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b5a10 con 0x55fa7e887580 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f35381b4c70 con 0x55fa7e887ea0 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353f7c7740 10 client.0 _ll_register_callbacks cb 0x7f3520009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 0x55fa7e88a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f3536d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 0x55fa7e88a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36718/0 (socket says 172.21.15.57:36718) 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f3536d9d640 1 -- 172.21.15.57:0/1442507672 learned_addr learned my addr 172.21.15.57:0/1442507672 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f353659c640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e898be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f3536d9d640 1 -- 172.21.15.57:0/1442507672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887ea0 msgr2=0x55fa7e899120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.792+0000 7f352dd9b640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887ea0 0x55fa7e899120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f3536d9d640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887ea0 0x55fa7e899120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f3536d9d640 1 -- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 msgr2=0x55fa7e898be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f3536d9d640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e898be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f3536d9d640 1 -- 172.21.15.57:0/1442507672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa7e88a9a0 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353659c640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e898be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f3536d9d640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 0x55fa7e88a170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3530009430 tx=0x7f3530004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9603a1fb3a4a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f351dffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f351dffb640 1 -- 172.21.15.57:0/1442507672 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530039040 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f351dffb640 1 -- 172.21.15.57:0/1442507672 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f3530004770 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3520003aa0 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f351dffb640 1 -- 172.21.15.57:0/1442507672 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3530051040 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 10 client.15942 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fa7e906ae0 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f351dffb640 1 -- 172.21.15.57:0/1442507672 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3520003aa0 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f351dffb640 1 -- 172.21.15.57:0/1442507672 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f353004b070 con 0x55fa7e890a80 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 10 client.15942 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:32.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 10 client.15942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 20 client.15942 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e906f30 0x55fa7e909320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 4 client.15942 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.793+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa7e7f3a70 con 0x55fa7e906f30 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.794+0000 7f353659c640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e906f30 0x55fa7e909320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.794+0000 7f351dffb640 10 client.15942 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:32.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.794+0000 7f353659c640 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e906f30 0x55fa7e909320 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72899fd5889431f2 server_cookie=c848d569aa460aed in_seq=0 out_seq=0 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.795+0000 7f351dffb640 1 -- 172.21.15.57:0/1442507672 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fa7e7f3a70 con 0x55fa7e906f30 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.795+0000 7f351dffb640 10 client.15942 handle_command_reply: tid=0 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.795+0000 7f353f7c7740 1 client.15942 shutdown 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.795+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 msgr2=0x55fa7e88a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.795+0000 7f353f7c7740 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 0x55fa7e88a170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3530009430 tx=0x7f3530004400 comp rx=0 tx=0).stop 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 shutdown_connections 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7e906f30 0x55fa7e909320 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa7e890a80 0x55fa7e88a170 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa7e887ea0 0x55fa7e899120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f351dffb640 0 client.15942 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 --2- 172.21.15.57:0/1442507672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa7e887580 0x55fa7e898be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 >> 172.21.15.57:0/1442507672 conn(0x55fa7e881cb0 msgr2=0x55fa7e883820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 shutdown_connections 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 1 -- 172.21.15.57:0/1442507672 wait complete. 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353f7c7740 20 client.15942 trim_cache size 0 max 16384 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:46:32.802 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:46:32.803 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:46:32.803 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:46:32.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f35240a3e40 msgr2=0x7f35240a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f35240a3e40 0x7f35240a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3530006010 tx=0x7f3530042000 comp rx=0 tx=0).stop 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 msgr2=0x7f35381103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f35381103b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f352800d770 tx=0x7f352800dc30 comp rx=0 tx=0).stop 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 shutdown_connections 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f35240a3e40 0x7f35240a6300 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3538105dd0 0x7f353810edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3538101a40 0x7f353810e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 --2- 172.21.15.57:0/186340212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35381010a0 0x7f35381103b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 >> 172.21.15.57:0/186340212 conn(0x7f3538077c40 msgr2=0x7f35380fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 shutdown_connections 2024-04-21T08:46:32.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.796+0000 7f353cbd5640 1 -- 172.21.15.57:0/186340212 wait complete. 2024-04-21T08:46:32.806 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:46:32.807 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 -- 172.21.15.57:0/1928266271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 msgr2=0x7fdd5410bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1928266271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5410bfd0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009900 tx=0x7fdd3c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 -- 172.21.15.57:0/1928266271 shutdown_connections 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1928266271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd5410c5a0 0x7fdd5410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1928266271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5410bfd0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1928266271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 0x7fdd541035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 -- 172.21.15.57:0/1928266271 >> 172.21.15.57:0/1928266271 conn(0x7fdd540771d0 msgr2=0x7fdd540775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 -- 172.21.15.57:0/1928266271 shutdown_connections 2024-04-21T08:46:32.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.915+0000 7fdd58d23640 1 -- 172.21.15.57:0/1928266271 wait complete. 2024-04-21T08:46:32.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 Processor -- start 2024-04-21T08:46:32.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 -- start start 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 0x7fdd5419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd5410c5a0 0x7fdd541985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdd54112240 con 0x7fdd54103af0 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541120c0 con 0x7fdd541031d0 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.916+0000 7fdd58d23640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdd541123c0 con 0x7fdd5410c5a0 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5419e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36738/0 (socket says 172.21.15.57:36738) 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 -- 172.21.15.57:0/1380167175 learned_addr learned my addr 172.21.15.57:0/1380167175 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd52d9d640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd5410c5a0 0x7fdd541985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd5259c640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 0x7fdd5419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 -- 172.21.15.57:0/1380167175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd5410c5a0 msgr2=0x7fdd541985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd5410c5a0 0x7fdd541985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 -- 172.21.15.57:0/1380167175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 msgr2=0x7fdd5419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 0x7fdd5419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 -- 172.21.15.57:0/1380167175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd54198b80 con 0x7fdd54103af0 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd5259c640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 0x7fdd5419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.917+0000 7fdd4bfff640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5419e950 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c005940 tx=0x7fdd3c02d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.918+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c043020 con 0x7fdd54103af0 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.918+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd3c02dc90 con 0x7fdd54103af0 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.918+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd54198e10 con 0x7fdd54103af0 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.918+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c003ab0 con 0x7fdd54103af0 2024-04-21T08:46:32.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.918+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd54199360 con 0x7fdd54103af0 2024-04-21T08:46:32.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.919+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd54069230 con 0x7fdd54103af0 2024-04-21T08:46:32.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.920+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fdd3c046020 con 0x7fdd54103af0 2024-04-21T08:46:32.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.921+0000 7fdd4b7fe640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdd400a3e40 0x7fdd400a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:32.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.921+0000 7fdd5259c640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdd400a3e40 0x7fdd400a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:32.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.921+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fdd3c0f2f10 con 0x7fdd54103af0 2024-04-21T08:46:32.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.921+0000 7fdd5259c640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdd400a3e40 0x7fdd400a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd44005d00 tx=0x7fdd44009550 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:32.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:32.925+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdd3c04d020 con 0x7fdd54103af0 2024-04-21T08:46:33.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.164+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd54107690 con 0x7fdd54103af0 2024-04-21T08:46:33.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.164+0000 7fdd4b7fe640 1 -- 172.21.15.57:0/1380167175 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fdd3c0bf110 con 0x7fdd54103af0 2024-04-21T08:46:33.164 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:46:33.164 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:46:33.164 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:46:33.166 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdd400a3e40 msgr2=0x7fdd400a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdd400a3e40 0x7fdd400a6300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd44005d00 tx=0x7fdd44009550 comp rx=0 tx=0).stop 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 msgr2=0x7fdd5419e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5419e950 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c005940 tx=0x7fdd3c02d890 comp rx=0 tx=0).stop 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 shutdown_connections 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fdd400a3e40 0x7fdd400a6300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd5410c5a0 0x7fdd541985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd54103af0 0x7fdd5419e950 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 --2- 172.21.15.57:0/1380167175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd541031d0 0x7fdd5419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 >> 172.21.15.57:0/1380167175 conn(0x7fdd540771d0 msgr2=0x7fdd540fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 shutdown_connections 2024-04-21T08:46:33.167 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.167+0000 7fdd58d23640 1 -- 172.21.15.57:0/1380167175 wait complete. 2024-04-21T08:46:33.178 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:46:33.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 -- 172.21.15.57:0/758476350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1870103050 msgr2=0x7f1870103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 --2- 172.21.15.57:0/758476350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1870103050 0x7f1870103430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f186c009900 tx=0x7f186c02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 -- 172.21.15.57:0/758476350 shutdown_connections 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 --2- 172.21.15.57:0/758476350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187010c570 0x7f187010e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 --2- 172.21.15.57:0/758476350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1870103970 0x7f187010c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 --2- 172.21.15.57:0/758476350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1870103050 0x7f1870103430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 -- 172.21.15.57:0/758476350 >> 172.21.15.57:0/758476350 conn(0x7f1870077fb0 msgr2=0x7f18700783c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 -- 172.21.15.57:0/758476350 shutdown_connections 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 -- 172.21.15.57:0/758476350 wait complete. 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 Processor -- start 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.285+0000 7f1877293640 1 -- start start 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1877293640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1870103050 0x7f187019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1877293640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1870103970 0x7f187019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1877293640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 0x7f18701985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1877293640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1870112260 con 0x7f187010c570 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1877293640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18701120e0 con 0x7f1870103970 2024-04-21T08:46:33.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1877293640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18701123e0 con 0x7f1870103050 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 0x7f18701985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 0x7f18701985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36756/0 (socket says 172.21.15.57:36756) 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 -- 172.21.15.57:0/1041492975 learned_addr learned my addr 172.21.15.57:0/1041492975 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f187482e640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1870103970 0x7f187019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f187502f640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1870103050 0x7f187019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 -- 172.21.15.57:0/1041492975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1870103050 msgr2=0x7f187019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1870103050 0x7f187019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 -- 172.21.15.57:0/1041492975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1870103970 msgr2=0x7f187019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1870103970 0x7f187019e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 -- 172.21.15.57:0/1041492975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1870198d40 con 0x7f187010c570 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f1875830640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 0x7f18701985c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f186800ea60 tx=0x7f186800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.286+0000 7f18667fc640 1 -- 172.21.15.57:0/1041492975 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186800cd80 con 0x7f187010c570 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.287+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1870199030 con 0x7f187010c570 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.287+0000 7f18667fc640 1 -- 172.21.15.57:0/1041492975 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1868005020 con 0x7f187010c570 2024-04-21T08:46:33.288 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.287+0000 7f18667fc640 1 -- 172.21.15.57:0/1041492975 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868010690 con 0x7f187010c570 2024-04-21T08:46:33.289 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.287+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18701a52d0 con 0x7f187010c570 2024-04-21T08:46:33.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f18667fc640 1 -- 172.21.15.57:0/1041492975 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f18680054d0 con 0x7f187010c570 2024-04-21T08:46:33.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f1879f66740 1 Processor -- start 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f1879f66740 1 -- start start 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f18667fc640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f185c0a3e40 0x7f185c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f1879f66740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x558063537790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f1879f66740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063537d60 0x558063540890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.289+0000 7f1879f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x5580635412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f1879f66740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55806349c1f0 con 0x5580635373b0 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f1879f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5580634394b0 con 0x558063540e60 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f1879f66740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5580634981b0 con 0x558063537d60 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187482e640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f185c0a3e40 0x7f185c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x558063537790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f18667fc640 1 -- 172.21.15.57:0/1041492975 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f18680d12d0 con 0x7f187010c570 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x558063537790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36762/0 (socket says 172.21.15.57:36762) 2024-04-21T08:46:33.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 -- 172.21.15.57:0/2196522915 learned_addr learned my addr 172.21.15.57:0/2196522915 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f1875830640 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x5580635412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187482e640 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063537d60 0x558063540890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 -- 172.21.15.57:0/2196522915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063537d60 msgr2=0x558063540890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063537d60 0x558063540890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 -- 172.21.15.57:0/2196522915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 msgr2=0x5580635412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x5580635412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 -- 172.21.15.57:0/2196522915 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55806342b1b0 con 0x5580635373b0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f1875830640 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x5580635412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187482e640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f185c0a3e40 0x7f185c0a6300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f187010e3d0 tx=0x7f18600156d0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f187502f640 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x558063537790 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f186c006010 tx=0x7f186c02c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393a1f0f48b2a6fb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f185b7fe640 1 -- 172.21.15.57:0/2196522915 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c004140 con 0x5580635373b0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f185b7fe640 1 -- 172.21.15.57:0/2196522915 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c02cd30 con 0x5580635373b0 2024-04-21T08:46:33.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.290+0000 7f185b7fe640 1 -- 172.21.15.57:0/2196522915 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c0046c0 con 0x5580635373b0 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 -- 172.21.15.57:0/2196522915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 msgr2=0x558063537790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x558063537790 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f186c006010 tx=0x7f186c02c710 comp rx=0 tx=0).stop 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 -- 172.21.15.57:0/2196522915 shutdown_connections 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x5580635412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063537d60 0x558063540890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 --2- 172.21.15.57:0/2196522915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x558063537790 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 -- 172.21.15.57:0/2196522915 >> 172.21.15.57:0/2196522915 conn(0x558063531ba0 msgr2=0x558063531f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 -- 172.21.15.57:0/2196522915 shutdown_connections 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.291+0000 7f1879f66740 1 -- 172.21.15.57:0/2196522915 wait complete. 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 -- start start 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 Processor -- start 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x5580635471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 0x558063544580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063540e60 0x558063544ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x558063547d40 con 0x558063537d60 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558063455500 con 0x5580635373b0 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x558063547ec0 con 0x558063540e60 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f1879f66740 10 client.0 _ll_register_callbacks cb 0x558063423b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 0x558063544580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 0x558063544580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36772/0 (socket says 172.21.15.57:36772) 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 -- 172.21.15.57:0/1257759795 learned_addr learned my addr 172.21.15.57:0/1257759795 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187502f640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x5580635471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 -- 172.21.15.57:0/1257759795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063540e60 msgr2=0x558063544ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063540e60 0x558063544ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 -- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 msgr2=0x5580635471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.292+0000 7f187482e640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x5580635471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f187482e640 1 -- 172.21.15.57:0/1257759795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580631ae3f0 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f187502f640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x5580635471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f187482e640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 0x558063544580 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f18600189b0 tx=0x7f18600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a34c938163c3be8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1858ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1858ff9640 1 -- 172.21.15.57:0/1257759795 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860017b70 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1858ff9640 1 -- 172.21.15.57:0/1257759795 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1860017d10 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f186c02cd30 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1858ff9640 1 -- 172.21.15.57:0/1257759795 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1860032050 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1879f66740 10 client.15960 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5580632787d0 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1858ff9640 1 -- 172.21.15.57:0/1257759795 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c02cd30 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1858ff9640 1 -- 172.21.15.57:0/1257759795 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f186002d3b0 con 0x558063537d60 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1879f66740 10 client.15960 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.293+0000 7f1879f66740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:33.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f1879f66740 20 client.15960 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f1879f66740 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558063554f70 0x558063557410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f1879f66740 4 client.15960 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55806343b930 con 0x558063554f70 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f187502f640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558063554f70 0x558063557410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f1858ff9640 10 client.15960 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.294+0000 7f187502f640 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558063554f70 0x558063557410 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c71936eb75c026 server_cookie=64695c02708b8f77 in_seq=0 out_seq=0 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1858ff9640 1 -- 172.21.15.57:0/1257759795 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55806343b930 con 0x558063554f70 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1858ff9640 10 client.15960 handle_command_reply: tid=0 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 client.15960 shutdown 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 msgr2=0x558063544580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 0x558063544580 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f18600189b0 tx=0x7f18600046c0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 shutdown_connections 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558063554f70 0x558063557410 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063540e60 0x558063544ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063537d60 0x558063544580 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1858ff9640 0 client.15960 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 --2- 172.21.15.57:0/1257759795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x5580635471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 >> 172.21.15.57:0/1257759795 conn(0x558063531ba0 msgr2=0x5580635338f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 shutdown_connections 2024-04-21T08:46:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 1 -- 172.21.15.57:0/1257759795 wait complete. 2024-04-21T08:46:33.305 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.295+0000 7f1879f66740 20 client.15960 trim_cache size 0 max 16384 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 Processor -- start 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 -- start start 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x558063550c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 0x55806353ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x5580635470f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f18701ab030 con 0x558063540e60 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18701ad890 con 0x5580635373b0 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1879f66740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18701ab680 con 0x558063546cb0 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 0x55806353ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 0x55806353ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36774/0 (socket says 172.21.15.57:36774) 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 -- 172.21.15.57:0/1741295952 learned_addr learned my addr 172.21.15.57:0/1741295952 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:33.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187502f640 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x558063550c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f1875830640 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x5580635470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 -- 172.21.15.57:0/1741295952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 msgr2=0x5580635470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x5580635470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 -- 172.21.15.57:0/1741295952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 msgr2=0x558063550c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x558063550c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.309+0000 7f187482e640 1 -- 172.21.15.57:0/1741295952 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558063536560 con 0x558063540e60 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f187482e640 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 0x55806353ef00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f186002d7d0 tx=0x7f1860015c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c2af875e1c2f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f185affd640 1 -- 172.21.15.57:0/1741295952 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186003b070 con 0x558063540e60 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f185affd640 1 -- 172.21.15.57:0/1741295952 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1860029070 con 0x558063540e60 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f185affd640 1 -- 172.21.15.57:0/1741295952 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18600059c0 con 0x558063540e60 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 -- 172.21.15.57:0/1741295952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 msgr2=0x55806353ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 0x55806353ef00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f186002d7d0 tx=0x7f1860015c20 comp rx=0 tx=0).stop 2024-04-21T08:46:33.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 -- 172.21.15.57:0/1741295952 shutdown_connections 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x5580635470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x558063540e60 0x55806353ef00 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 --2- 172.21.15.57:0/1741295952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5580635373b0 0x558063550c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 -- 172.21.15.57:0/1741295952 >> 172.21.15.57:0/1741295952 conn(0x558063531ba0 msgr2=0x55806353e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 -- 172.21.15.57:0/1741295952 shutdown_connections 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.310+0000 7f1879f66740 1 -- 172.21.15.57:0/1741295952 wait complete. 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 -- start start 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 Processor -- start 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x55806354d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x55806355c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x55806355c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f18701abd50 con 0x5580635373b0 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18701ad1c0 con 0x558063540e60 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f18701ac420 con 0x558063546cb0 2024-04-21T08:46:33.315 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1879f66740 10 client.0 _ll_register_callbacks cb 0x7f18600186b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f187502f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x55806354d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f187482e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x55806355c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f187502f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x55806354d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36782/0 (socket says 172.21.15.57:36782) 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f187502f640 1 -- 172.21.15.57:0/2014466650 learned_addr learned my addr 172.21.15.57:0/2014466650 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f1875830640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x55806355c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.311+0000 7f187502f640 1 -- 172.21.15.57:0/2014466650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 msgr2=0x55806355c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f187502f640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x55806355c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f187502f640 1 -- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 msgr2=0x55806355c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f187502f640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x55806355c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f187502f640 1 -- 172.21.15.57:0/2014466650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55806355ce90 con 0x5580635373b0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f187482e640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x55806355c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f187502f640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x55806354d5e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f186c0096c0 tx=0x7f186c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4bb1c035da017e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1858ff9640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:33.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1858ff9640 1 -- 172.21.15.57:0/2014466650 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c0044d0 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1860029070 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1858ff9640 1 -- 172.21.15.57:0/2014466650 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f186c02c9c0 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1858ff9640 1 -- 172.21.15.57:0/2014466650 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f186c02cbc0 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 10 client.15966 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5580635bb5f0 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1858ff9640 1 -- 172.21.15.57:0/2014466650 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1860029070 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1858ff9640 1 -- 172.21.15.57:0/2014466650 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f186c054080 con 0x5580635373b0 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 10 client.15966 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 20 client.15966 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5580635bb930 0x5580635bdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 4 client.15966 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.312+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558063544e50 con 0x5580635bb930 2024-04-21T08:46:33.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.313+0000 7f187482e640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5580635bb930 0x5580635bdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:33.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.313+0000 7f187482e640 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5580635bb930 0x5580635bdd20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=729ba653471c8c71 server_cookie=da947773a80e543b in_seq=0 out_seq=0 2024-04-21T08:46:33.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.313+0000 7f1858ff9640 10 client.15966 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1858ff9640 1 -- 172.21.15.57:0/2014466650 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558063544e50 con 0x5580635bb930 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1858ff9640 10 client.15966 handle_command_reply: tid=0 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 client.15966 shutdown 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 msgr2=0x55806354d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x55806354d5e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f186c0096c0 tx=0x7f186c009260 comp rx=0 tx=0).stop 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 shutdown_connections 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5580635bb930 0x5580635bdd20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x558063546cb0 0x55806355c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1858ff9640 0 client.15966 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558063540e60 0x55806355c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 --2- 172.21.15.57:0/2014466650 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5580635373b0 0x55806354d5e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 >> 172.21.15.57:0/2014466650 conn(0x558063531ba0 msgr2=0x5580635432e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 shutdown_connections 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 1 -- 172.21.15.57:0/2014466650 wait complete. 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.314+0000 7f1879f66740 20 client.15966 trim_cache size 0 max 16384 2024-04-21T08:46:33.320 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f185c0a3e40 msgr2=0x7f185c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f185c0a3e40 0x7f185c0a6300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f187010e3d0 tx=0x7f18600156d0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 msgr2=0x7f18701985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 0x7f18701985c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f186800ea60 tx=0x7f186800ee60 comp rx=0 tx=0).stop 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 shutdown_connections 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f185c0a3e40 0x7f185c0a6300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f187010c570 0x7f18701985c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1870103970 0x7f187019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 --2- 172.21.15.57:0/1041492975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1870103050 0x7f187019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:33.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 >> 172.21.15.57:0/1041492975 conn(0x7f1870077fb0 msgr2=0x7f18700fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:33.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 shutdown_connections 2024-04-21T08:46:33.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:33.315+0000 7f1877293640 1 -- 172.21.15.57:0/1041492975 wait complete. 2024-04-21T08:46:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:34 smithi129 ceph-mon[31272]: pgmap v37: 129 pgs: 129 active+clean; 139 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 11 MiB/s wr, 32 op/s 2024-04-21T08:46:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:34 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1380167175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:34.326 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:46:34.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:46:34.326 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:34 smithi057 ceph-mon[27968]: pgmap v37: 129 pgs: 129 active+clean; 139 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 11 MiB/s wr, 32 op/s 2024-04-21T08:46:34.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:34 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1380167175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:34 smithi136 ceph-mon[31791]: pgmap v37: 129 pgs: 129 active+clean; 139 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 11 MiB/s wr, 32 op/s 2024-04-21T08:46:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:34 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1380167175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:46:34.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2875246967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe8810e900 msgr2=0x7fbe88110cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2875246967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe8810e900 0x7fbe88110cf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009ca0 tx=0x7fbe8002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2875246967 shutdown_connections 2024-04-21T08:46:34.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2875246967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe8810e900 0x7fbe88110cf0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2875246967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe88101670 0x7fbe8810e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2875246967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe88100d50 0x7fbe88101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.436+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2875246967 >> 172.21.15.57:0/2875246967 conn(0x7fbe88077f50 msgr2=0x7fbe88078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2875246967 shutdown_connections 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2875246967 wait complete. 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 Processor -- start 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 -- start start 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe88100d50 0x7fbe881a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 0x7fbe881a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.437+0000 7fbe8dad2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8810e900 0x7fbe8819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe8dad2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881143c0 con 0x7fbe88101670 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe8dad2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88114240 con 0x7fbe88100d50 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe8dad2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88114540 con 0x7fbe8810e900 2024-04-21T08:46:34.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 0x7fbe881a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 0x7fbe881a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36802/0 (socket says 172.21.15.57:36802) 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 -- 172.21.15.57:0/2817181932 learned_addr learned my addr 172.21.15.57:0/2817181932 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe88100d50 0x7fbe881a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8810e900 0x7fbe8819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 -- 172.21.15.57:0/2817181932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8810e900 msgr2=0x7fbe8819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8810e900 0x7fbe8819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 -- 172.21.15.57:0/2817181932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe88100d50 msgr2=0x7fbe881a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe88100d50 0x7fbe881a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 -- 172.21.15.57:0/2817181932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8819b000 con 0x7fbe88101670 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 0x7fbe881a0bc0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c00e9f0 tx=0x7fbe7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe84ff9640 1 -- 172.21.15.57:0/2817181932 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c00cd80 con 0x7fbe88101670 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe84ff9640 1 -- 172.21.15.57:0/2817181932 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe7c0108e0 con 0x7fbe88101670 2024-04-21T08:46:34.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe84ff9640 1 -- 172.21.15.57:0/2817181932 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c018980 con 0x7fbe88101670 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8819b2f0 con 0x7fbe88101670 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.438+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe881a7600 con 0x7fbe88101670 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 Processor -- start 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe84ff9640 1 -- 172.21.15.57:0/2817181932 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fbe7c010430 con 0x7fbe88101670 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 -- start start 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c83568e0 0x55c8c8356cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 0x55c8c835f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c835fe10 0x55c8c8362260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55c8c82baf00 con 0x55c8c8357290 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c8c8258590 con 0x55c8c83568e0 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe9079f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55c8c82b6ec0 con 0x55c8c835fe10 2024-04-21T08:46:34.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 0x55c8c835f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 0x55c8c835f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36818/0 (socket says 172.21.15.57:36818) 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe877fe640 1 -- 172.21.15.57:0/2983526966 learned_addr learned my addr 172.21.15.57:0/2983526966 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe87fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c83568e0 0x55c8c8356cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c835fe10 0x55c8c8362260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe877fe640 1 -- 172.21.15.57:0/2983526966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c835fe10 msgr2=0x55c8c8362260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.441+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c835fe10 0x55c8c8362260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe877fe640 1 -- 172.21.15.57:0/2983526966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c83568e0 msgr2=0x55c8c8356cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c83568e0 0x55c8c8356cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe877fe640 1 -- 172.21.15.57:0/2983526966 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8c80957e0 con 0x55c8c8357290 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c835fe10 0x55c8c8362260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe84ff9640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbe700a3ec0 0x7fbe700a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 0x55c8c835f840 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b760 tx=0x7fbe7800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948a604f3f4ae964 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbe700a3ec0 0x7fbe700a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe84ff9640 1 -- 172.21.15.57:0/2817181932 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbe7c0d1090 con 0x7fbe88101670 2024-04-21T08:46:34.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe6dffb640 1 -- 172.21.15.57:0/2983526966 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78003dd0 con 0x55c8c8357290 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe6dffb640 1 -- 172.21.15.57:0/2983526966 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe78015070 con 0x55c8c8357290 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.442+0000 7fbe6dffb640 1 -- 172.21.15.57:0/2983526966 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78004d90 con 0x55c8c8357290 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 -- 172.21.15.57:0/2983526966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 msgr2=0x55c8c835f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 0x55c8c835f840 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b760 tx=0x7fbe7800bc20 comp rx=0 tx=0).stop 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbe700a3ec0 0x7fbe700a6380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe80006010 tx=0x7fbe80005d80 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 -- 172.21.15.57:0/2983526966 shutdown_connections 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c835fe10 0x55c8c8362260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c8357290 0x55c8c835f840 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2983526966 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c83568e0 0x55c8c8356cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 -- 172.21.15.57:0/2983526966 >> 172.21.15.57:0/2983526966 conn(0x55c8c8350fb0 msgr2=0x55c8c8351390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 -- 172.21.15.57:0/2983526966 shutdown_connections 2024-04-21T08:46:34.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 -- 172.21.15.57:0/2983526966 wait complete. 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 -- start start 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.443+0000 7fbe9079f740 1 Processor -- start 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c835d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c835d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55c8c8362d80 con 0x55c8c83568e0 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c8c8274580 con 0x55c8c835fe10 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55c8c8362f00 con 0x55c8c8357290 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 10 client.0 _ll_register_callbacks cb 0x55c8c827a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36826/0 (socket says 172.21.15.57:36826) 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 -- 172.21.15.57:0/2957422939 learned_addr learned my addr 172.21.15.57:0/2957422939 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c835d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c835d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 -- 172.21.15.57:0/2957422939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 msgr2=0x55c8c835d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c835d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 -- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 msgr2=0x55c8c835d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c835d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 -- 172.21.15.57:0/2957422939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8c80f5f70 con 0x55c8c83568e0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c835d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835ce20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbe78018000 tx=0x7fbe78003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc441343c1c3d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe66ffd640 1 -- 172.21.15.57:0/2957422939 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe780046c0 con 0x55c8c83568e0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe66ffd640 1 -- 172.21.15.57:0/2957422939 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbe78011650 con 0x55c8c83568e0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.444+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe78015070 con 0x55c8c83568e0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe66ffd640 1 -- 172.21.15.57:0/2957422939 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbe78030050 con 0x55c8c83568e0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 10 client.15978 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c8c7f8d170 con 0x55c8c83568e0 2024-04-21T08:46:34.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe66ffd640 1 -- 172.21.15.57:0/2957422939 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe78015070 con 0x55c8c83568e0 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe66ffd640 1 -- 172.21.15.57:0/2957422939 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbe78025840 con 0x55c8c83568e0 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 10 client.15978 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 20 client.15978 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c8370250 0x55c8c8372640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 4 client.15978 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8c82c8920 con 0x55c8c8370250 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.445+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c8370250 0x55c8c8372640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.446+0000 7fbe66ffd640 10 client.15978 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.446+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c8370250 0x55c8c8372640 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c1be138c4515f17 server_cookie=5768f01c77355dff in_seq=0 out_seq=0 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.446+0000 7fbe66ffd640 1 -- 172.21.15.57:0/2957422939 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c8c82c8920 con 0x55c8c8370250 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.446+0000 7fbe66ffd640 10 client.15978 handle_command_reply: tid=0 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.446+0000 7fbe9079f740 1 client.15978 shutdown 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 msgr2=0x55c8c835ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835ce20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbe78018000 tx=0x7fbe78003e00 comp rx=0 tx=0).stop 2024-04-21T08:46:34.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 shutdown_connections 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c8370250 0x55c8c8372640 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c835d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c835d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2957422939 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835ce20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe66ffd640 0 client.15978 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 >> 172.21.15.57:0/2957422939 conn(0x55c8c8350fb0 msgr2=0x55c8c8352d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 shutdown_connections 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 1 -- 172.21.15.57:0/2957422939 wait complete. 2024-04-21T08:46:34.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.447+0000 7fbe9079f740 20 client.15978 trim_cache size 0 max 16384 2024-04-21T08:46:34.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.465+0000 7fbe9079f740 1 Processor -- start 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.465+0000 7fbe9079f740 1 -- start start 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c836bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c834fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe9079f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881af500 con 0x55c8c83568e0 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe9079f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881b1d60 con 0x55c8c835fe10 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe9079f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881afb50 con 0x55c8c8357290 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36836/0 (socket says 172.21.15.57:36836) 2024-04-21T08:46:34.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 -- 172.21.15.57:0/2387148385 learned_addr learned my addr 172.21.15.57:0/2387148385 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe87fff640 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c834fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 -- 172.21.15.57:0/2387148385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 msgr2=0x55c8c836bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c836bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 -- 172.21.15.57:0/2387148385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 msgr2=0x55c8c834fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c834fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 -- 172.21.15.57:0/2387148385 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8c835b370 con 0x55c8c83568e0 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe877fe640 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835f550 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b3d0 tx=0x7fbe78005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b6b1314d7d4ed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe6d7fa640 1 -- 172.21.15.57:0/2387148385 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78003e70 con 0x55c8c83568e0 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe6d7fa640 1 -- 172.21.15.57:0/2387148385 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe78015070 con 0x55c8c83568e0 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.466+0000 7fbe6d7fa640 1 -- 172.21.15.57:0/2387148385 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78011670 con 0x55c8c83568e0 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 -- 172.21.15.57:0/2387148385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 msgr2=0x55c8c835f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835f550 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800b3d0 tx=0x7fbe78005f30 comp rx=0 tx=0).stop 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 -- 172.21.15.57:0/2387148385 shutdown_connections 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c834fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c836bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 --2- 172.21.15.57:0/2387148385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c835f550 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 -- 172.21.15.57:0/2387148385 >> 172.21.15.57:0/2387148385 conn(0x55c8c8350fb0 msgr2=0x55c8c834f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 -- 172.21.15.57:0/2387148385 shutdown_connections 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 -- 172.21.15.57:0/2387148385 wait complete. 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.467+0000 7fbe9079f740 1 -- start start 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 Processor -- start 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c83ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c83cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c8364440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881b0220 con 0x55c8c83568e0 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881b1690 con 0x55c8c835fe10 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbe881b08f0 con 0x55c8c8357290 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe9079f740 10 client.0 _ll_register_callbacks cb 0x7fbe80037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c83ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c83ce8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36846/0 (socket says 172.21.15.57:36846) 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 -- 172.21.15.57:0/614268311 learned_addr learned my addr 172.21.15.57:0/614268311 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe87fff640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c8364440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c83cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 -- 172.21.15.57:0/614268311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 msgr2=0x55c8c83cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c83cee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 -- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 msgr2=0x55c8c8364440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c8364440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 -- 172.21.15.57:0/614268311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8c8364b30 con 0x55c8c83568e0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c83cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe877fe640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c83ce8f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbe78003b70 tx=0x7fbe7800b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d70ad85cb7fc30 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.468+0000 7fbe66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe66ffd640 1 -- 172.21.15.57:0/614268311 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78003e60 con 0x55c8c83568e0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe78015070 con 0x55c8c83568e0 2024-04-21T08:46:34.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe66ffd640 1 -- 172.21.15.57:0/614268311 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbe78025430 con 0x55c8c83568e0 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe66ffd640 1 -- 172.21.15.57:0/614268311 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbe7804b050 con 0x55c8c83568e0 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 10 client.15984 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c8c83d7200 con 0x55c8c83568e0 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe66ffd640 1 -- 172.21.15.57:0/614268311 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe78015070 con 0x55c8c83568e0 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe66ffd640 1 -- 172.21.15.57:0/614268311 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbe7803d450 con 0x55c8c83568e0 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 10 client.15984 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 20 client.15984 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c83d7830 0x55c8c83d9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 4 client.15984 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:46:34.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c8c82c2790 con 0x55c8c83d7830 2024-04-21T08:46:34.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.469+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c83d7830 0x55c8c83d9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.470+0000 7fbe66ffd640 10 client.15984 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.470+0000 7fbe86ffd640 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c83d7830 0x55c8c83d9c50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ab831d59ac05ded server_cookie=6557f0074b40a738 in_seq=0 out_seq=0 2024-04-21T08:46:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.471+0000 7fbe66ffd640 1 -- 172.21.15.57:0/614268311 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c8c82c2790 con 0x55c8c83d7830 2024-04-21T08:46:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.471+0000 7fbe66ffd640 10 client.15984 handle_command_reply: tid=0 2024-04-21T08:46:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.471+0000 7fbe9079f740 1 client.15984 shutdown 2024-04-21T08:46:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.471+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 msgr2=0x55c8c83ce8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.471+0000 7fbe9079f740 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c83ce8f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fbe78003b70 tx=0x7fbe7800b470 comp rx=0 tx=0).stop 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 shutdown_connections 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c8c83d7830 0x55c8c83d9c50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c8c835fe10 0x55c8c8364440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c8c8357290 0x55c8c83cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe66ffd640 0 client.15984 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 --2- 172.21.15.57:0/614268311 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c8c83568e0 0x55c8c83ce8f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 >> 172.21.15.57:0/614268311 conn(0x55c8c8350fb0 msgr2=0x55c8c8350010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 shutdown_connections 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 1 -- 172.21.15.57:0/614268311 wait complete. 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe9079f740 20 client.15984 trim_cache size 0 max 16384 2024-04-21T08:46:34.477 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:46:34.478 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:46:34.478 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "e6b26d66-400c-435e-9acc-65df2e3760be", 2024-04-21T08:46:34.478 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:46:34.478 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbe700a3ec0 msgr2=0x7fbe700a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbe700a3ec0 0x7fbe700a6380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe80006010 tx=0x7fbe80005d80 comp rx=0 tx=0).stop 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 msgr2=0x7fbe881a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 0x7fbe881a0bc0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c00e9f0 tx=0x7fbe7c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 shutdown_connections 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbe700a3ec0 0x7fbe700a6380 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8810e900 0x7fbe8819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbe88101670 0x7fbe881a0bc0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 --2- 172.21.15.57:0/2817181932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbe88100d50 0x7fbe881a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.472+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 >> 172.21.15.57:0/2817181932 conn(0x7fbe88077f50 msgr2=0x7fbe88110030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.473+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 shutdown_connections 2024-04-21T08:46:34.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.473+0000 7fbe8dad2640 1 -- 172.21.15.57:0/2817181932 wait complete. 2024-04-21T08:46:34.483 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:46:34.483 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:46:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 -- 172.21.15.57:0/1451679964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8100d40 msgr2=0x7fb2d8101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/1451679964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8100d40 0x7fb2d8101120 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009900 tx=0x7fb2c802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 -- 172.21.15.57:0/1451679964 shutdown_connections 2024-04-21T08:46:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/1451679964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2d810e8f0 0x7fb2d8110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/1451679964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d8101660 0x7fb2d810e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/1451679964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8100d40 0x7fb2d8101120 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 -- 172.21.15.57:0/1451679964 >> 172.21.15.57:0/1451679964 conn(0x7fb2d8078000 msgr2=0x7fb2d8078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 -- 172.21.15.57:0/1451679964 shutdown_connections 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 -- 172.21.15.57:0/1451679964 wait complete. 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 Processor -- start 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.632+0000 7fb2de3b2640 1 -- start start 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2de3b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2d8100d40 0x7fb2d81a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2de3b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 0x7fb2d81a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2de3b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d810e8f0 0x7fb2d819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2de3b2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8114410 con 0x7fb2d8101660 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2de3b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8114290 con 0x7fb2d810e8f0 2024-04-21T08:46:34.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2de3b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8114590 con 0x7fb2d8100d40 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 0x7fb2d81a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 0x7fb2d81a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36858/0 (socket says 172.21.15.57:36858) 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3555112520 learned_addr learned my addr 172.21.15.57:0/3555112520 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d810e8f0 0x7fb2d819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2d8100d40 0x7fb2d81a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3555112520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2d8100d40 msgr2=0x7fb2d81a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2d8100d40 0x7fb2d81a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3555112520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d810e8f0 msgr2=0x7fb2d819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d810e8f0 0x7fb2d819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3555112520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d819aff0 con 0x7fb2d8101660 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d810e8f0 0x7fb2d819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.633+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 0x7fb2d81a0bb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000e9f0 tx=0x7fb2c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.634+0000 7fb2d57fa640 1 -- 172.21.15.57:0/3555112520 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000cd80 con 0x7fb2d8101660 2024-04-21T08:46:34.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.634+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d819b2e0 con 0x7fb2d8101660 2024-04-21T08:46:34.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.634+0000 7fb2d57fa640 1 -- 172.21.15.57:0/3555112520 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c00108e0 con 0x7fb2d8101660 2024-04-21T08:46:34.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.634+0000 7fb2d57fa640 1 -- 172.21.15.57:0/3555112520 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0018980 con 0x7fb2d8101660 2024-04-21T08:46:34.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.634+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d81a75f0 con 0x7fb2d8101660 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.636+0000 7fb2d57fa640 1 -- 172.21.15.57:0/3555112520 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fb2c0010430 con 0x7fb2d8101660 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.636+0000 7fb2e0fa2740 1 Processor -- start 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 -- start start 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d57fa640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb2c40a3e40 0x7fb2c40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31a6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31af540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55beb310b1e0 con 0x55beb31afa80 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55beb30a8390 con 0x55beb31a6580 2024-04-21T08:46:34.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55beb31071a0 con 0x55beb31a6ea0 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb2c40a3e40 0x7fb2c40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57764/0 (socket says 172.21.15.57:57764) 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d7fff640 1 -- 172.21.15.57:0/1647934021 learned_addr learned my addr 172.21.15.57:0/1647934021 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d57fa640 1 -- 172.21.15.57:0/3555112520 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fb2c00d10d0 con 0x7fb2d8101660 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 -- 172.21.15.57:0/1647934021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 msgr2=0x55beb31af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 -- 172.21.15.57:0/1647934021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 msgr2=0x55beb31a6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31a6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2dc94f640 1 -- 172.21.15.57:0/1647934021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beb2ee77d0 con 0x55beb31afa80 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.637+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb2c40a3e40 0x7fb2c40a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009fd0 tx=0x7fb2d00116b0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b1e70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb2d806b6f0 tx=0x7fb2d001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465999942f7e6f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2ce7fc640 1 -- 172.21.15.57:0/1647934021 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0024070 con 0x55beb31afa80 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2ce7fc640 1 -- 172.21.15.57:0/1647934021 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d0020c50 con 0x55beb31afa80 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2ce7fc640 1 -- 172.21.15.57:0/1647934021 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0005630 con 0x55beb31afa80 2024-04-21T08:46:34.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/1647934021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 msgr2=0x55beb31b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.638+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b1e70 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb2d806b6f0 tx=0x7fb2d001d000 comp rx=0 tx=0).stop 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/1647934021 shutdown_connections 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b1e70 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/1647934021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31a6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/1647934021 >> 172.21.15.57:0/1647934021 conn(0x55beb31a0cb0 msgr2=0x55beb31a1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/1647934021 shutdown_connections 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/1647934021 wait complete. 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- start start 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 Processor -- start 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6580 0x55beb31aca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31acf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31afa80 0x55beb31ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55beb31b2810 con 0x55beb31a6ea0 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55beb30c42f0 con 0x55beb31afa80 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55beb31b2990 con 0x55beb31a6580 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.639+0000 7fb2e0fa2740 10 client.0 _ll_register_callbacks cb 0x55beb30d1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31acf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36872/0 (socket says 172.21.15.57:36872) 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3992323893 learned_addr learned my addr 172.21.15.57:0/3992323893 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31afa80 0x55beb31ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6580 0x55beb31aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3992323893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6580 msgr2=0x55beb31aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6580 0x55beb31aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31afa80 msgr2=0x55beb31ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31afa80 0x55beb31ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 -- 172.21.15.57:0/3992323893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beb2e1d3f0 con 0x55beb31a6ea0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6580 0x55beb31aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31acf90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb2c009cb00 tx=0x7fb2c0010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eae59d6965f5e46 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3992323893 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0010a90 con 0x55beb31a6ea0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2d0020c50 con 0x55beb31a6ea0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3992323893 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fb2c0087370 con 0x55beb31a6ea0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3992323893 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb2c0094ff0 con 0x55beb31a6ea0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2e0fa2740 10 client.15996 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.640+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55beb2ddd0f0 con 0x55beb31a6ea0 2024-04-21T08:46:34.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3992323893 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2d0020c50 con 0x55beb31a6ea0 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3992323893 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb2c0018b20 con 0x55beb31a6ea0 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2e0fa2740 10 client.15996 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2e0fa2740 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2e0fa2740 20 client.15996 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb31c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2e0fa2740 4 client.15996 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55beb3118c00 con 0x55beb31bf9c0 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb31c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb31c1e60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0edb3036758bf4 server_cookie=b02ebaa2d7061e07 in_seq=0 out_seq=0 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.641+0000 7fb2b77fe640 10 client.15996 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.643+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3992323893 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55beb3118c00 con 0x55beb31bf9c0 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.643+0000 7fb2b77fe640 10 client.15996 handle_command_reply: tid=0 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.643+0000 7fb2e0fa2740 1 client.15996 shutdown 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.643+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 msgr2=0x55beb31acf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31acf90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb2c009cb00 tx=0x7fb2c0010860 comp rx=0 tx=0).stop 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 shutdown_connections 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb31c1e60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31afa80 0x55beb31ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2b77fe640 0 client.15996 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31acf90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3992323893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6580 0x55beb31aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.651 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 >> 172.21.15.57:0/3992323893 conn(0x55beb31a0cb0 msgr2=0x55beb31a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 shutdown_connections 2024-04-21T08:46:34.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3992323893 wait complete. 2024-04-21T08:46:34.652 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.644+0000 7fb2e0fa2740 20 client.15996 trim_cache size 0 max 16384 2024-04-21T08:46:34.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 Processor -- start 2024-04-21T08:46:34.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 -- start start 2024-04-21T08:46:34.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31af180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31ab990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31afa80 0x55beb31bb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81af3d0 con 0x55beb31a6ea0 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81b1c30 con 0x55beb31a6580 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81afa20 con 0x55beb31afa80 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31ab990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36888/0 (socket says 172.21.15.57:36888) 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 -- 172.21.15.57:0/2979653335 learned_addr learned my addr 172.21.15.57:0/2979653335 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31afa80 0x55beb31bb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 -- 172.21.15.57:0/2979653335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31afa80 msgr2=0x55beb31bb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31afa80 0x55beb31bb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 -- 172.21.15.57:0/2979653335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 msgr2=0x55beb31af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2d77fe640 1 -- 172.21.15.57:0/2979653335 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55beb31a0ae0 con 0x55beb31a6ea0 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.662+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31afa80 0x55beb31bb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31ab990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0098fd0 tx=0x7fb2c0018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bf41f8700efd3c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2cdffb640 1 -- 172.21.15.57:0/2979653335 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c00b5030 con 0x55beb31a6ea0 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2cdffb640 1 -- 172.21.15.57:0/2979653335 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0010a90 con 0x55beb31a6ea0 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2cdffb640 1 -- 172.21.15.57:0/2979653335 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0087370 con 0x55beb31a6ea0 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/2979653335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 msgr2=0x55beb31ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31ab990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0098fd0 tx=0x7fb2c0018420 comp rx=0 tx=0).stop 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/2979653335 shutdown_connections 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31afa80 0x55beb31bb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31a6ea0 0x55beb31ab990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/2979653335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/2979653335 >> 172.21.15.57:0/2979653335 conn(0x55beb31a0cb0 msgr2=0x55beb31a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/2979653335 shutdown_connections 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.663+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/2979653335 wait complete. 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 -- start start 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 Processor -- start 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81b00f0 con 0x55beb31afa80 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81b1560 con 0x55beb31a6580 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d81b07c0 con 0x55beb31a6ea0 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2e0fa2740 10 client.0 _ll_register_callbacks cb 0x7fb2c000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2dc94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2dc94f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36896/0 (socket says 172.21.15.57:36896) 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2dc94f640 1 -- 172.21.15.57:0/3577980616 learned_addr learned my addr 172.21.15.57:0/3577980616 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:34.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.664+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2dc94f640 1 -- 172.21.15.57:0/3577980616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 msgr2=0x55beb31b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2dc94f640 1 -- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 msgr2=0x55beb31b7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31b7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2dc94f640 1 -- 172.21.15.57:0/3577980616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55beb31b4cd0 con 0x55beb31afa80 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2d77fe640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2dc94f640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b4710 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005fd0 tx=0x7fb2d0005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d1977ef71f9a19 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3577980616 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0024040 con 0x55beb31afa80 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3577980616 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fb2d001f470 con 0x55beb31afa80 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c0010a90 con 0x55beb31afa80 2024-04-21T08:46:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3577980616 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb2d0040050 con 0x55beb31afa80 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2e0fa2740 10 client.16002 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55beb3226d70 con 0x55beb31afa80 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3577980616 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c0010a90 con 0x55beb31afa80 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.665+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3577980616 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb2d003b030 con 0x55beb31afa80 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2e0fa2740 10 client.16002 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2e0fa2740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2e0fa2740 20 client.16002 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb32291e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2e0fa2740 4 client.16002 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55beb3112a70 con 0x55beb31bf9c0 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb32291e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2d7fff640 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb32291e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5a70525d9ed6b0a server_cookie=f85484530828484 in_seq=0 out_seq=0 2024-04-21T08:46:34.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.666+0000 7fb2b77fe640 10 client.16002 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2b77fe640 1 -- 172.21.15.57:0/3577980616 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55beb3112a70 con 0x55beb31bf9c0 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2b77fe640 10 client.16002 handle_command_reply: tid=0 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 client.16002 shutdown 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 msgr2=0x55beb31b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b4710 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0005fd0 tx=0x7fb2d0005df0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 shutdown_connections 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55beb31bf9c0 0x55beb32291e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55beb31afa80 0x55beb31b4710 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2b77fe640 0 client.16002 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:34.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55beb31a6ea0 0x55beb31b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 --2- 172.21.15.57:0/3577980616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55beb31a6580 0x55beb31b7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 >> 172.21.15.57:0/3577980616 conn(0x55beb31a0cb0 msgr2=0x55beb31a20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 shutdown_connections 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 1 -- 172.21.15.57:0/3577980616 wait complete. 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.672+0000 7fb2e0fa2740 20 client.16002 trim_cache size 0 max 16384 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (1119 inodes in the stack)", 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stdout: "e6b26d66-400c-435e-9acc-65df2e3760be": { 2024-04-21T08:46:34.674 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:46:34.675 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "e6b26d66-400c-435e-9acc-65df2e3760be", 2024-04-21T08:46:34.675 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:46:34.675 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:46:34.675 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:46:34.675 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb2c40a3e40 msgr2=0x7fb2c40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb2c40a3e40 0x7fb2c40a6300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009fd0 tx=0x7fb2d00116b0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 msgr2=0x7fb2d81a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 0x7fb2d81a0bb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000e9f0 tx=0x7fb2c000eeb0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 shutdown_connections 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb2c40a3e40 0x7fb2c40a6300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2d810e8f0 0x7fb2d819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2d8101660 0x7fb2d81a0bb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 --2- 172.21.15.57:0/3555112520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2d8100d40 0x7fb2d81a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 >> 172.21.15.57:0/3555112520 conn(0x7fb2d8078000 msgr2=0x7fb2d810ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 shutdown_connections 2024-04-21T08:46:34.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:46:34.673+0000 7fb2de3b2640 1 -- 172.21.15.57:0/3555112520 wait complete. 2024-04-21T08:46:34.684 INFO:tasks.cephfs.filesystem:scrub status for tag:e6b26d66-400c-435e-9acc-65df2e3760be - {'path': '/', 'tag': 'e6b26d66-400c-435e-9acc-65df2e3760be', 'options': 'recursive,force'} 2024-04-21T08:46:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:36 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:46:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:36 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:36 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:46:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:36 smithi129 ceph-mon[31272]: pgmap v38: 129 pgs: 129 active+clean; 139 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 11 MiB/s wr, 32 op/s 2024-04-21T08:46:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:36 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:46:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:36 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:36 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:46:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:36 smithi057 ceph-mon[27968]: pgmap v38: 129 pgs: 129 active+clean; 139 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 11 MiB/s wr, 32 op/s 2024-04-21T08:46:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:36 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:46:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:36 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:36 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:46:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:36 smithi136 ceph-mon[31791]: pgmap v38: 129 pgs: 129 active+clean; 139 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 11 MiB/s wr, 32 op/s 2024-04-21T08:46:37.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:37 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:37.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:37 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:46:37.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:37 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:37.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:37 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:46:37.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:37 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:46:37.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:37 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:46:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.631+0000 7fec09fbd640 1 -- 172.21.15.57:0/4174474875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec041010f0 msgr2=0x7fec041014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:37.634 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.631+0000 7fec09fbd640 1 --2- 172.21.15.57:0/4174474875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec041010f0 0x7fec041014d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7febf4009900 tx=0x7febf402bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 -- 172.21.15.57:0/4174474875 shutdown_connections 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 --2- 172.21.15.57:0/4174474875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec04106170 0x7fec0410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 --2- 172.21.15.57:0/4174474875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec04101aa0 0x7fec04105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 --2- 172.21.15.57:0/4174474875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec041010f0 0x7fec041014d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 -- 172.21.15.57:0/4174474875 >> 172.21.15.57:0/4174474875 conn(0x7fec04077fd0 msgr2=0x7fec040783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 -- 172.21.15.57:0/4174474875 shutdown_connections 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 -- 172.21.15.57:0/4174474875 wait complete. 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.632+0000 7fec09fbd640 1 Processor -- start 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 -- start start 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec041010f0 0x7fec04198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 0x7fec04198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.635 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec0419cfd0 0x7fec0419d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fec041110e0 con 0x7fec04106170 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec04110f60 con 0x7fec041010f0 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec09fbd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fec04111260 con 0x7fec0419cfd0 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 0x7fec04198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 0x7fec04198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36918/0 (socket says 172.21.15.57:36918) 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec041010f0 0x7fec04198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 -- 172.21.15.57:0/3548996099 learned_addr learned my addr 172.21.15.57:0/3548996099 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec03fff640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec0419cfd0 0x7fec0419d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 -- 172.21.15.57:0/3548996099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec0419cfd0 msgr2=0x7fec0419d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec0419cfd0 0x7fec0419d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 -- 172.21.15.57:0/3548996099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec041010f0 msgr2=0x7fec04198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:37.636 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec041010f0 0x7fec04198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 -- 172.21.15.57:0/3548996099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0419dac0 con 0x7fec04106170 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec03fff640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec0419cfd0 0x7fec0419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.633+0000 7fec02ffd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 0x7fec04198c50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7febf800e9c0 tx=0x7febf800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.634+0000 7fec00ff9640 1 -- 172.21.15.57:0/3548996099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf800cd80 con 0x7fec04106170 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.634+0000 7fec00ff9640 1 -- 172.21.15.57:0/3548996099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febf80054d0 con 0x7fec04106170 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.634+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0419ddb0 con 0x7fec04106170 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.634+0000 7fec00ff9640 1 -- 172.21.15.57:0/3548996099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8010430 con 0x7fec04106170 2024-04-21T08:46:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.634+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec041a55f0 con 0x7fec04106170 2024-04-21T08:46:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.636+0000 7fec00ff9640 1 -- 172.21.15.57:0/3548996099 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7febf8004fd0 con 0x7fec04106170 2024-04-21T08:46:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.636+0000 7fec0cbb4740 1 Processor -- start 2024-04-21T08:46:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 -- start start 2024-04-21T08:46:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf773620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf773fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.644 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf7838e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55edbf6d88d0 con 0x55edbf7814f0 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55edbf675e80 con 0x55edbf773b60 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55edbf6d4890 con 0x55edbf7748d0 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf773fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec02ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf7838e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec03fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf773620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec00ff9640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7febec0a3f60 0x7febec0a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf773fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57824/0 (socket says 172.21.15.57:57824) 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 -- 172.21.15.57:0/2322538378 learned_addr learned my addr 172.21.15.57:0/2322538378 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec03fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf773620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:58760/0 (socket says 172.21.15.57:58760) 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 -- 172.21.15.57:0/2322538378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 msgr2=0x55edbf773620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.645 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec00ff9640 1 -- 172.21.15.57:0/3548996099 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7febf80d1210 con 0x7fec04106170 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf773620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec03fff640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7febec0a3f60 0x7febec0a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 -- 172.21.15.57:0/2322538378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 msgr2=0x55edbf7838e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf7838e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.637+0000 7fec037fe640 1 -- 172.21.15.57:0/2322538378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbf697320 con 0x55edbf773b60 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.638+0000 7fec037fe640 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf773fe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febf4009900 tx=0x7febf402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55fe9ffe27c681 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.638+0000 7fec03fff640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7febec0a3f60 0x7febec0a6420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7febfc0127f0 tx=0x7febfc004ab0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.638+0000 7febe9ffb640 1 -- 172.21.15.57:0/2322538378 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4045020 con 0x55edbf773b60 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.638+0000 7febe9ffb640 1 -- 172.21.15.57:0/2322538378 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febf402c7e0 con 0x55edbf773b60 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.638+0000 7febe9ffb640 1 -- 172.21.15.57:0/2322538378 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf40046c0 con 0x55edbf773b60 2024-04-21T08:46:37.646 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2322538378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 msgr2=0x55edbf773fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf773fe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7febf4009900 tx=0x7febf402c6c0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2322538378 shutdown_connections 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf7838e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf773fe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2322538378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf773620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2322538378 >> 172.21.15.57:0/2322538378 conn(0x55edbf73ac90 msgr2=0x55edbf76e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2322538378 shutdown_connections 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2322538378 wait complete. 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 -- start start 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.639+0000 7fec0cbb4740 1 Processor -- start 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf780e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.647 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf77e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf77e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55edbf7841c0 con 0x55edbf7814f0 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55edbf660eb0 con 0x55edbf773b60 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55edbf784340 con 0x55edbf7748d0 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec0cbb4740 10 client.0 _ll_register_callbacks cb 0x55edbf68db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf77e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf780e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf77e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36944/0 (socket says 172.21.15.57:36944) 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 -- 172.21.15.57:0/182407345 learned_addr learned my addr 172.21.15.57:0/182407345 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec02ffd640 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf77e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 -- 172.21.15.57:0/182407345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 msgr2=0x55edbf77e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.648 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf77e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 -- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 msgr2=0x55edbf780e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf780e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 -- 172.21.15.57:0/182407345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbf3eb210 con 0x55edbf7814f0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7fec03fff640 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf77e7c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7febfc00a780 tx=0x7febfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4015b46aa199ee9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.640+0000 7febe2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7febe2ffd640 1 -- 172.21.15.57:0/182407345 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc018070 con 0x55edbf7814f0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7febe2ffd640 1 -- 172.21.15.57:0/182407345 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7febfc003970 con 0x55edbf7814f0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf402c7e0 con 0x55edbf7814f0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7febe2ffd640 1 -- 172.21.15.57:0/182407345 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7febfc02f050 con 0x55edbf7814f0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 10 client.16014 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55edbf4b7620 con 0x55edbf7814f0 2024-04-21T08:46:37.649 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7febe2ffd640 1 -- 172.21.15.57:0/182407345 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febf402c7e0 con 0x55edbf7814f0 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7febe2ffd640 1 -- 172.21.15.57:0/182407345 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7febfc027400 con 0x55edbf7814f0 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 10 client.16014 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 20 client.16014 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf791550 0x55edbf793960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 4 client.16014 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.641+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edbf6ec880 con 0x55edbf791550 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.642+0000 7fec037fe640 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf791550 0x55edbf793960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.642+0000 7fec037fe640 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf791550 0x55edbf793960 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9609f7c4f8add364 server_cookie=79f1b09ced3f29b6 in_seq=0 out_seq=0 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.642+0000 7febe2ffd640 10 client.16014 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7febe2ffd640 1 -- 172.21.15.57:0/182407345 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55edbf6ec880 con 0x55edbf791550 2024-04-21T08:46:37.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7febe2ffd640 10 client.16014 handle_command_reply: tid=0 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 client.16014 shutdown 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 msgr2=0x55edbf77e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf77e7c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7febfc00a780 tx=0x7febfc004a40 comp rx=0 tx=0).stop 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 shutdown_connections 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf791550 0x55edbf793960 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf77e7c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf77e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/182407345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf780e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7febe2ffd640 0 client.16014 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 >> 172.21.15.57:0/182407345 conn(0x55edbf73ac90 msgr2=0x55edbf7705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 shutdown_connections 2024-04-21T08:46:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 1 -- 172.21.15.57:0/182407345 wait complete. 2024-04-21T08:46:37.652 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.644+0000 7fec0cbb4740 20 client.16014 trim_cache size 0 max 16384 2024-04-21T08:46:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.659+0000 7fec0cbb4740 1 Processor -- start 2024-04-21T08:46:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 -- start start 2024-04-21T08:46:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf78d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf780950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf780e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fec041aafa0 con 0x55edbf7814f0 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec041ad800 con 0x55edbf773b60 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fec041ab5f0 con 0x55edbf7748d0 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf780e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf780e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36956/0 (socket says 172.21.15.57:36956) 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec037fe640 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf78d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 -- 172.21.15.57:0/2606624220 learned_addr learned my addr 172.21.15.57:0/2606624220 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec02ffd640 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf780950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 -- 172.21.15.57:0/2606624220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 msgr2=0x55edbf780950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf780950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 -- 172.21.15.57:0/2606624220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 msgr2=0x55edbf78d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf78d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 -- 172.21.15.57:0/2606624220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edbf783d50 con 0x55edbf7814f0 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec02ffd640 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf780950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.660+0000 7fec03fff640 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf780e90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7febfc0277d0 tx=0x7febfc00a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332c87c8145b4f9a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7febe97fa640 1 -- 172.21.15.57:0/2606624220 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc039070 con 0x55edbf7814f0 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7febe97fa640 1 -- 172.21.15.57:0/2606624220 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febfc010c00 con 0x55edbf7814f0 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7febe97fa640 1 -- 172.21.15.57:0/2606624220 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc018040 con 0x55edbf7814f0 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2606624220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 msgr2=0x55edbf780e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf780e90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7febfc0277d0 tx=0x7febfc00a5a0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2606624220 shutdown_connections 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf780e90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf780950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/2606624220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf78d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2606624220 >> 172.21.15.57:0/2606624220 conn(0x55edbf73ac90 msgr2=0x55edbf76e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2606624220 shutdown_connections 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.661+0000 7fec0cbb4740 1 -- 172.21.15.57:0/2606624220 wait complete. 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 -- start start 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 Processor -- start 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf77bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf7895d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf782490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fec041abcc0 con 0x55edbf7814f0 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec041ad130 con 0x55edbf773b60 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fec041ac390 con 0x55edbf7748d0 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec0cbb4740 10 client.0 _ll_register_callbacks cb 0x7febfc00ae10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec03fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf782490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec03fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf782490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36972/0 (socket says 172.21.15.57:36972) 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec037fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf77bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.662+0000 7fec03fff640 1 -- 172.21.15.57:0/559199041 learned_addr learned my addr 172.21.15.57:0/559199041 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec02ffd640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf7895d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec03fff640 1 -- 172.21.15.57:0/559199041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 msgr2=0x55edbf7895d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec03fff640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf7895d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec03fff640 1 -- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 msgr2=0x55edbf77bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec03fff640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf77bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec03fff640 1 -- 172.21.15.57:0/559199041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edbf7829d0 con 0x55edbf7814f0 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec02ffd640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf7895d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec037fe640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf77bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec03fff640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf782490 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7febfc0108c0 tx=0x7febfc0276b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c55ba470f1bd47b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7febe2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7febe2ffd640 1 -- 172.21.15.57:0/559199041 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc039040 con 0x55edbf7814f0 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7febe2ffd640 1 -- 172.21.15.57:0/559199041 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7febfc027bd0 con 0x55edbf7814f0 2024-04-21T08:46:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55edbf6783d0 con 0x55edbf7814f0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7febe2ffd640 1 -- 172.21.15.57:0/559199041 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7febfc04a050 con 0x55edbf7814f0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec0cbb4740 10 client.16020 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55edbf7f8170 con 0x55edbf7814f0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.663+0000 7febe2ffd640 1 -- 172.21.15.57:0/559199041 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55edbf6783d0 con 0x55edbf7814f0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7febe2ffd640 1 -- 172.21.15.57:0/559199041 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7febfc01f400 con 0x55edbf7814f0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec0cbb4740 10 client.16020 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec0cbb4740 10 client.16020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec0cbb4740 20 client.16020 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf7f87b0 0x55edbf7fabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec0cbb4740 4 client.16020 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55edbf6e62f0 con 0x55edbf7f87b0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec037fe640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf7f87b0 0x55edbf7fabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7febe2ffd640 10 client.16020 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:37.664+0000 7fec037fe640 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf7f87b0 0x55edbf7fabd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c66374ce3d888988 server_cookie=5f7cf8aa0d608955 in_seq=0 out_seq=0 2024-04-21T08:46:38.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:38 smithi129 ceph-mon[31272]: pgmap v39: 129 pgs: 129 active+clean; 145 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 12 MiB/s wr, 44 op/s 2024-04-21T08:46:38.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:38 smithi057 ceph-mon[27968]: pgmap v39: 129 pgs: 129 active+clean; 145 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 12 MiB/s wr, 44 op/s 2024-04-21T08:46:38.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:38 smithi136 ceph-mon[31791]: pgmap v39: 129 pgs: 129 active+clean; 145 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 12 MiB/s wr, 44 op/s 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.228+0000 7febe2ffd640 1 -- 172.21.15.57:0/559199041 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55edbf6e62f0 con 0x55edbf7f87b0 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.228+0000 7febe2ffd640 10 client.16020 handle_command_reply: tid=0 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.228+0000 7fec0cbb4740 1 client.16020 shutdown 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 msgr2=0x55edbf782490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf782490 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7febfc0108c0 tx=0x7febfc0276b0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 shutdown_connections 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55edbf7f87b0 0x55edbf7fabd0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7febe2ffd640 0 client.16020 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55edbf7814f0 0x55edbf782490 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55edbf7748d0 0x55edbf7895d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 --2- 172.21.15.57:0/559199041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55edbf773b60 0x55edbf77bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 >> 172.21.15.57:0/559199041 conn(0x55edbf73ac90 msgr2=0x55edbf76d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 shutdown_connections 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 1 -- 172.21.15.57:0/559199041 wait complete. 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.229+0000 7fec0cbb4740 20 client.16020 trim_cache size 0 max 16384 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:46:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:46:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:46:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:46:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7febec0a3f60 msgr2=0x7febec0a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7febec0a3f60 0x7febec0a6420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7febfc0127f0 tx=0x7febfc004ab0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 msgr2=0x7fec04198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 0x7fec04198c50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7febf800e9c0 tx=0x7febf800ee80 comp rx=0 tx=0).stop 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 shutdown_connections 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7febec0a3f60 0x7febec0a6420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec0419cfd0 0x7fec0419d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec04106170 0x7fec04198c50 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 --2- 172.21.15.57:0/3548996099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec041010f0 0x7fec04198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 >> 172.21.15.57:0/3548996099 conn(0x7fec04077fd0 msgr2=0x7fec040699b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 shutdown_connections 2024-04-21T08:46:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:40.230+0000 7fec09fbd640 1 -- 172.21.15.57:0/3548996099 wait complete. 2024-04-21T08:46:40.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:40 smithi129 ceph-mon[31272]: pgmap v40: 129 pgs: 129 active+clean; 145 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 12 MiB/s wr, 39 op/s 2024-04-21T08:46:40.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:40 smithi057 ceph-mon[27968]: pgmap v40: 129 pgs: 129 active+clean; 145 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 12 MiB/s wr, 39 op/s 2024-04-21T08:46:40.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:40 smithi136 ceph-mon[31791]: pgmap v40: 129 pgs: 129 active+clean; 145 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 12 MiB/s wr, 39 op/s 2024-04-21T08:46:42.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:42 smithi129 ceph-mon[31272]: pgmap v41: 129 pgs: 129 active+clean; 140 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-04-21T08:46:42.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:42 smithi057 ceph-mon[27968]: pgmap v41: 129 pgs: 129 active+clean; 140 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-04-21T08:46:42.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:42 smithi136 ceph-mon[31791]: pgmap v41: 129 pgs: 129 active+clean; 140 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-04-21T08:46:44.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:44 smithi129 ceph-mon[31272]: pgmap v42: 129 pgs: 129 active+clean; 140 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 7.9 MiB/s wr, 1.41k op/s 2024-04-21T08:46:44.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:44 smithi057 ceph-mon[27968]: pgmap v42: 129 pgs: 129 active+clean; 140 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 7.9 MiB/s wr, 1.41k op/s 2024-04-21T08:46:44.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:44 smithi136 ceph-mon[31791]: pgmap v42: 129 pgs: 129 active+clean; 140 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 7.9 MiB/s wr, 1.41k op/s 2024-04-21T08:46:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:46 smithi129 ceph-mon[31272]: pgmap v43: 129 pgs: 129 active+clean; 140 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 1.2 MiB/s wr, 1.40k op/s 2024-04-21T08:46:46.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:46 smithi057 ceph-mon[27968]: pgmap v43: 129 pgs: 129 active+clean; 140 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 1.2 MiB/s wr, 1.40k op/s 2024-04-21T08:46:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:46 smithi136 ceph-mon[31791]: pgmap v43: 129 pgs: 129 active+clean; 140 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 1.2 MiB/s wr, 1.40k op/s 2024-04-21T08:46:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:48 smithi129 ceph-mon[31272]: pgmap v44: 129 pgs: 129 active+clean; 147 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 1.40k op/s 2024-04-21T08:46:48.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:48 smithi057 ceph-mon[27968]: pgmap v44: 129 pgs: 129 active+clean; 147 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 1.40k op/s 2024-04-21T08:46:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:48 smithi136 ceph-mon[31791]: pgmap v44: 129 pgs: 129 active+clean; 147 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 1.40k op/s 2024-04-21T08:46:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:50 smithi129 ceph-mon[31272]: pgmap v45: 129 pgs: 129 active+clean; 147 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 1.2 MiB/s wr, 1.39k op/s 2024-04-21T08:46:50.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:50 smithi057 ceph-mon[27968]: pgmap v45: 129 pgs: 129 active+clean; 147 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 1.2 MiB/s wr, 1.39k op/s 2024-04-21T08:46:50.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:50 smithi136 ceph-mon[31791]: pgmap v45: 129 pgs: 129 active+clean; 147 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 1.2 MiB/s wr, 1.39k op/s 2024-04-21T08:46:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:52 smithi129 ceph-mon[31272]: pgmap v46: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 2.14k op/s 2024-04-21T08:46:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:52 smithi057 ceph-mon[27968]: pgmap v46: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 2.14k op/s 2024-04-21T08:46:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:52 smithi136 ceph-mon[31791]: pgmap v46: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 2.14k op/s 2024-04-21T08:46:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 -- 172.21.15.57:0/832588107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e70102b80 msgr2=0x7f7e70106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/832588107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e70102b80 0x7f7e70106bc0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009900 tx=0x7f7e5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:46:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 -- 172.21.15.57:0/832588107 shutdown_connections 2024-04-21T08:46:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/832588107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701072a0 0x7f7e7010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/832588107 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e70102b80 0x7f7e70106bc0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/832588107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e701021d0 0x7f7e701025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 -- 172.21.15.57:0/832588107 >> 172.21.15.57:0/832588107 conn(0x7f7e700793e0 msgr2=0x7f7e700797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 -- 172.21.15.57:0/832588107 shutdown_connections 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.323+0000 7f7e75e5c640 1 -- 172.21.15.57:0/832588107 wait complete. 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 Processor -- start 2024-04-21T08:46:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 -- start start 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701021d0 0x7f7e70198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e701072a0 0x7f7e70198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 0x7f7e7019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701122b0 con 0x7f7e7019cfd0 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70112130 con 0x7f7e701072a0 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e75e5c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70112430 con 0x7f7e701021d0 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 0x7f7e7019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e6ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 0x7f7e7019d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48882/0 (socket says 172.21.15.57:48882) 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e6ffff640 1 -- 172.21.15.57:0/2340504799 learned_addr learned my addr 172.21.15.57:0/2340504799 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e701072a0 0x7f7e70198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.324+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701021d0 0x7f7e70198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6ffff640 1 -- 172.21.15.57:0/2340504799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701021d0 msgr2=0x7f7e70198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6ffff640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701021d0 0x7f7e70198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6ffff640 1 -- 172.21.15.57:0/2340504799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e701072a0 msgr2=0x7f7e70198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6ffff640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e701072a0 0x7f7e70198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6ffff640 1 -- 172.21.15.57:0/2340504799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7019dac0 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701021d0 0x7f7e70198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6ffff640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 0x7f7e7019d480 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800cab0 tx=0x7f7e6800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6d7fa640 1 -- 172.21.15.57:0/2340504799 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6800adc0 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6d7fa640 1 -- 172.21.15.57:0/2340504799 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e6800f900 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7019ddb0 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e6d7fa640 1 -- 172.21.15.57:0/2340504799 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680189a0 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.325+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e701a5570 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.327+0000 7f7e6d7fa640 1 -- 172.21.15.57:0/2340504799 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f7e6800f450 con 0x7f7e7019cfd0 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.327+0000 7f7e78a54740 1 Processor -- start 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 -- start start 2024-04-21T08:46:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdf4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdf4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691ce004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6d7fa640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7e5c0a3ee0 0x7f7e5c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55691cd598d0 con 0x55691cdf58d0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55691ccf6e80 con 0x55691cdf4b60 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e78a54740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55691cd55890 con 0x55691cdfe0e0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdf4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6ffff640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7e5c0a3ee0 0x7f7e5c0a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdf4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdf4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48892/0 (socket says 172.21.15.57:48892) 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e66ffd640 1 -- 172.21.15.57:0/4123541612 learned_addr learned my addr 172.21.15.57:0/4123541612 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6d7fa640 1 -- 172.21.15.57:0/2340504799 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f7e680d10a0 con 0x7f7e7019cfd0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691ce004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/4123541612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 msgr2=0x55691ce004d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691ce004d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/4123541612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 msgr2=0x55691cdf4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdf4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/4123541612 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55691cd18320 con 0x55691cdf4b60 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdf4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6ffff640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7e5c0a3ee0 0x7f7e5c0a63a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e680ce370 tx=0x7f7e680bd040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.328+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdf4fe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e60007760 tx=0x7f7e60007c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f3d8e2a078b397c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e65ffb640 1 -- 172.21.15.57:0/4123541612 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000d950 con 0x55691cdf4b60 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e65ffb640 1 -- 172.21.15.57:0/4123541612 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e60018070 con 0x55691cdf4b60 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e65ffb640 1 -- 172.21.15.57:0/4123541612 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e60013df0 con 0x55691cdf4b60 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 -- 172.21.15.57:0/4123541612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 msgr2=0x55691cdf4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdf4fe0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e60007760 tx=0x7f7e60007c20 comp rx=0 tx=0).stop 2024-04-21T08:46:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 -- 172.21.15.57:0/4123541612 shutdown_connections 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691ce004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdf4fe0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 --2- 172.21.15.57:0/4123541612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdf4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 -- 172.21.15.57:0/4123541612 >> 172.21.15.57:0/4123541612 conn(0x55691cdbbc90 msgr2=0x55691cdef850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 -- 172.21.15.57:0/4123541612 shutdown_connections 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.329+0000 7f7e78a54740 1 -- 172.21.15.57:0/4123541612 wait complete. 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 -- start start 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 Processor -- start 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdfdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdfae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691cdfb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55691ce00e10 con 0x55691cdf58d0 2024-04-21T08:46:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55691cce1eb0 con 0x55691cdf4b60 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55691ce00f90 con 0x55691cdfe0e0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e78a54740 10 client.0 _ll_register_callbacks cb 0x55691cd0eb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdfae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e6f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdfae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48896/0 (socket says 172.21.15.57:48896) 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/494354038 learned_addr learned my addr 172.21.15.57:0/494354038 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.330+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdfdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/494354038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 msgr2=0x55691cdfb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691cdfb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691cdfb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 msgr2=0x55691cdfdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdfdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e6f7fe640 1 -- 172.21.15.57:0/494354038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55691ca6c210 con 0x55691cdf58d0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdfae80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7e60004810 tx=0x7f7e600136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6300d2f68a32cc8d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e4effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e4effd640 1 -- 172.21.15.57:0/494354038 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e60007e70 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e4effd640 1 -- 172.21.15.57:0/494354038 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f7e6000d950 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e60018070 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e4effd640 1 -- 172.21.15.57:0/494354038 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7e6002f050 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e78a54740 10 client.16032 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55691cb38620 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e4effd640 1 -- 172.21.15.57:0/494354038 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e60018070 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e4effd640 1 -- 172.21.15.57:0/494354038 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7e60024b50 con 0x55691cdf58d0 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e78a54740 10 client.16032 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.331+0000 7f7e78a54740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e78a54740 20 client.16032 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e78a54740 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce0e110 0x55691ce10500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.340 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e78a54740 4 client.16032 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:46:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55691cd6d880 con 0x55691ce0e110 2024-04-21T08:46:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce0e110 0x55691ce10500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e4effd640 10 client.16032 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:53.341 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.332+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce0e110 0x55691ce10500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a53150f1fcc683b server_cookie=e378f050a3f2643c in_seq=0 out_seq=0 2024-04-21T08:46:53.342 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.334+0000 7f7e4effd640 1 -- 172.21.15.57:0/494354038 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55691cd6d880 con 0x55691ce0e110 2024-04-21T08:46:53.342 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.334+0000 7f7e4effd640 10 client.16032 handle_command_reply: tid=0 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 client.16032 shutdown 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 msgr2=0x55691cdfae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdfae80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7e60004810 tx=0x7f7e600136b0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 shutdown_connections 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce0e110 0x55691ce10500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdfe0e0 0x55691cdfb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdf58d0 0x55691cdfae80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e4effd640 0 client.16032 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 --2- 172.21.15.57:0/494354038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691cdfdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 >> 172.21.15.57:0/494354038 conn(0x55691cdbbc90 msgr2=0x55691cdf15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 shutdown_connections 2024-04-21T08:46:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 1 -- 172.21.15.57:0/494354038 wait complete. 2024-04-21T08:46:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.335+0000 7f7e78a54740 20 client.16032 trim_cache size 0 max 16384 2024-04-21T08:46:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 Processor -- start 2024-04-21T08:46:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 -- start start 2024-04-21T08:46:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce09d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdfd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdfdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701ad1f0 con 0x55691cdfe0e0 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701afa50 con 0x55691cdf4b60 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e78a54740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701ad840 con 0x55691cdf58d0 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdfdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdfdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48900/0 (socket says 172.21.15.57:48900) 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 -- 172.21.15.57:0/2283367931 learned_addr learned my addr 172.21.15.57:0/2283367931 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce09d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdfd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 -- 172.21.15.57:0/2283367931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 msgr2=0x55691cdfd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdfd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 -- 172.21.15.57:0/2283367931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 msgr2=0x55691ce09d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce09d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.350+0000 7f7e66ffd640 1 -- 172.21.15.57:0/2283367931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55691cdf3e30 con 0x55691cdfe0e0 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdfdaf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009230 tx=0x7f7e58005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294a7505096916c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e657fa640 1 -- 172.21.15.57:0/2283367931 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5802c8e0 con 0x55691cdfe0e0 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e657fa640 1 -- 172.21.15.57:0/2283367931 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e580054f0 con 0x55691cdfe0e0 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e657fa640 1 -- 172.21.15.57:0/2283367931 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5804d4e0 con 0x55691cdfe0e0 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 -- 172.21.15.57:0/2283367931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 msgr2=0x55691cdfdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdfdaf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009230 tx=0x7f7e58005a20 comp rx=0 tx=0).stop 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 -- 172.21.15.57:0/2283367931 shutdown_connections 2024-04-21T08:46:53.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdfdaf0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdfd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 --2- 172.21.15.57:0/2283367931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce09d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 -- 172.21.15.57:0/2283367931 >> 172.21.15.57:0/2283367931 conn(0x55691cdbbc90 msgr2=0x55691cdf6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 -- 172.21.15.57:0/2283367931 shutdown_connections 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.351+0000 7f7e78a54740 1 -- 172.21.15.57:0/2283367931 wait complete. 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 -- start start 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 Processor -- start 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce06410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdf8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdf8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701adf10 con 0x55691cdfe0e0 2024-04-21T08:46:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701af380 con 0x55691cdf4b60 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7e701ae5e0 con 0x55691cdf58d0 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e78a54740 10 client.0 _ll_register_callbacks cb 0x7f7e58005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdf8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdf8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48902/0 (socket says 172.21.15.57:48902) 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e66ffd640 1 -- 172.21.15.57:0/3611747003 learned_addr learned my addr 172.21.15.57:0/3611747003 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdf8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.352+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce06410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e66ffd640 1 -- 172.21.15.57:0/3611747003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 msgr2=0x55691cdf8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdf8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e66ffd640 1 -- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 msgr2=0x55691ce06410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce06410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e66ffd640 1 -- 172.21.15.57:0/3611747003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55691cdf8bb0 con 0x55691cdfe0e0 2024-04-21T08:46:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e66ffd640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdf8640 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7e580057c0 tx=0x7f7e5804d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103e791baa0866c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e4effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e4effd640 1 -- 172.21.15.57:0/3611747003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5802c860 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e4effd640 1 -- 172.21.15.57:0/3611747003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f7e5802ca00 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e580054f0 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e4effd640 1 -- 172.21.15.57:0/3611747003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7e5802cc00 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e78a54740 10 client.16038 fetch_fsmap learned FSMap version 21 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55691cdff430 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e4effd640 1 -- 172.21.15.57:0/3611747003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e580054f0 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.353+0000 7f7e4effd640 1 -- 172.21.15.57:0/3611747003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7e58040040 con 0x55691cdfe0e0 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e78a54740 10 client.16038 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e78a54740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e78a54740 20 client.16038 populate_metadata read hostname 'smithi057' 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e78a54740 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce74780 0x55691ce76ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:46:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e78a54740 4 client.16038 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:46:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55691ccf93d0 con 0x55691ce74780 2024-04-21T08:46:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce74780 0x55691ce76ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:46:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e6f7fe640 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce74780 0x55691ce76ba0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fd057ee2f398d43 server_cookie=cc589f34eb19866b in_seq=0 out_seq=0 2024-04-21T08:46:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:53.354+0000 7f7e4effd640 10 client.16038 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:54.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:54 smithi129 ceph-mon[31272]: pgmap v47: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.0 MiB/s wr, 964 op/s 2024-04-21T08:46:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:54 smithi057 ceph-mon[27968]: pgmap v47: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.0 MiB/s wr, 964 op/s 2024-04-21T08:46:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:54 smithi136 ceph-mon[31791]: pgmap v47: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.0 MiB/s wr, 964 op/s 2024-04-21T08:46:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:56 smithi057 ceph-mon[27968]: pgmap v48: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.0 MiB/s wr, 756 op/s 2024-04-21T08:46:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:56 smithi136 ceph-mon[31791]: pgmap v48: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.0 MiB/s wr, 756 op/s 2024-04-21T08:46:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:56 smithi129 ceph-mon[31272]: pgmap v48: 129 pgs: 129 active+clean; 194 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 5.0 MiB/s wr, 756 op/s 2024-04-21T08:46:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.368+0000 7f7e6d7fa640 1 -- 172.21.15.57:0/2340504799 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7e68099100 con 0x7f7e7019cfd0 2024-04-21T08:46:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e4effd640 1 -- 172.21.15.57:0/3611747003 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55691ccf93d0 con 0x55691ce74780 2024-04-21T08:46:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e4effd640 10 client.16038 handle_command_reply: tid=0 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 client.16038 shutdown 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 msgr2=0x55691cdf8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdf8640 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7e580057c0 tx=0x7f7e5804d2e0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 shutdown_connections 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55691ce74780 0x55691ce76ba0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55691cdfe0e0 0x55691cdf8640 secure :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7e580057c0 tx=0x7f7e5804d2e0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55691cdf58d0 0x55691cdf8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e4effd640 0 client.16038 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 --2- 172.21.15.57:0/3611747003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55691cdf4b60 0x55691ce06410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 >> 172.21.15.57:0/3611747003 conn(0x55691cdbbc90 msgr2=0x55691cdf96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 shutdown_connections 2024-04-21T08:46:57.842 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.838+0000 7f7e78a54740 1 -- 172.21.15.57:0/3611747003 wait complete. 2024-04-21T08:46:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e78a54740 20 client.16038 trim_cache size 0 max 16384 2024-04-21T08:46:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:46:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:46:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:46:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:46:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7e5c0a3ee0 msgr2=0x7f7e5c0a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7e5c0a3ee0 0x7f7e5c0a63a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7e680ce370 tx=0x7f7e680bd040 comp rx=0 tx=0).stop 2024-04-21T08:46:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 msgr2=0x7f7e7019d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:46:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 0x7f7e7019d480 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800cab0 tx=0x7f7e6800cf70 comp rx=0 tx=0).stop 2024-04-21T08:46:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 shutdown_connections 2024-04-21T08:46:57.846 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7e5c0a3ee0 0x7f7e5c0a63a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7e7019cfd0 0x7f7e7019d480 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e701072a0 0x7f7e70198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 --2- 172.21.15.57:0/2340504799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e701021d0 0x7f7e70198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:46:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.839+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 >> 172.21.15.57:0/2340504799 conn(0x7f7e700793e0 msgr2=0x7f7e700ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:46:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.840+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 shutdown_connections 2024-04-21T08:46:57.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:46:57.840+0000 7f7e75e5c640 1 -- 172.21.15.57:0/2340504799 wait complete. 2024-04-21T08:46:58.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:58 smithi129 ceph-mon[31272]: pgmap v49: 129 pgs: 129 active+clean; 214 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.7 MiB/s wr, 891 op/s 2024-04-21T08:46:58.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:46:58 smithi129 ceph-mon[31272]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:46:58.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:58 smithi136 ceph-mon[31791]: pgmap v49: 129 pgs: 129 active+clean; 214 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.7 MiB/s wr, 891 op/s 2024-04-21T08:46:58.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:46:58 smithi136 ceph-mon[31791]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:46:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:58 smithi057 ceph-mon[27968]: pgmap v49: 129 pgs: 129 active+clean; 214 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.7 MiB/s wr, 891 op/s 2024-04-21T08:46:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:46:58 smithi057 ceph-mon[27968]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-21T08:47:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:00 smithi136 ceph-mon[31791]: pgmap v50: 129 pgs: 129 active+clean; 214 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 884 op/s 2024-04-21T08:47:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:00 smithi129 ceph-mon[31272]: pgmap v50: 129 pgs: 129 active+clean; 214 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 884 op/s 2024-04-21T08:47:00.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:00 smithi057 ceph-mon[27968]: pgmap v50: 129 pgs: 129 active+clean; 214 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 884 op/s 2024-04-21T08:47:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:02 smithi136 ceph-mon[31791]: pgmap v51: 129 pgs: 129 active+clean; 200 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 1.15k op/s 2024-04-21T08:47:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:02 smithi129 ceph-mon[31272]: pgmap v51: 129 pgs: 129 active+clean; 200 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 1.15k op/s 2024-04-21T08:47:02.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:02 smithi057 ceph-mon[27968]: pgmap v51: 129 pgs: 129 active+clean; 200 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 6.4 MiB/s wr, 1.15k op/s 2024-04-21T08:47:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:04 smithi136 ceph-mon[31791]: pgmap v52: 129 pgs: 129 active+clean; 188 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 1.60k op/s 2024-04-21T08:47:04.685 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:04.685 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:04.711 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:04 smithi057 ceph-mon[27968]: pgmap v52: 129 pgs: 129 active+clean; 188 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 1.60k op/s 2024-04-21T08:47:04.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.802+0000 7f3304bb6640 1 -- 172.21.15.57:0/3208367088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3300107270 msgr2=0x7f330010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.802+0000 7f3304bb6640 1 --2- 172.21.15.57:0/3208367088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3300107270 0x7f330010ee40 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f32f8009ca0 tx=0x7f32f802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 -- 172.21.15.57:0/3208367088 shutdown_connections 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 --2- 172.21.15.57:0/3208367088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3300107270 0x7f330010ee40 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 --2- 172.21.15.57:0/3208367088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3300102b50 0x7f3300106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 --2- 172.21.15.57:0/3208367088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33001021a0 0x7f3300102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 -- 172.21.15.57:0/3208367088 >> 172.21.15.57:0/3208367088 conn(0x7f33000fda40 msgr2=0x7f33000ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 -- 172.21.15.57:0/3208367088 shutdown_connections 2024-04-21T08:47:04.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 -- 172.21.15.57:0/3208367088 wait complete. 2024-04-21T08:47:04.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.803+0000 7f3304bb6640 1 Processor -- start 2024-04-21T08:47:04.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 -- start start 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 0x7f330019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3300107270 0x7f330019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3300198630 0x7f3300198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f33001121e0 con 0x7f33001021a0 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3300112060 con 0x7f3300107270 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f3304bb6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3300112360 con 0x7f3300198630 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 0x7f330019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 0x7f330019e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49800/0 (socket says 172.21.15.57:49800) 2024-04-21T08:47:04.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 -- 172.21.15.57:0/2107464348 learned_addr learned my addr 172.21.15.57:0/2107464348 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3300107270 0x7f330019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fed9d640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3300198630 0x7f3300198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 -- 172.21.15.57:0/2107464348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3300198630 msgr2=0x7f3300198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3300198630 0x7f3300198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 -- 172.21.15.57:0/2107464348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3300107270 msgr2=0x7f330019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3300107270 0x7f330019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 -- 172.21.15.57:0/2107464348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3300199150 con 0x7f33001021a0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.804+0000 7f32fe59c640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 0x7f330019e430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f32f000d770 tx=0x7f32f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.805+0000 7f32f77fe640 1 -- 172.21.15.57:0/2107464348 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0017070 con 0x7f33001021a0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.805+0000 7f32f77fe640 1 -- 172.21.15.57:0/2107464348 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32f00054d0 con 0x7f33001021a0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.805+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3300199440 con 0x7f33001021a0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.805+0000 7f32f77fe640 1 -- 172.21.15.57:0/2107464348 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f0004c30 con 0x7f33001021a0 2024-04-21T08:47:04.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.805+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33001a55f0 con 0x7f33001021a0 2024-04-21T08:47:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f32f77fe640 1 -- 172.21.15.57:0/2107464348 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f32f0004e10 con 0x7f33001021a0 2024-04-21T08:47:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 Processor -- start 2024-04-21T08:47:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 -- start start 2024-04-21T08:47:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f963690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f963bd0 0x55bb4f970950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f973280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55bb4f8c81e0 con 0x55bb4f970e90 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bb4f865390 con 0x55bb4f963bd0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f33077af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55bb4f8c41a0 con 0x55bb4f9632b0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f32f77fe640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32ec0a3d20 0x7f32ec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.808+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f973280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f973280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49816/0 (socket says 172.21.15.57:49816) 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1187334116 learned_addr learned my addr 172.21.15.57:0/1187334116 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fed9d640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32ec0a3d20 0x7f32ec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fe59c640 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f963bd0 0x55bb4f970950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fed9d640 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f963690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32f77fe640 1 -- 172.21.15.57:0/2107464348 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f32f00d05e0 con 0x7f33001021a0 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1187334116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 msgr2=0x55bb4f963690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f963690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1187334116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f963bd0 msgr2=0x55bb4f970950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f963bd0 0x55bb4f970950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1187334116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb4f6a47d0 con 0x55bb4f970e90 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f973280 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f32e800bbb0 tx=0x7f32e80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8bb2dd7cead486 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32fed9d640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32ec0a3d20 0x7f32ec0a61e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f32f8004af0 tx=0x7f32f802e040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32f4ff9640 1 -- 172.21.15.57:0/1187334116 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8004030 con 0x55bb4f970e90 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32f4ff9640 1 -- 172.21.15.57:0/1187334116 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e8005020 con 0x55bb4f970e90 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.809+0000 7f32f4ff9640 1 -- 172.21.15.57:0/1187334116 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8004b40 con 0x55bb4f970e90 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 -- 172.21.15.57:0/1187334116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 msgr2=0x55bb4f973280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f973280 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f32e800bbb0 tx=0x7f32e80107a0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 -- 172.21.15.57:0/1187334116 shutdown_connections 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f973280 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f963bd0 0x55bb4f970950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 --2- 172.21.15.57:0/1187334116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f963690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 -- 172.21.15.57:0/1187334116 >> 172.21.15.57:0/1187334116 conn(0x55bb4f95dcb0 msgr2=0x55bb4f95e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 -- 172.21.15.57:0/1187334116 shutdown_connections 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.810+0000 7f33077af740 1 -- 172.21.15.57:0/1187334116 wait complete. 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 -- start start 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 Processor -- start 2024-04-21T08:47:04.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f9632b0 0x55bb4f96deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f96e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55bb4f973c50 con 0x55bb4f970e90 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bb4f8812f0 con 0x55bb4f9632b0 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55bb4f973dd0 con 0x55bb4f963bd0 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f33077af740 10 client.0 _ll_register_callbacks cb 0x55bb4f88e580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f96e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f96e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49826/0 (socket says 172.21.15.57:49826) 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 -- 172.21.15.57:0/3735983807 learned_addr learned my addr 172.21.15.57:0/3735983807 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fe59c640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f9632b0 0x55bb4f96deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 -- 172.21.15.57:0/3735983807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 msgr2=0x55bb4f96e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 -- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f9632b0 msgr2=0x55bb4f96deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f9632b0 0x55bb4f96deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fed9d640 1 -- 172.21.15.57:0/3735983807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb4f5da3f0 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.811+0000 7f32fe59c640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f9632b0 0x55bb4f96deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32fed9d640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f96e930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f32f8036000 tx=0x7f32f802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25415d2b37c5e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32e5ffb640 1 -- 172.21.15.57:0/3735983807 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8054060 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32e5ffb640 1 -- 172.21.15.57:0/3735983807 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f32f8032bc0 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e8005020 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32e5ffb640 1 -- 172.21.15.57:0/3735983807 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f32f805b050 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 10 client.16050 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb4f59a0f0 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32e5ffb640 1 -- 172.21.15.57:0/3735983807 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e8005020 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f32e5ffb640 1 -- 172.21.15.57:0/3735983807 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f32f804f740 con 0x55bb4f970e90 2024-04-21T08:47:04.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 10 client.16050 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 20 client.16050 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f980e70 0x55bb4f9832c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 4 client.16050 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.812+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb4f8d5c00 con 0x55bb4f980e70 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.813+0000 7f32fe59c640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f980e70 0x55bb4f9832c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.813+0000 7f32e5ffb640 10 client.16050 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.813+0000 7f32fe59c640 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f980e70 0x55bb4f9832c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3219f270c2909e5b server_cookie=2ef7a23a4b9987e3 in_seq=0 out_seq=0 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f32e5ffb640 1 -- 172.21.15.57:0/3735983807 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb4f8d5c00 con 0x55bb4f980e70 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f32e5ffb640 10 client.16050 handle_command_reply: tid=0 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 client.16050 shutdown 2024-04-21T08:47:04.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 msgr2=0x55bb4f96e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f96e930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f32f8036000 tx=0x7f32f802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 shutdown_connections 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f980e70 0x55bb4f9832c0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f970e90 0x55bb4f96e930 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 --2- 172.21.15.57:0/3735983807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f9632b0 0x55bb4f96deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f32e5ffb640 0 client.16050 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 >> 172.21.15.57:0/3735983807 conn(0x55bb4f95dcb0 msgr2=0x55bb4f95fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 shutdown_connections 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 1 -- 172.21.15.57:0/3735983807 wait complete. 2024-04-21T08:47:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.814+0000 7f33077af740 20 client.16050 trim_cache size 0 max 16384 2024-04-21T08:47:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:04 smithi129 ceph-mon[31272]: pgmap v52: 129 pgs: 129 active+clean; 188 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 1.60k op/s 2024-04-21T08:47:04.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 Processor -- start 2024-04-21T08:47:04.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 -- start start 2024-04-21T08:47:04.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f9705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 0x55bb4f96cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f97cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f33001ab060 con 0x55bb4f963bd0 2024-04-21T08:47:04.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f33001ad8c0 con 0x55bb4f970e90 2024-04-21T08:47:04.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f33077af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f33001ab6b0 con 0x55bb4f9632b0 2024-04-21T08:47:04.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 0x55bb4f96cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 0x55bb4f96cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49828/0 (socket says 172.21.15.57:49828) 2024-04-21T08:47:04.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1342579563 learned_addr learned my addr 172.21.15.57:0/1342579563 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:04.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fe59c640 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f9705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1342579563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 msgr2=0x55bb4f9705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f9705e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1342579563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 msgr2=0x55bb4f97cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f97cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 -- 172.21.15.57:0/1342579563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb4f973280 con 0x55bb4f963bd0 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fe59c640 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f9705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.832+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 0x55bb4f96cdf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x55bb4f95ee40 tx=0x7f32e800b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542aabf42a54ed76 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f32e7fff640 1 -- 172.21.15.57:0/1342579563 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e80050b0 con 0x55bb4f963bd0 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f32e7fff640 1 -- 172.21.15.57:0/1342579563 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e80035f0 con 0x55bb4f963bd0 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f32e7fff640 1 -- 172.21.15.57:0/1342579563 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e80116a0 con 0x55bb4f963bd0 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 -- 172.21.15.57:0/1342579563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 msgr2=0x55bb4f96cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 0x55bb4f96cdf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x55bb4f95ee40 tx=0x7f32e800b820 comp rx=0 tx=0).stop 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 -- 172.21.15.57:0/1342579563 shutdown_connections 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f97cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f963bd0 0x55bb4f96cdf0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 --2- 172.21.15.57:0/1342579563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f9632b0 0x55bb4f9705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 -- 172.21.15.57:0/1342579563 >> 172.21.15.57:0/1342579563 conn(0x55bb4f95dcb0 msgr2=0x55bb4f95cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 -- 172.21.15.57:0/1342579563 shutdown_connections 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 -- 172.21.15.57:0/1342579563 wait complete. 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.833+0000 7f33077af740 1 -- start start 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 Processor -- start 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 0x55bb4f96b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f9e6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f33001abd80 con 0x55bb4f9632b0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f33001ad1f0 con 0x55bb4f970e90 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f33001ac450 con 0x55bb4f963bd0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f33077af740 10 client.0 _ll_register_callbacks cb 0x7f32f80328a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 0x55bb4f96b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 0x55bb4f96b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49842/0 (socket says 172.21.15.57:49842) 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 -- 172.21.15.57:0/966824461 learned_addr learned my addr 172.21.15.57:0/966824461 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:04.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fed9d640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f9e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 -- 172.21.15.57:0/966824461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 msgr2=0x55bb4f96b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 -- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 msgr2=0x55bb4f9e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f9e6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 -- 172.21.15.57:0/966824461 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb4f9e6940 con 0x55bb4f9632b0 2024-04-21T08:47:04.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fed9d640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f9e6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32fe59c640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 0x55bb4f96b220 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f32f009c200 tx=0x7f32f001fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5ed8f1706efd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.834+0000 7f32e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f32e5ffb640 1 -- 172.21.15.57:0/966824461 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f00a0570 con 0x55bb4f9632b0 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f32e5ffb640 1 -- 172.21.15.57:0/966824461 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f32f00988e0 con 0x55bb4f9632b0 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e80035f0 con 0x55bb4f9632b0 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f32e5ffb640 1 -- 172.21.15.57:0/966824461 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f32f00a4e30 con 0x55bb4f9632b0 2024-04-21T08:47:04.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 10 client.16056 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb4f9e7b90 con 0x55bb4f9632b0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f32e5ffb640 1 -- 172.21.15.57:0/966824461 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32e80035f0 con 0x55bb4f9632b0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f32e5ffb640 1 -- 172.21.15.57:0/966824461 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f32f00a0120 con 0x55bb4f9632b0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 10 client.16056 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 20 client.16056 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f9e82c0 0x55bb4f9ea6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 4 client.16056 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bb4f8cfa70 con 0x55bb4f9e82c0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.835+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f9e82c0 0x55bb4f9ea6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f32fdd9b640 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f9e82c0 0x55bb4f9ea6e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7027d3254c90f4a1 server_cookie=dcee1d8bb5f6a194 in_seq=0 out_seq=0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f32e5ffb640 10 client.16056 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f32e5ffb640 1 -- 172.21.15.57:0/966824461 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bb4f8cfa70 con 0x55bb4f9e82c0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f32e5ffb640 10 client.16056 handle_command_reply: tid=0 2024-04-21T08:47:04.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 client.16056 shutdown 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 msgr2=0x55bb4f96b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 0x55bb4f96b220 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f32f009c200 tx=0x7f32f001fe80 comp rx=0 tx=0).stop 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 shutdown_connections 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb4f9e82c0 0x55bb4f9ea6e0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb4f970e90 0x55bb4f9e6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f32e5ffb640 0 client.16056 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb4f963bd0 0x55bb4f96b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 --2- 172.21.15.57:0/966824461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb4f9632b0 0x55bb4f96b220 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 >> 172.21.15.57:0/966824461 conn(0x55bb4f95dcb0 msgr2=0x55bb4f96de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 shutdown_connections 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 1 -- 172.21.15.57:0/966824461 wait complete. 2024-04-21T08:47:04.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.836+0000 7f33077af740 20 client.16056 trim_cache size 0 max 16384 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:47:04.853 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:47:04.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32ec0a3d20 msgr2=0x7f32ec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32ec0a3d20 0x7f32ec0a61e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f32f8004af0 tx=0x7f32f802e040 comp rx=0 tx=0).stop 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 msgr2=0x7f330019e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 0x7f330019e430 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f32f000d770 tx=0x7f32f000dc30 comp rx=0 tx=0).stop 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 shutdown_connections 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32ec0a3d20 0x7f32ec0a61e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3300198630 0x7f3300198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3300107270 0x7f330019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 --2- 172.21.15.57:0/2107464348 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33001021a0 0x7f330019e430 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 >> 172.21.15.57:0/2107464348 conn(0x7f33000fda40 msgr2=0x7f33000ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 shutdown_connections 2024-04-21T08:47:04.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.837+0000 7f3304bb6640 1 -- 172.21.15.57:0/2107464348 wait complete. 2024-04-21T08:47:04.856 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:47:04.856 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 -- 172.21.15.57:0/2904110536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074b20 msgr2=0x7f602410e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 --2- 172.21.15.57:0/2904110536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074b20 0x7f602410e320 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f6018009900 tx=0x7f601802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 -- 172.21.15.57:0/2904110536 shutdown_connections 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 --2- 172.21.15.57:0/2904110536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 0x7f6024110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 --2- 172.21.15.57:0/2904110536 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074b20 0x7f602410e320 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 --2- 172.21.15.57:0/2904110536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6024074200 0x7f60240745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 -- 172.21.15.57:0/2904110536 >> 172.21.15.57:0/2904110536 conn(0x7f60240793a0 msgr2=0x7f60240797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 -- 172.21.15.57:0/2904110536 shutdown_connections 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.998+0000 7f6029f32640 1 -- 172.21.15.57:0/2904110536 wait complete. 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 Processor -- start 2024-04-21T08:47:05.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 -- start start 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 0x7f60241a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6024074b20 0x7f60241a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 0x7f602419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f60241143e0 con 0x7f6024074200 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6024114260 con 0x7f602410e860 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6029f32640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6024114560 con 0x7f6024074b20 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 0x7f60241a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 0x7f60241a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49848/0 (socket says 172.21.15.57:49848) 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 -- 172.21.15.57:0/2050918595 learned_addr learned my addr 172.21.15.57:0/2050918595 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6023fff640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 0x7f602419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 -- 172.21.15.57:0/2050918595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6024074b20 msgr2=0x7f60241a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6024074b20 0x7f60241a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 -- 172.21.15.57:0/2050918595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 msgr2=0x7f602419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 0x7f602419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f60237fe640 1 -- 172.21.15.57:0/2050918595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602419af90 con 0x7f6024074200 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:04.999+0000 7f6023fff640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 0x7f602419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.000+0000 7f60237fe640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 0x7f60241a0650 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f601400c890 tx=0x7f601400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.000+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014004220 con 0x7f6024074200 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.000+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60140043c0 con 0x7f6024074200 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.000+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602419b280 con 0x7f6024074200 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.000+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6014004a00 con 0x7f6024074200 2024-04-21T08:47:05.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.000+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60241a74a0 con 0x7f6024074200 2024-04-21T08:47:05.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.001+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6024075940 con 0x7f6024074200 2024-04-21T08:47:05.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.002+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f60140079b0 con 0x7f6024074200 2024-04-21T08:47:05.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.003+0000 7f6020ff9640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f600c0a3da0 0x7f600c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.003+0000 7f6022ffd640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f600c0a3da0 0x7f600c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.003+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f60140d0940 con 0x7f6024074200 2024-04-21T08:47:05.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.003+0000 7f6022ffd640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f600c0a3da0 0x7f600c0a6260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60180096c0 tx=0x7f6018000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.007+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6014014030 con 0x7f6024074200 2024-04-21T08:47:05.233 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.234+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6024069230 con 0x7f6024074200 2024-04-21T08:47:05.234 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.235+0000 7f6020ff9640 1 -- 172.21.15.57:0/2050918595 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f601409cc20 con 0x7f6024074200 2024-04-21T08:47:05.234 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:47:05.235 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:47:05.235 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:47:05.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f600c0a3da0 msgr2=0x7f600c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f600c0a3da0 0x7f600c0a6260 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60180096c0 tx=0x7f6018000bc0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 msgr2=0x7f60241a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 0x7f60241a0650 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f601400c890 tx=0x7f601400cd50 comp rx=0 tx=0).stop 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 shutdown_connections 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f600c0a3da0 0x7f600c0a6260 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f602410e860 0x7f602419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6024074b20 0x7f60241a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 --2- 172.21.15.57:0/2050918595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6024074200 0x7f60241a0650 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 >> 172.21.15.57:0/2050918595 conn(0x7f60240793a0 msgr2=0x7f60240fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 shutdown_connections 2024-04-21T08:47:05.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.237+0000 7f6029f32640 1 -- 172.21.15.57:0/2050918595 wait complete. 2024-04-21T08:47:05.247 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:47:05.353 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 -- 172.21.15.57:0/276991583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b4107270 msgr2=0x7f47b410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 --2- 172.21.15.57:0/276991583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b4107270 0x7f47b410ee40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009ca0 tx=0x7f47ac02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 -- 172.21.15.57:0/276991583 shutdown_connections 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 --2- 172.21.15.57:0/276991583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b4107270 0x7f47b410ee40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 --2- 172.21.15.57:0/276991583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b406b1f0 0x7f47b4106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 --2- 172.21.15.57:0/276991583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b406a840 0x7f47b406ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 -- 172.21.15.57:0/276991583 >> 172.21.15.57:0/276991583 conn(0x7f47b4077eb0 msgr2=0x7f47b40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 -- 172.21.15.57:0/276991583 shutdown_connections 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.352+0000 7f47bc006640 1 -- 172.21.15.57:0/276991583 wait complete. 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 Processor -- start 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 -- start start 2024-04-21T08:47:05.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 0x7f47b4100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b4107270 0x7f47b41013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b4102b20 0x7f47b4101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41121d0 con 0x7f47b406a840 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47b4112050 con 0x7f47b4107270 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47bc006640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f47b4112350 con 0x7f47b4102b20 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 0x7f47b4100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 0x7f47b4100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49868/0 (socket says 172.21.15.57:49868) 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 -- 172.21.15.57:0/1732625563 learned_addr learned my addr 172.21.15.57:0/1732625563 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b95a1640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b4107270 0x7f47b41013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b4102b20 0x7f47b4101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 -- 172.21.15.57:0/1732625563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b4102b20 msgr2=0x7f47b4101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b4102b20 0x7f47b4101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 -- 172.21.15.57:0/1732625563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b4107270 msgr2=0x7f47b41013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b4107270 0x7f47b41013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 -- 172.21.15.57:0/1732625563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b41ab380 con 0x7f47b406a840 2024-04-21T08:47:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.353+0000 7f47b9da2640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 0x7f47b4100e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f47b000d770 tx=0x7f47b000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.354+0000 7f47aaffd640 1 -- 172.21.15.57:0/1732625563 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0017070 con 0x7f47b406a840 2024-04-21T08:47:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.354+0000 7f47aaffd640 1 -- 172.21.15.57:0/1732625563 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47b00054d0 con 0x7f47b406a840 2024-04-21T08:47:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.354+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b41ab670 con 0x7f47b406a840 2024-04-21T08:47:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.354+0000 7f47aaffd640 1 -- 172.21.15.57:0/1732625563 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0004c30 con 0x7f47b406a840 2024-04-21T08:47:05.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.354+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b41abb70 con 0x7f47b406a840 2024-04-21T08:47:05.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47aaffd640 1 -- 172.21.15.57:0/1732625563 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47b0004e10 con 0x7f47b406a840 2024-04-21T08:47:05.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 Processor -- start 2024-04-21T08:47:05.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 -- start start 2024-04-21T08:47:05.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c3f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c48430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c4ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e779ba41f0 con 0x55e779c3f4d0 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e779b414b0 con 0x55e779c3fe80 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47bebf8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e779ba01b0 con 0x55e779c48a00 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47ba5a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c3f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47ba5a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c3f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49870/0 (socket says 172.21.15.57:49870) 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47ba5a3640 1 -- 172.21.15.57:0/1848202201 learned_addr learned my addr 172.21.15.57:0/1848202201 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47b9da2640 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c48430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47aaffd640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47a00a3d20 0x7f47a00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.357+0000 7f47b95a1640 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c4ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 -- 172.21.15.57:0/1848202201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 msgr2=0x55e779c4ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c4ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 -- 172.21.15.57:0/1848202201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 msgr2=0x55e779c48430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c48430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 -- 172.21.15.57:0/1848202201 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e779b331b0 con 0x55e779c3f4d0 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47b95a1640 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c4ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47aaffd640 1 -- 172.21.15.57:0/1732625563 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f47b00d06e0 con 0x7f47b406a840 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47a00a3d20 0x7f47a00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c3f8b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f47ac037040 tx=0x7f47ac009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86487d1868fa07de server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.358+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47a00a3d20 0x7f47a00a61e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f47ac04b000 tx=0x7f47ac044000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.360+0000 7f479ffff640 1 -- 172.21.15.57:0/1848202201 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac039040 con 0x55e779c3f4d0 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.360+0000 7f479ffff640 1 -- 172.21.15.57:0/1848202201 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47ac0348f0 con 0x55e779c3f4d0 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.360+0000 7f479ffff640 1 -- 172.21.15.57:0/1848202201 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac04b620 con 0x55e779c3f4d0 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 -- 172.21.15.57:0/1848202201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 msgr2=0x55e779c3f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c3f8b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f47ac037040 tx=0x7f47ac009a00 comp rx=0 tx=0).stop 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 -- 172.21.15.57:0/1848202201 shutdown_connections 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c4ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c48430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1848202201 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c3f8b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 -- 172.21.15.57:0/1848202201 >> 172.21.15.57:0/1848202201 conn(0x55e779c39ba0 msgr2=0x55e779c39f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 -- 172.21.15.57:0/1848202201 shutdown_connections 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.361+0000 7f47bebf8740 1 -- 172.21.15.57:0/1848202201 wait complete. 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 -- start start 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 Processor -- start 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3f4d0 0x55e779c48350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3fe80 0x55e779c456e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 0x55e779c45cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e779c4b970 con 0x55e779c48a00 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e779b5d500 con 0x55e779c3f4d0 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e779c4baf0 con 0x55e779c3fe80 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47bebf8740 10 client.0 _ll_register_callbacks cb 0x55e779b2bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47ba5a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 0x55e779c45cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47ba5a3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 0x55e779c45cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49884/0 (socket says 172.21.15.57:49884) 2024-04-21T08:47:05.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47ba5a3640 1 -- 172.21.15.57:0/774654597 learned_addr learned my addr 172.21.15.57:0/774654597 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:05.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47b9da2640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3f4d0 0x55e779c48350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.362+0000 7f47b95a1640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3fe80 0x55e779c456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47ba5a3640 1 -- 172.21.15.57:0/774654597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3fe80 msgr2=0x55e779c456e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3fe80 0x55e779c456e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47ba5a3640 1 -- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3f4d0 msgr2=0x55e779c48350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3f4d0 0x55e779c48350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47ba5a3640 1 -- 172.21.15.57:0/774654597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7798b63f0 con 0x55e779c48a00 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47b9da2640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3f4d0 0x55e779c48350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 0x55e779c45cf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47ac02c130 tx=0x7f47ac009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79de772e1180d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.363+0000 7f479d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:05.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f479d7fa640 1 -- 172.21.15.57:0/774654597 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac034de0 con 0x55e779c48a00 2024-04-21T08:47:05.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f479d7fa640 1 -- 172.21.15.57:0/774654597 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f47ac003970 con 0x55e779c48a00 2024-04-21T08:47:05.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47ac0348f0 con 0x55e779c48a00 2024-04-21T08:47:05.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f479d7fa640 1 -- 172.21.15.57:0/774654597 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f47ac061050 con 0x55e779c48a00 2024-04-21T08:47:05.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 10 client.16074 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e7799807d0 con 0x55e779c48a00 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f479d7fa640 1 -- 172.21.15.57:0/774654597 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47ac0348f0 con 0x55e779c48a00 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f479d7fa640 1 -- 172.21.15.57:0/774654597 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f47ac03f070 con 0x55e779c48a00 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 10 client.16074 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 20 client.16074 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779c58ae0 0x55e779c5aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 4 client.16074 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.367+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e779b43930 con 0x55e779c58ae0 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.368+0000 7f47b9da2640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779c58ae0 0x55e779c5aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.368+0000 7f479d7fa640 10 client.16074 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.368+0000 7f47b9da2640 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779c58ae0 0x55e779c5aed0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bce7239ce20c7157 server_cookie=6b4b18d8716dae62 in_seq=0 out_seq=0 2024-04-21T08:47:05.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f479d7fa640 1 -- 172.21.15.57:0/774654597 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e779b43930 con 0x55e779c58ae0 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f479d7fa640 10 client.16074 handle_command_reply: tid=0 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 client.16074 shutdown 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 msgr2=0x55e779c45cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 0x55e779c45cf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f47ac02c130 tx=0x7f47ac009560 comp rx=0 tx=0).stop 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 shutdown_connections 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779c58ae0 0x55e779c5aed0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c48a00 0x55e779c45cf0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f479d7fa640 0 client.16074 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3fe80 0x55e779c456e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 --2- 172.21.15.57:0/774654597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3f4d0 0x55e779c48350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 >> 172.21.15.57:0/774654597 conn(0x55e779c39ba0 msgr2=0x55e779c3b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 shutdown_connections 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 1 -- 172.21.15.57:0/774654597 wait complete. 2024-04-21T08:47:05.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.369+0000 7f47bebf8740 20 client.16074 trim_cache size 0 max 16384 2024-04-21T08:47:05.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.382+0000 7f47bebf8740 1 Processor -- start 2024-04-21T08:47:05.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 -- start start 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3f4d0 0x55e779c54830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 0x55e779c40af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c48a00 0x55e779c604d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b3760 con 0x55e779c3fe80 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b5fc0 con 0x55e779c48a00 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47bebf8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b3db0 con 0x55e779c3f4d0 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 0x55e779c40af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 0x55e779c40af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49890/0 (socket says 172.21.15.57:49890) 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 -- 172.21.15.57:0/1144994112 learned_addr learned my addr 172.21.15.57:0/1144994112 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 -- 172.21.15.57:0/1144994112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3f4d0 msgr2=0x55e779c54830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3f4d0 0x55e779c54830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 -- 172.21.15.57:0/1144994112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c48a00 msgr2=0x55e779c604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c48a00 0x55e779c604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 -- 172.21.15.57:0/1144994112 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e779c3a4e0 con 0x55e779c3fe80 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f47b95a1640 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 0x55e779c40af0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f47b419c7e0 tx=0x7f47a4011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dac5289c20cd9fb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.388 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.383+0000 7f479f7fe640 1 -- 172.21.15.57:0/1144994112 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a4011e30 con 0x55e779c3fe80 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f479f7fe640 1 -- 172.21.15.57:0/1144994112 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a40035f0 con 0x55e779c3fe80 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f479f7fe640 1 -- 172.21.15.57:0/1144994112 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a401b6d0 con 0x55e779c3fe80 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 -- 172.21.15.57:0/1144994112 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 msgr2=0x55e779c40af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 0x55e779c40af0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f47b419c7e0 tx=0x7f47a4011440 comp rx=0 tx=0).stop 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 -- 172.21.15.57:0/1144994112 shutdown_connections 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c48a00 0x55e779c604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3fe80 0x55e779c40af0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 --2- 172.21.15.57:0/1144994112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c3f4d0 0x55e779c54830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 -- 172.21.15.57:0/1144994112 >> 172.21.15.57:0/1144994112 conn(0x55e779c39ba0 msgr2=0x55e779c3b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 -- 172.21.15.57:0/1144994112 shutdown_connections 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 -- 172.21.15.57:0/1144994112 wait complete. 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 -- start start 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.384+0000 7f47bebf8740 1 Processor -- start 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c51010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c41fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c424f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b4480 con 0x55e779c3f4d0 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b58f0 con 0x55e779c3fe80 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f47b41b4b50 con 0x55e779c48a00 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47bebf8740 10 client.0 _ll_register_callbacks cb 0x7f47ac005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c51010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c51010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49892/0 (socket says 172.21.15.57:49892) 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 -- 172.21.15.57:0/3002107667 learned_addr learned my addr 172.21.15.57:0/3002107667 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b95a1640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c41fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47ba5a3640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 -- 172.21.15.57:0/3002107667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 msgr2=0x55e779c424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 -- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 msgr2=0x55e779c41fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c41fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 -- 172.21.15.57:0/3002107667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e779c42bb0 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f47b9da2640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c51010 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009f40 tx=0x7f47b009bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840d9d45a36c5996 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.385+0000 7f479d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f479d7fa640 1 -- 172.21.15.57:0/3002107667 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b0020040 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f479d7fa640 1 -- 172.21.15.57:0/3002107667 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f47b00cdd50 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47a40035f0 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f479d7fa640 1 -- 172.21.15.57:0/3002107667 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f47b00a06d0 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 10 client.16080 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e779cbec70 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f479d7fa640 1 -- 172.21.15.57:0/3002107667 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a40035f0 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f479d7fa640 1 -- 172.21.15.57:0/3002107667 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f47b00a4b00 con 0x55e779c3f4d0 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 10 client.16080 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 10 client.16080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 20 client.16080 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779cbf390 0x55e779cc17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 4 client.16080 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:05.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e779bb1c10 con 0x55e779cbf390 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.386+0000 7f47b95a1640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779cbf390 0x55e779cc17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f479d7fa640 10 client.16080 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47b95a1640 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779cbf390 0x55e779cc17b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8a8b607d5edd2fb server_cookie=1cfa7e870f28490c in_seq=0 out_seq=0 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f479d7fa640 1 -- 172.21.15.57:0/3002107667 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e779bb1c10 con 0x55e779cbf390 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f479d7fa640 10 client.16080 handle_command_reply: tid=0 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 client.16080 shutdown 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 msgr2=0x55e779c51010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c51010 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009f40 tx=0x7f47b009bd40 comp rx=0 tx=0).stop 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 shutdown_connections 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e779cbf390 0x55e779cc17b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e779c48a00 0x55e779c424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f479d7fa640 0 client.16080 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e779c3fe80 0x55e779c41fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 --2- 172.21.15.57:0/3002107667 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e779c3f4d0 0x55e779c51010 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 >> 172.21.15.57:0/3002107667 conn(0x55e779c39ba0 msgr2=0x55e779c40af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 shutdown_connections 2024-04-21T08:47:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 1 -- 172.21.15.57:0/3002107667 wait complete. 2024-04-21T08:47:05.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.387+0000 7f47bebf8740 20 client.16080 trim_cache size 0 max 16384 2024-04-21T08:47:05.393 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47a00a3d20 msgr2=0x7f47a00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47a00a3d20 0x7f47a00a61e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f47ac04b000 tx=0x7f47ac044000 comp rx=0 tx=0).stop 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 msgr2=0x7f47b4100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 0x7f47b4100e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f47b000d770 tx=0x7f47b000dc30 comp rx=0 tx=0).stop 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 shutdown_connections 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47a00a3d20 0x7f47a00a61e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b4102b20 0x7f47b4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b4107270 0x7f47b41013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 --2- 172.21.15.57:0/1732625563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47b406a840 0x7f47b4100e80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 >> 172.21.15.57:0/1732625563 conn(0x7f47b4077eb0 msgr2=0x7f47b40fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 shutdown_connections 2024-04-21T08:47:05.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:05.388+0000 7f47bc006640 1 -- 172.21.15.57:0/1732625563 wait complete. 2024-04-21T08:47:05.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:05 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2050918595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:05.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:05 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2050918595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:05.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:05 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2050918595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:06.399 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:47:06.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:47:06.399 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 -- 172.21.15.57:0/1727584683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d74106160 msgr2=0x7f5d7410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 --2- 172.21.15.57:0/1727584683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d74106160 0x7f5d7410dd30 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009be0 tx=0x7f5d6c02bef0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 -- 172.21.15.57:0/1727584683 shutdown_connections 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 --2- 172.21.15.57:0/1727584683 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d74106160 0x7f5d7410dd30 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 --2- 172.21.15.57:0/1727584683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d74101a40 0x7f5d74105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 --2- 172.21.15.57:0/1727584683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74101090 0x7f5d74101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 -- 172.21.15.57:0/1727584683 >> 172.21.15.57:0/1727584683 conn(0x7f5d74077f50 msgr2=0x7f5d74078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 -- 172.21.15.57:0/1727584683 shutdown_connections 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 -- 172.21.15.57:0/1727584683 wait complete. 2024-04-21T08:47:06.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.519+0000 7f5d7944e640 1 Processor -- start 2024-04-21T08:47:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 -- start start 2024-04-21T08:47:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d74101090 0x7f5d7410e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 0x7f5d7410ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7410ef40 0x7f5d741a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741121d0 con 0x7f5d7410ef40 2024-04-21T08:47:06.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74112050 con 0x7f5d74106160 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d7944e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5d74112350 con 0x7f5d74101090 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 0x7f5d7410ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7410ef40 0x7f5d741a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d72ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d74101090 0x7f5d7410e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 0x7f5d7410ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:60104/0 (socket says 172.21.15.57:60104) 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 -- 172.21.15.57:0/2613426278 learned_addr learned my addr 172.21.15.57:0/2613426278 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 -- 172.21.15.57:0/2613426278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d74101090 msgr2=0x7f5d7410e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d74101090 0x7f5d7410e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 -- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7410ef40 msgr2=0x7f5d741a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7410ef40 0x7f5d741a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d6a7fc640 1 -- 172.21.15.57:0/2613426278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d741a6f80 con 0x7f5d74106160 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.520+0000 7f5d737fe640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7410ef40 0x7f5d741a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.521+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 0x7f5d7410ea00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00d770 tx=0x7f5d5c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.521+0000 7f5d70ff9640 1 -- 172.21.15.57:0/2613426278 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c017070 con 0x7f5d74106160 2024-04-21T08:47:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.521+0000 7f5d70ff9640 1 -- 172.21.15.57:0/2613426278 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d5c009c30 con 0x7f5d74106160 2024-04-21T08:47:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.521+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d741a7270 con 0x7f5d74106160 2024-04-21T08:47:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.521+0000 7f5d70ff9640 1 -- 172.21.15.57:0/2613426278 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00f620 con 0x7f5d74106160 2024-04-21T08:47:06.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.521+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d741a77c0 con 0x7f5d74106160 2024-04-21T08:47:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.523+0000 7f5d70ff9640 1 -- 172.21.15.57:0/2613426278 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5d5c01a070 con 0x7f5d74106160 2024-04-21T08:47:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.523+0000 7f5d7c043740 1 Processor -- start 2024-04-21T08:47:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.523+0000 7f5d7c043740 1 -- start start 2024-04-21T08:47:06.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d5f8e0 0x5649d0d5fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d68840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d6b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d7c043740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5649d0cc3f00 con 0x5649d0d5f8e0 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d7c043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649d0c61590 con 0x5649d0d60290 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d7c043740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5649d0cbfec0 con 0x5649d0d68e10 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d5f8e0 0x5649d0d5fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d68840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d5f8e0 0x5649d0d5fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49922/0 (socket says 172.21.15.57:49922) 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d737fe640 1 -- 172.21.15.57:0/597593130 learned_addr learned my addr 172.21.15.57:0/597593130 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d6b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 -- 172.21.15.57:0/597593130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 msgr2=0x5649d0d6b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d6b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 -- 172.21.15.57:0/597593130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d5f8e0 msgr2=0x5649d0d5fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d5f8e0 0x5649d0d5fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 -- 172.21.15.57:0/597593130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d0a9e7e0 con 0x5649d0d60290 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d70ff9640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5d600a3d20 0x7f5d600a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d68840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d64005fd0 tx=0x7f5d64011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20df98f4ebe7f931 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.524+0000 7f5d737fe640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5d600a3d20 0x7f5d600a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.525+0000 7f5d70ff9640 1 -- 172.21.15.57:0/2613426278 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5d5c0d0590 con 0x7f5d74106160 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.525+0000 7f5d697fa640 1 -- 172.21.15.57:0/597593130 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64004020 con 0x5649d0d60290 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.525+0000 7f5d697fa640 1 -- 172.21.15.57:0/597593130 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d640041c0 con 0x5649d0d60290 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.525+0000 7f5d697fa640 1 -- 172.21.15.57:0/597593130 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d640197a0 con 0x5649d0d60290 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- 172.21.15.57:0/597593130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 msgr2=0x5649d0d68840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d68840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d64005fd0 tx=0x7f5d64011810 comp rx=0 tx=0).stop 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d737fe640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5d600a3d20 0x7f5d600a61e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0099a0 tx=0x7f5d6c009280 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- 172.21.15.57:0/597593130 shutdown_connections 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d6b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d68840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- 172.21.15.57:0/597593130 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d5f8e0 0x5649d0d5fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- 172.21.15.57:0/597593130 >> 172.21.15.57:0/597593130 conn(0x5649d0d59fb0 msgr2=0x5649d0d5a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- 172.21.15.57:0/597593130 shutdown_connections 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- 172.21.15.57:0/597593130 wait complete. 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- start start 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 Processor -- start 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d65db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 0x5649d0d662f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d66830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5649d0d6bd80 con 0x5649d0d60290 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5649d0c7d580 con 0x5649d0d5f8e0 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.526+0000 7f5d7c043740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5649d0d6bf00 con 0x5649d0d68e10 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d7c043740 10 client.0 _ll_register_callbacks cb 0x5649d0c83650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 0x5649d0d662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d65db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 0x5649d0d662f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49930/0 (socket says 172.21.15.57:49930) 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d65db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:60128/0 (socket says 172.21.15.57:60128) 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d6a7fc640 1 -- 172.21.15.57:0/1625992524 learned_addr learned my addr 172.21.15.57:0/1625992524 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d66830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 -- 172.21.15.57:0/1625992524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 msgr2=0x5649d0d66830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d66830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 -- 172.21.15.57:0/1625992524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 msgr2=0x5649d0d662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 0x5649d0d662f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 -- 172.21.15.57:0/1625992524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d0afef70 con 0x5649d0d5f8e0 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d66830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 0x5649d0d662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d65db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5649d0d5b020 tx=0x7f5d64005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4de43aea4e205dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d5a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:06.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1625992524 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64029020 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1625992524 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5d640035f0 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.527+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d640041c0 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1625992524 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5d64034050 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 10 client.25607 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5649d0996170 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1625992524 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d640041c0 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1625992524 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5d64004b10 con 0x5649d0d5f8e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 10 client.25607 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 10 client.25607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 20 client.25607 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0d792e0 0x5649d0d7b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 4 client.25607 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649d0cd1920 con 0x5649d0d792e0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.528+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0d792e0 0x5649d0d7b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.529+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0d792e0 0x5649d0d7b6d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=889eab68af965d63 server_cookie=6676156d8b1aeb7f in_seq=0 out_seq=0 2024-04-21T08:47:06.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.529+0000 7f5d5a7fc640 10 client.25607 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.530+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1625992524 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649d0cd1920 con 0x5649d0d792e0 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.530+0000 7f5d5a7fc640 10 client.25607 handle_command_reply: tid=0 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 client.25607 shutdown 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 msgr2=0x5649d0d65db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d65db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5649d0d5b020 tx=0x7f5d64005e70 comp rx=0 tx=0).stop 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 shutdown_connections 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0d792e0 0x5649d0d7b6d0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d68e10 0x5649d0d66830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d60290 0x5649d0d662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1625992524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d65db0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d5a7fc640 0 client.25607 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 >> 172.21.15.57:0/1625992524 conn(0x5649d0d59fb0 msgr2=0x5649d0d5bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 shutdown_connections 2024-04-21T08:47:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 1 -- 172.21.15.57:0/1625992524 wait complete. 2024-04-21T08:47:06.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.531+0000 7f5d7c043740 20 client.25607 trim_cache size 0 max 16384 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 Processor -- start 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 -- start start 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d60290 0x5649d0d74c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d64f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741af460 con 0x5649d0d68e10 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b1cc0 con 0x5649d0d5f8e0 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d7c043740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741afab0 con 0x5649d0d60290 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d64f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.553 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d64f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49938/0 (socket says 172.21.15.57:49938) 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 -- 172.21.15.57:0/1300470875 learned_addr learned my addr 172.21.15.57:0/1300470875 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d60290 0x5649d0d74c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 -- 172.21.15.57:0/1300470875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d60290 msgr2=0x5649d0d74c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d60290 0x5649d0d74c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 -- 172.21.15.57:0/1300470875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 msgr2=0x5649d0d684e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d684e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d737fe640 1 -- 172.21.15.57:0/1300470875 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649d0d5a7f0 con 0x5649d0d68e10 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d60290 0x5649d0d74c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.549+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d684e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d64f10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c036040 tx=0x7f5d6c042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76843a006be8c775 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d68ff9640 1 -- 172.21.15.57:0/1300470875 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c02cde0 con 0x5649d0d68e10 2024-04-21T08:47:06.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d68ff9640 1 -- 172.21.15.57:0/1300470875 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6c0348e0 con 0x5649d0d68e10 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d68ff9640 1 -- 172.21.15.57:0/1300470875 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c004460 con 0x5649d0d68e10 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 -- 172.21.15.57:0/1300470875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 msgr2=0x5649d0d64f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d64f10 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c036040 tx=0x7f5d6c042000 comp rx=0 tx=0).stop 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 -- 172.21.15.57:0/1300470875 shutdown_connections 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d64f10 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d60290 0x5649d0d74c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1300470875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d5f8e0 0x5649d0d684e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.550+0000 7f5d7c043740 1 -- 172.21.15.57:0/1300470875 >> 172.21.15.57:0/1300470875 conn(0x5649d0d59fb0 msgr2=0x5649d0d58780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 -- 172.21.15.57:0/1300470875 shutdown_connections 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 -- 172.21.15.57:0/1300470875 wait complete. 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 -- start start 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 Processor -- start 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d5f8e0 0x5649d0d6cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d6d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d6d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b0180 con 0x5649d0d68e10 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b15f0 con 0x5649d0d60290 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5d741b0850 con 0x5649d0d5f8e0 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d7c043740 10 client.0 _ll_register_callbacks cb 0x7f5d6c009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d6d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d737fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d6d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49940/0 (socket says 172.21.15.57:49940) 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d737fe640 1 -- 172.21.15.57:0/1999645018 learned_addr learned my addr 172.21.15.57:0/1999645018 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d6d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.551+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d5f8e0 0x5649d0d6cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d737fe640 1 -- 172.21.15.57:0/1999645018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d5f8e0 msgr2=0x5649d0d6cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d5f8e0 0x5649d0d6cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d737fe640 1 -- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 msgr2=0x5649d0d6d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d6d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d737fe640 1 -- 172.21.15.57:0/1999645018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649d0dde720 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d6a7fc640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d6d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d737fe640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d6d5d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c02c430 tx=0x7f5d6c004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab925c837dd522e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d5a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1999645018 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c05e070 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1999645018 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5d6c0041a0 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d6c0348e0 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1999645018 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5d6c004b40 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d7c043740 10 client.16098 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5649d0ddfa00 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.552+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1999645018 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d6c0348e0 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1999645018 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5d6c063ae0 con 0x5649d0d68e10 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d7c043740 10 client.16098 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d7c043740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d7c043740 20 client.16098 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:06.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0de00a0 0x5649d0de24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d7c043740 4 client.16098 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:47:06.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5649d0ccb790 con 0x5649d0de00a0 2024-04-21T08:47:06.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0de00a0 0x5649d0de24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d72ffd640 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0de00a0 0x5649d0de24c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7f74a20186c86bb server_cookie=d0fb032cdb3258dc in_seq=0 out_seq=0 2024-04-21T08:47:06.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.553+0000 7f5d5a7fc640 10 client.16098 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d5a7fc640 1 -- 172.21.15.57:0/1999645018 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5649d0ccb790 con 0x5649d0de00a0 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d5a7fc640 10 client.16098 handle_command_reply: tid=0 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 client.16098 shutdown 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 msgr2=0x5649d0d6d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d6d5d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c02c430 tx=0x7f5d6c004000 comp rx=0 tx=0).stop 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 shutdown_connections 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5649d0de00a0 0x5649d0de24c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5649d0d68e10 0x5649d0d6d5d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5649d0d60290 0x5649d0d6d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 --2- 172.21.15.57:0/1999645018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5649d0d5f8e0 0x5649d0d6cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 >> 172.21.15.57:0/1999645018 conn(0x5649d0d59fb0 msgr2=0x5649d0d5b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d5a7fc640 0 client.16098 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 shutdown_connections 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.604+0000 7f5d7c043740 1 -- 172.21.15.57:0/1999645018 wait complete. 2024-04-21T08:47:06.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7c043740 20 client.16098 trim_cache size 0 max 16384 2024-04-21T08:47:06.606 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:47:06.606 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:47:06.606 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "048beeba-ca32-4dd1-8222-3e1746d25f99", 2024-04-21T08:47:06.606 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:47:06.606 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5d600a3d20 msgr2=0x7f5d600a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5d600a3d20 0x7f5d600a61e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0099a0 tx=0x7f5d6c009280 comp rx=0 tx=0).stop 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 msgr2=0x7f5d7410ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 0x7f5d7410ea00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00d770 tx=0x7f5d5c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 shutdown_connections 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5d600a3d20 0x7f5d600a61e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5d7410ef40 0x7f5d741a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d74106160 0x7f5d7410ea00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 --2- 172.21.15.57:0/2613426278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d74101090 0x7f5d7410e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 >> 172.21.15.57:0/2613426278 conn(0x7f5d74077f50 msgr2=0x7f5d740ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 shutdown_connections 2024-04-21T08:47:06.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.605+0000 7f5d7944e640 1 -- 172.21.15.57:0/2613426278 wait complete. 2024-04-21T08:47:06.614 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:06 smithi057 ceph-mon[27968]: pgmap v53: 129 pgs: 129 active+clean; 188 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 1.60k op/s 2024-04-21T08:47:06.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:06.616 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:06 smithi136 ceph-mon[31791]: pgmap v53: 129 pgs: 129 active+clean; 188 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 1.60k op/s 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 -- 172.21.15.57:0/3208146960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c1010c0 msgr2=0x7efd6c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 --2- 172.21.15.57:0/3208146960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c1010c0 0x7efd6c106bf0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efd54009900 tx=0x7efd5402bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 -- 172.21.15.57:0/3208146960 shutdown_connections 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 --2- 172.21.15.57:0/3208146960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd6c1072d0 0x7efd6c10ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 --2- 172.21.15.57:0/3208146960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c1010c0 0x7efd6c106bf0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 --2- 172.21.15.57:0/3208146960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6c100710 0x7efd6c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 -- 172.21.15.57:0/3208146960 >> 172.21.15.57:0/3208146960 conn(0x7efd6c077360 msgr2=0x7efd6c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 -- 172.21.15.57:0/3208146960 shutdown_connections 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 -- 172.21.15.57:0/3208146960 wait complete. 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.723+0000 7efd712b3640 1 Processor -- start 2024-04-21T08:47:06.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 -- start start 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd6c100710 0x7efd6c101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6c1072d0 0x7efd6c101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 0x7efd6c1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c112180 con 0x7efd6c102100 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c112000 con 0x7efd6c1072d0 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd712b3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c112300 con 0x7efd6c100710 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 0x7efd6c1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 0x7efd6c1a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49964/0 (socket says 172.21.15.57:49964) 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 -- 172.21.15.57:0/1780931935 learned_addr learned my addr 172.21.15.57:0/1780931935 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6c1072d0 0x7efd6c101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6affd640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd6c100710 0x7efd6c101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 -- 172.21.15.57:0/1780931935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd6c100710 msgr2=0x7efd6c101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd6c100710 0x7efd6c101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 -- 172.21.15.57:0/1780931935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6c1072d0 msgr2=0x7efd6c101bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6c1072d0 0x7efd6c101bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 -- 172.21.15.57:0/1780931935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c1a7010 con 0x7efd6c102100 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.724+0000 7efd6b7fe640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 0x7efd6c1a6910 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efd6400e970 tx=0x7efd6400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.725+0000 7efd63fff640 1 -- 172.21.15.57:0/1780931935 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6400ccc0 con 0x7efd6c102100 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.725+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6c1a7300 con 0x7efd6c102100 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.725+0000 7efd63fff640 1 -- 172.21.15.57:0/1780931935 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd6400ce60 con 0x7efd6c102100 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.725+0000 7efd63fff640 1 -- 172.21.15.57:0/1780931935 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd64010430 con 0x7efd6c102100 2024-04-21T08:47:06.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.725+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6c1a7820 con 0x7efd6c102100 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.727+0000 7efd63fff640 1 -- 172.21.15.57:0/1780931935 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efd64004f90 con 0x7efd6c102100 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.727+0000 7efd73f7e740 1 Processor -- start 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.727+0000 7efd73f7e740 1 -- start start 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 0x5619632d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd63fff640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efd580a3d50 0x7efd580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632dea80 0x5619632e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd73f7e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56196323a1e0 con 0x5619632dea80 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd73f7e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619631d7390 con 0x5619632d5580 2024-04-21T08:47:06.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd73f7e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5619632361a0 con 0x5619632d5ea0 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632dea80 0x5619632e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6b7fe640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efd580a3d50 0x7efd580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 0x5619632d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 0x5619632d5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:60172/0 (socket says 172.21.15.57:60172) 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 -- 172.21.15.57:0/2618728665 learned_addr learned my addr 172.21.15.57:0/2618728665 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6affd640 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd63fff640 1 -- 172.21.15.57:0/1780931935 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7efd640d0f50 con 0x7efd6c102100 2024-04-21T08:47:06.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 -- 172.21.15.57:0/2618728665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 msgr2=0x5619632de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 -- 172.21.15.57:0/2618728665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632dea80 msgr2=0x5619632e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632dea80 0x5619632e0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.728+0000 7efd6a7fc640 1 -- 172.21.15.57:0/2618728665 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619630167d0 con 0x5619632d5580 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.729+0000 7efd6b7fe640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efd580a3d50 0x7efd580a6210 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd640ce940 tx=0x7efd640049b0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.729+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 0x5619632d5960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efd6c074660 tx=0x7efd54005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb3b47ee94f53765 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.729+0000 7efd617fa640 1 -- 172.21.15.57:0/2618728665 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5402c7e0 con 0x5619632d5580 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.729+0000 7efd617fa640 1 -- 172.21.15.57:0/2618728665 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd5404a070 con 0x5619632d5580 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.729+0000 7efd617fa640 1 -- 172.21.15.57:0/2618728665 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54004920 con 0x5619632d5580 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 -- 172.21.15.57:0/2618728665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 msgr2=0x5619632d5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 0x5619632d5960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efd6c074660 tx=0x7efd54005fd0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 -- 172.21.15.57:0/2618728665 shutdown_connections 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632dea80 0x5619632e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- 172.21.15.57:0/2618728665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5580 0x5619632d5960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 -- 172.21.15.57:0/2618728665 >> 172.21.15.57:0/2618728665 conn(0x5619632cfcb0 msgr2=0x5619632d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 -- 172.21.15.57:0/2618728665 shutdown_connections 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 -- 172.21.15.57:0/2618728665 wait complete. 2024-04-21T08:47:06.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 -- start start 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 Processor -- start 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5580 0x5619632dbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 0x5619632dc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.730+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632dc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd73f7e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5619632e1810 con 0x5619632d5ea0 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd73f7e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619631f32f0 con 0x5619632dea80 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd73f7e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5619632e1990 con 0x5619632d5580 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd73f7e740 10 client.0 _ll_register_callbacks cb 0x561963200580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 0x5619632dc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 0x5619632dc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49970/0 (socket says 172.21.15.57:49970) 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 -- 172.21.15.57:0/3417570699 learned_addr learned my addr 172.21.15.57:0/3417570699 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632dc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5580 0x5619632dbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 -- 172.21.15.57:0/3417570699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5580 msgr2=0x5619632dbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5580 0x5619632dbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 -- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 msgr2=0x5619632dc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632dc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 -- 172.21.15.57:0/3417570699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561962f4c3f0 con 0x5619632d5ea0 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd6affd640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 0x5619632dc010 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efd5c003fc0 tx=0x7efd5c0044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da72a669d065d70 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd527fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd527fc640 1 -- 172.21.15.57:0/3417570699 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c011900 con 0x5619632d5ea0 2024-04-21T08:47:06.739 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd527fc640 1 -- 172.21.15.57:0/3417570699 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7efd5c011aa0 con 0x5619632d5ea0 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd5404a070 con 0x5619632d5ea0 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.731+0000 7efd527fc640 1 -- 172.21.15.57:0/3417570699 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efd5c024050 con 0x5619632d5ea0 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 10 client.16110 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561962f0c0f0 con 0x5619632d5ea0 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd527fc640 1 -- 172.21.15.57:0/3417570699 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd5404a070 con 0x5619632d5ea0 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd527fc640 1 -- 172.21.15.57:0/3417570699 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7efd5c010440 con 0x5619632d5ea0 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 10 client.16110 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 20 client.16110 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5619632eeb00 0x5619632f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 4 client.16110 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561963247c00 con 0x5619632eeb00 2024-04-21T08:47:06.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5619632eeb00 0x5619632f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5619632eeb00 0x5619632f0f20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7648de947af81600 server_cookie=319170d940caa184 in_seq=0 out_seq=0 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.732+0000 7efd527fc640 10 client.16110 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd527fc640 1 -- 172.21.15.57:0/3417570699 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561963247c00 con 0x5619632eeb00 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd527fc640 10 client.16110 handle_command_reply: tid=0 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 client.16110 shutdown 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 msgr2=0x5619632dc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 0x5619632dc010 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efd5c003fc0 tx=0x7efd5c0044d0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 shutdown_connections 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5619632eeb00 0x5619632f0f20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632dc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5ea0 0x5619632dc010 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 --2- 172.21.15.57:0/3417570699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5580 0x5619632dbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd527fc640 0 client.16110 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 >> 172.21.15.57:0/3417570699 conn(0x5619632cfcb0 msgr2=0x5619632d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 shutdown_connections 2024-04-21T08:47:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 1 -- 172.21.15.57:0/3417570699 wait complete. 2024-04-21T08:47:06.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.734+0000 7efd73f7e740 20 client.16110 trim_cache size 0 max 16384 2024-04-21T08:47:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.753+0000 7efd73f7e740 1 Processor -- start 2024-04-21T08:47:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.753+0000 7efd73f7e740 1 -- start start 2024-04-21T08:47:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632daa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5ea0 0x5619632ea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632dea80 0x5619632dc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd73f7e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1af3c0 con 0x5619632d5580 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd73f7e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1b1c20 con 0x5619632d5ea0 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd73f7e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1afa10 con 0x5619632dea80 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632daa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632daa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49976/0 (socket says 172.21.15.57:49976) 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 -- 172.21.15.57:0/160923960 learned_addr learned my addr 172.21.15.57:0/160923960 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6affd640 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5ea0 0x5619632ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632dea80 0x5619632dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 -- 172.21.15.57:0/160923960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632dea80 msgr2=0x5619632dc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632dea80 0x5619632dc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 -- 172.21.15.57:0/160923960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5ea0 msgr2=0x5619632ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5ea0 0x5619632ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 -- 172.21.15.57:0/160923960 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619632da020 con 0x5619632d5580 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6affd640 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5ea0 0x5619632ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632daa10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7efd54009900 tx=0x7efd54005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8beac23c13433468 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.754+0000 7efd60ff9640 1 -- 172.21.15.57:0/160923960 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5402c9e0 con 0x5619632d5580 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd60ff9640 1 -- 172.21.15.57:0/160923960 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd5404a040 con 0x5619632d5580 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd60ff9640 1 -- 172.21.15.57:0/160923960 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd540407e0 con 0x5619632d5580 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 -- 172.21.15.57:0/160923960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 msgr2=0x5619632daa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632daa10 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7efd54009900 tx=0x7efd54005570 comp rx=0 tx=0).stop 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 -- 172.21.15.57:0/160923960 shutdown_connections 2024-04-21T08:47:06.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632dea80 0x5619632dc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632d5ea0 0x5619632ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 --2- 172.21.15.57:0/160923960 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632daa10 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 -- 172.21.15.57:0/160923960 >> 172.21.15.57:0/160923960 conn(0x5619632cfcb0 msgr2=0x5619632ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 -- 172.21.15.57:0/160923960 shutdown_connections 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.755+0000 7efd73f7e740 1 -- 172.21.15.57:0/160923960 wait complete. 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 -- start start 2024-04-21T08:47:06.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 Processor -- start 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632d8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632d8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632d92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1b00e0 con 0x5619632d5580 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1b1550 con 0x5619632dea80 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1b07b0 con 0x5619632d5ea0 2024-04-21T08:47:06.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd73f7e740 10 client.0 _ll_register_callbacks cb 0x7efd5c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632d8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd6a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632d8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49984/0 (socket says 172.21.15.57:49984) 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd6a7fc640 1 -- 172.21.15.57:0/1734568261 learned_addr learned my addr 172.21.15.57:0/1734568261 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632d92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.756+0000 7efd6affd640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632d8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd6a7fc640 1 -- 172.21.15.57:0/1734568261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 msgr2=0x5619632d8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632d8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd6a7fc640 1 -- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 msgr2=0x5619632d92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632d92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd6a7fc640 1 -- 172.21.15.57:0/1734568261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561963354170 con 0x5619632d5580 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd6a7fc640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632d8870 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7efd540404c0 tx=0x7efd5405c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d1d6514f551719 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd527fc640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd527fc640 1 -- 172.21.15.57:0/1734568261 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5405c4c0 con 0x5619632d5580 2024-04-21T08:47:06.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd527fc640 1 -- 172.21.15.57:0/1734568261 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7efd5405c660 con 0x5619632d5580 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd5404a040 con 0x5619632d5580 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd527fc640 1 -- 172.21.15.57:0/1734568261 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7efd5402cc80 con 0x5619632d5580 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd73f7e740 10 client.16116 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561963355480 con 0x5619632d5580 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.757+0000 7efd527fc640 1 -- 172.21.15.57:0/1734568261 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd5404a040 con 0x5619632d5580 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd527fc640 1 -- 172.21.15.57:0/1734568261 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7efd540434e0 con 0x5619632d5580 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd73f7e740 10 client.16116 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd73f7e740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd73f7e740 20 client.16116 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd73f7e740 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561963355760 0x561963357b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd73f7e740 4 client.16116 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561963241a70 con 0x561963355760 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd6affd640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561963355760 0x561963357b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd527fc640 10 client.16116 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.758+0000 7efd6affd640 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561963355760 0x561963357b50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13ae21374b555662 server_cookie=bdc12cb3b3b9d7dd in_seq=0 out_seq=0 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd527fc640 1 -- 172.21.15.57:0/1734568261 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x561963241a70 con 0x561963355760 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd527fc640 10 client.16116 handle_command_reply: tid=0 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 client.16116 shutdown 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 msgr2=0x5619632d8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632d8870 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7efd540404c0 tx=0x7efd5405c9f0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 shutdown_connections 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561963355760 0x561963357b50 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619632dea80 0x5619632d92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5619632d5ea0 0x5619632d8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 --2- 172.21.15.57:0/1734568261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5619632d5580 0x5619632d8870 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 >> 172.21.15.57:0/1734568261 conn(0x5619632cfcb0 msgr2=0x5619632d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd527fc640 0 client.16116 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 shutdown_connections 2024-04-21T08:47:06.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 1 -- 172.21.15.57:0/1734568261 wait complete. 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.761+0000 7efd73f7e740 20 client.16116 trim_cache size 0 max 16384 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (1613 inodes in the stack)", 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: "048beeba-ca32-4dd1-8222-3e1746d25f99": { 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "048beeba-ca32-4dd1-8222-3e1746d25f99", 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:47:06.766 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:47:06.767 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:47:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efd580a3d50 msgr2=0x7efd580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efd580a3d50 0x7efd580a6210 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd640ce940 tx=0x7efd640049b0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 msgr2=0x7efd6c1a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 0x7efd6c1a6910 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7efd6400e970 tx=0x7efd6400ed70 comp rx=0 tx=0).stop 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 shutdown_connections 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7efd580a3d50 0x7efd580a6210 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efd6c102100 0x7efd6c1a6910 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6c1072d0 0x7efd6c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 --2- 172.21.15.57:0/1780931935 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd6c100710 0x7efd6c101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 >> 172.21.15.57:0/1780931935 conn(0x7efd6c077360 msgr2=0x7efd6c105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 shutdown_connections 2024-04-21T08:47:06.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:06.762+0000 7efd712b3640 1 -- 172.21.15.57:0/1780931935 wait complete. 2024-04-21T08:47:06.772 INFO:tasks.cephfs.filesystem:scrub status for tag:048beeba-ca32-4dd1-8222-3e1746d25f99 - {'path': '/', 'tag': '048beeba-ca32-4dd1-8222-3e1746d25f99', 'options': 'recursive,force'} 2024-04-21T08:47:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:06 smithi129 ceph-mon[31272]: pgmap v53: 129 pgs: 129 active+clean; 188 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 1.60k op/s 2024-04-21T08:47:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:07 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:47:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:07 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:07 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:47:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:07 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:47:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:07 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:07 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:47:07.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:07 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:47:07.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:07 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:07.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:07 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:47:08.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:08 smithi136 ceph-mon[31791]: pgmap v54: 129 pgs: 129 active+clean; 190 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 1.60k op/s 2024-04-21T08:47:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:08 smithi129 ceph-mon[31272]: pgmap v54: 129 pgs: 129 active+clean; 190 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 1.60k op/s 2024-04-21T08:47:08.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:08 smithi057 ceph-mon[27968]: pgmap v54: 129 pgs: 129 active+clean; 190 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.6 MiB/s wr, 1.60k op/s 2024-04-21T08:47:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:10 smithi136 ceph-mon[31791]: pgmap v55: 129 pgs: 129 active+clean; 190 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 861 KiB/s wr, 1.47k op/s 2024-04-21T08:47:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:10 smithi129 ceph-mon[31272]: pgmap v55: 129 pgs: 129 active+clean; 190 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 861 KiB/s wr, 1.47k op/s 2024-04-21T08:47:10.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:10 smithi057 ceph-mon[27968]: pgmap v55: 129 pgs: 129 active+clean; 190 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 861 KiB/s wr, 1.47k op/s 2024-04-21T08:47:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 -- 172.21.15.57:0/2787956747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc1021a0 msgr2=0x7fd1cc102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 --2- 172.21.15.57:0/2787956747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc102580 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009900 tx=0x7fd1bc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 -- 172.21.15.57:0/2787956747 shutdown_connections 2024-04-21T08:47:10.940 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 --2- 172.21.15.57:0/2787956747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1cc107270 0x7fd1cc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 --2- 172.21.15.57:0/2787956747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc102b50 0x7fd1cc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 --2- 172.21.15.57:0/2787956747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc102580 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 -- 172.21.15.57:0/2787956747 >> 172.21.15.57:0/2787956747 conn(0x7fd1cc0793e0 msgr2=0x7fd1cc0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 -- 172.21.15.57:0/2787956747 shutdown_connections 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 -- 172.21.15.57:0/2787956747 wait complete. 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 Processor -- start 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.938+0000 7fd1d234b640 1 -- start start 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d234b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d234b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1cc107270 0x7fd1cc198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d234b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 0x7fd1cc19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d234b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc112200 con 0x7fd1cc19cf40 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d234b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc112080 con 0x7fd1cc107270 2024-04-21T08:47:10.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d234b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc112380 con 0x7fd1cc1021a0 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 0x7fd1cc19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 0x7fd1cc19d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50002/0 (socket says 172.21.15.57:50002) 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 -- 172.21.15.57:0/3899692384 learned_addr learned my addr 172.21.15.57:0/3899692384 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1cc107270 0x7fd1cc198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 -- 172.21.15.57:0/3899692384 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc1021a0 msgr2=0x7fd1cc198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 -- 172.21.15.57:0/3899692384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1cc107270 msgr2=0x7fd1cc198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1cc107270 0x7fd1cc198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1d08e8640 1 -- 172.21.15.57:0/3899692384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc19da30 con 0x7fd1cc19cf40 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.939+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.940+0000 7fd1d08e8640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 0x7fd1cc19d3f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400e960 tx=0x7fd1c400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:10.942 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.940+0000 7fd1c97fa640 1 -- 172.21.15.57:0/3899692384 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4004d20 con 0x7fd1cc19cf40 2024-04-21T08:47:10.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.940+0000 7fd1c97fa640 1 -- 172.21.15.57:0/3899692384 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c4005020 con 0x7fd1cc19cf40 2024-04-21T08:47:10.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.940+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1cc19dd20 con 0x7fd1cc19cf40 2024-04-21T08:47:10.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.940+0000 7fd1c97fa640 1 -- 172.21.15.57:0/3899692384 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40116d0 con 0x7fd1cc19cf40 2024-04-21T08:47:10.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.940+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1cc1a5510 con 0x7fd1cc19cf40 2024-04-21T08:47:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.942+0000 7fd1c97fa640 1 -- 172.21.15.57:0/3899692384 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd1c40051d0 con 0x7fd1cc19cf40 2024-04-21T08:47:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.942+0000 7fd1d5014740 1 Processor -- start 2024-04-21T08:47:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 -- start start 2024-04-21T08:47:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x555881513620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881513fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.953 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x5558815238e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5558814788d0 con 0x5558815214f0 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555881415e80 con 0x5558815148d0 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d5014740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x555881474890 con 0x555881513b60 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x5558815238e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x5558815238e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50018/0 (socket says 172.21.15.57:50018) 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1457395077 learned_addr learned my addr 172.21.15.57:0/1457395077 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1c97fa640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd1b40a3e40 0x7fd1b40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881513fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x555881513620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1d08e8640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd1b40a3e40 0x7fd1b40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1457395077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 msgr2=0x555881513fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881513fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1457395077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 msgr2=0x555881513620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x555881513620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1457395077 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555881437320 con 0x5558815214f0 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.943+0000 7fd1c97fa640 1 -- 172.21.15.57:0/3899692384 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd1c40d0bc0 con 0x7fd1cc19cf40 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.944+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x5558815238e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0005c50 tx=0x7fd1c000a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a382903e139020d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.944+0000 7fd1d08e8640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd1b40a3e40 0x7fd1b40a6300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40991d0 tx=0x7fd1c4098b90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.944+0000 7fd1b27fc640 1 -- 172.21.15.57:0/1457395077 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0005220 con 0x5558815214f0 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.944+0000 7fd1b27fc640 1 -- 172.21.15.57:0/1457395077 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c0004c50 con 0x5558815214f0 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.944+0000 7fd1b27fc640 1 -- 172.21.15.57:0/1457395077 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c000fb50 con 0x5558815214f0 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 -- 172.21.15.57:0/1457395077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 msgr2=0x5558815238e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x5558815238e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0005c50 tx=0x7fd1c000a730 comp rx=0 tx=0).stop 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 -- 172.21.15.57:0/1457395077 shutdown_connections 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x5558815238e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881513fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1457395077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x555881513620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 -- 172.21.15.57:0/1457395077 >> 172.21.15.57:0/1457395077 conn(0x5558814dac90 msgr2=0x55588150e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 -- 172.21.15.57:0/1457395077 shutdown_connections 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 -- 172.21.15.57:0/1457395077 wait complete. 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 -- start start 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.945+0000 7fd1d5014740 1 Processor -- start 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881520f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x55588151e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x55588151ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x555881524220 con 0x5558815214f0 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555881400eb0 con 0x5558815148d0 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5558815243a0 con 0x555881513b60 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1d5014740 10 client.0 _ll_register_callbacks cb 0x55588142db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x55588151ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x55588151ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50028/0 (socket says 172.21.15.57:50028) 2024-04-21T08:47:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1425313395 learned_addr learned my addr 172.21.15.57:0/1425313395 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x55588151e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1425313395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 msgr2=0x555881520f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881520f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 msgr2=0x55588151e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x55588151e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1425313395 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588118b210 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x55588151ea30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000f830 tx=0x7fd1c0005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3238b3e58a3535 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.946+0000 7fd1ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/1425313395 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0004e00 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/1425313395 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd1c00048d0 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c0004c50 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/1425313395 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd1c0005000 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 10 client.16128 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555881257620 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/1425313395 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c0004c50 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/1425313395 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd1c001b030 con 0x5558815214f0 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 10 client.16128 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 20 client.16128 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881531470 0x555881533860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 4 client.16128 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.947+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55588148c880 con 0x555881531470 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.948+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881531470 0x555881533860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.948+0000 7fd1ab7fe640 10 client.16128 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.948+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881531470 0x555881533860 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=574fa2c57d517dbf server_cookie=a9b3b3e90304f79d in_seq=0 out_seq=0 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/1425313395 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55588148c880 con 0x555881531470 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1ab7fe640 10 client.16128 handle_command_reply: tid=0 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 client.16128 shutdown 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 msgr2=0x55588151ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x55588151ea30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000f830 tx=0x7fd1c0005e30 comp rx=0 tx=0).stop 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 shutdown_connections 2024-04-21T08:47:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881531470 0x555881533860 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815214f0 0x55588151ea30 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815148d0 0x55588151e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1425313395 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881520f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1ab7fe640 0 client.16128 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 >> 172.21.15.57:0/1425313395 conn(0x5558814dac90 msgr2=0x5558815105a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 shutdown_connections 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 1 -- 172.21.15.57:0/1425313395 wait complete. 2024-04-21T08:47:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.949+0000 7fd1d5014740 20 client.16128 trim_cache size 0 max 16384 2024-04-21T08:47:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.964+0000 7fd1d5014740 1 Processor -- start 2024-04-21T08:47:10.969 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.964+0000 7fd1d5014740 1 -- start start 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.964+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 0x55588152d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.964+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558815148d0 0x555881538f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x5558815394c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1d5014740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1aafe0 con 0x555881513b60 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1d5014740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1ad840 con 0x5558815214f0 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1d5014740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1ab630 con 0x5558815148d0 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 0x55588152d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 0x55588152d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50042/0 (socket says 172.21.15.57:50042) 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1125796249 learned_addr learned my addr 172.21.15.57:0/1125796249 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1125796249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558815148d0 msgr2=0x555881538f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558815148d0 0x555881538f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1125796249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 msgr2=0x5558815394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.970 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x5558815394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 -- 172.21.15.57:0/1125796249 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588150f5e0 con 0x555881513b60 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 0x55588152d120 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0028fd0 tx=0x7fd1c000fbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42af3cc8f2ee49c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1b1ffb640 1 -- 172.21.15.57:0/1125796249 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c000a590 con 0x555881513b60 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1b1ffb640 1 -- 172.21.15.57:0/1125796249 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c000a340 con 0x555881513b60 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.965+0000 7fd1b1ffb640 1 -- 172.21.15.57:0/1125796249 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00048d0 con 0x555881513b60 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 -- 172.21.15.57:0/1125796249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 msgr2=0x55588152d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 0x55588152d120 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0028fd0 tx=0x7fd1c000fbe0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 -- 172.21.15.57:0/1125796249 shutdown_connections 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x5558815394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5558815148d0 0x555881538f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 --2- 172.21.15.57:0/1125796249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555881513b60 0x55588152d120 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 -- 172.21.15.57:0/1125796249 >> 172.21.15.57:0/1125796249 conn(0x5558814dac90 msgr2=0x555881519490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:10.971 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 -- 172.21.15.57:0/1125796249 shutdown_connections 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 -- 172.21.15.57:0/1125796249 wait complete. 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 -- start start 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.966+0000 7fd1d5014740 1 Processor -- start 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881529820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 0x55588151b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x55588151bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1abd00 con 0x5558815148d0 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1ad170 con 0x5558815214f0 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1ac3d0 con 0x555881513b60 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1d5014740 10 client.0 _ll_register_callbacks cb 0x7fd1bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 0x55588151b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 0x55588151b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50054/0 (socket says 172.21.15.57:50054) 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 -- 172.21.15.57:0/4118815489 learned_addr learned my addr 172.21.15.57:0/4118815489 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881529820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x55588151bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 -- 172.21.15.57:0/4118815489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 msgr2=0x555881529820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881529820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 -- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 msgr2=0x55588151bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x55588151bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 -- 172.21.15.57:0/4118815489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588151bff0 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x55588151bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1cbfff640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 0x55588151b570 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009230 tx=0x7fd1bc004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca0276bfa79714c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.967+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/4118815489 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc047050 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/4118815489 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd1bc049bd0 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c000a340 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/4118815489 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd1bc056050 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 10 client.16134 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555881522ab0 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/4118815489 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c000a340 con 0x5558815148d0 2024-04-21T08:47:10.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/4118815489 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd1bc0492f0 con 0x5558815148d0 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 10 client.16134 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 10 client.16134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 20 client.16134 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881597d10 0x55588159a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 4 client.16134 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.968+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5558814183d0 con 0x555881597d10 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.970+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881597d10 0x55588159a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.973+0000 7fd1cb7fe640 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881597d10 0x55588159a130 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1667014caf711fe4 server_cookie=14dc0aeba132019f in_seq=0 out_seq=0 2024-04-21T08:47:10.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:10.973+0000 7fd1ab7fe640 10 client.16134 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:11.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:11 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:11.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:11 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:47:11.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:11 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:11.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:11 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:47:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:11 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:11 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:47:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.586+0000 7fd1ab7fe640 1 -- 172.21.15.57:0/4118815489 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5558814183d0 con 0x555881597d10 2024-04-21T08:47:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.586+0000 7fd1ab7fe640 10 client.16134 handle_command_reply: tid=0 2024-04-21T08:47:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.586+0000 7fd1d5014740 1 client.16134 shutdown 2024-04-21T08:47:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.586+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 msgr2=0x55588151b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.586+0000 7fd1d5014740 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 0x55588151b570 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009230 tx=0x7fd1bc004420 comp rx=0 tx=0).stop 2024-04-21T08:47:12.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 shutdown_connections 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555881597d10 0x55588159a130 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5558815214f0 0x55588151bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5558815148d0 0x55588151b570 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1ab7fe640 0 client.16134 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 --2- 172.21.15.57:0/4118815489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555881513b60 0x555881529820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 >> 172.21.15.57:0/4118815489 conn(0x5558814dac90 msgr2=0x55588151ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 shutdown_connections 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 1 -- 172.21.15.57:0/4118815489 wait complete. 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d5014740 20 client.16134 trim_cache size 0 max 16384 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:47:12.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:47:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:47:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:47:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd1b40a3e40 msgr2=0x7fd1b40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd1b40a3e40 0x7fd1b40a6300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40991d0 tx=0x7fd1c4098b90 comp rx=0 tx=0).stop 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 msgr2=0x7fd1cc19d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 0x7fd1cc19d3f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400e960 tx=0x7fd1c400ee20 comp rx=0 tx=0).stop 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 shutdown_connections 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd1b40a3e40 0x7fd1b40a6300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1cc19cf40 0x7fd1cc19d3f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1cc107270 0x7fd1cc198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 --2- 172.21.15.57:0/3899692384 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1cc1021a0 0x7fd1cc198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 >> 172.21.15.57:0/3899692384 conn(0x7fd1cc0793e0 msgr2=0x7fd1cc0ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 shutdown_connections 2024-04-21T08:47:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:12.587+0000 7fd1d234b640 1 -- 172.21.15.57:0/3899692384 wait complete. 2024-04-21T08:47:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:12 smithi129 ceph-mon[31272]: pgmap v56: 129 pgs: 129 active+clean; 192 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 1.63k op/s 2024-04-21T08:47:12.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:12 smithi057 ceph-mon[27968]: pgmap v56: 129 pgs: 129 active+clean; 192 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 1.63k op/s 2024-04-21T08:47:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:12 smithi136 ceph-mon[31791]: pgmap v56: 129 pgs: 129 active+clean; 192 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.0 MiB/s wr, 1.63k op/s 2024-04-21T08:47:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:14 smithi129 ceph-mon[31272]: pgmap v57: 129 pgs: 129 active+clean; 193 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 920 KiB/s wr, 2.11k op/s 2024-04-21T08:47:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:14 smithi057 ceph-mon[27968]: pgmap v57: 129 pgs: 129 active+clean; 193 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 920 KiB/s wr, 2.11k op/s 2024-04-21T08:47:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:14 smithi136 ceph-mon[31791]: pgmap v57: 129 pgs: 129 active+clean; 193 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 920 KiB/s wr, 2.11k op/s 2024-04-21T08:47:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:16 smithi129 ceph-mon[31272]: pgmap v58: 129 pgs: 129 active+clean; 193 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 544 KiB/s wr, 913 op/s 2024-04-21T08:47:16.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:16 smithi057 ceph-mon[27968]: pgmap v58: 129 pgs: 129 active+clean; 193 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 544 KiB/s wr, 913 op/s 2024-04-21T08:47:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:16 smithi136 ceph-mon[31791]: pgmap v58: 129 pgs: 129 active+clean; 193 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 544 KiB/s wr, 913 op/s 2024-04-21T08:47:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:18 smithi129 ceph-mon[31272]: pgmap v59: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 2.02k op/s 2024-04-21T08:47:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:18 smithi057 ceph-mon[27968]: pgmap v59: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 2.02k op/s 2024-04-21T08:47:18.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:18 smithi136 ceph-mon[31791]: pgmap v59: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 2.02k op/s 2024-04-21T08:47:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:20 smithi129 ceph-mon[31272]: pgmap v60: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 2.01k op/s 2024-04-21T08:47:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:20 smithi057 ceph-mon[27968]: pgmap v60: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 2.01k op/s 2024-04-21T08:47:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:20 smithi136 ceph-mon[31791]: pgmap v60: 129 pgs: 129 active+clean; 213 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.3 MiB/s wr, 2.01k op/s 2024-04-21T08:47:22.826 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:22 smithi129 ceph-mon[31272]: pgmap v61: 129 pgs: 129 active+clean; 214 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 2.02k op/s 2024-04-21T08:47:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:22 smithi057 ceph-mon[27968]: pgmap v61: 129 pgs: 129 active+clean; 214 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 2.02k op/s 2024-04-21T08:47:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:22 smithi136 ceph-mon[31791]: pgmap v61: 129 pgs: 129 active+clean; 214 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.4 MiB/s wr, 2.02k op/s 2024-04-21T08:47:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:24 smithi129 ceph-mon[31272]: pgmap v62: 129 pgs: 129 active+clean; 218 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 1.86k op/s 2024-04-21T08:47:24.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:24 smithi057 ceph-mon[27968]: pgmap v62: 129 pgs: 129 active+clean; 218 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 1.86k op/s 2024-04-21T08:47:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:24 smithi136 ceph-mon[31791]: pgmap v62: 129 pgs: 129 active+clean; 218 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.6 MiB/s wr, 1.86k op/s 2024-04-21T08:47:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 -- 172.21.15.57:0/12883063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c102b50 msgr2=0x7f825c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 --2- 172.21.15.57:0/12883063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c102b50 0x7f825c106b90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f824c009900 tx=0x7f824c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 -- 172.21.15.57:0/12883063 shutdown_connections 2024-04-21T08:47:25.682 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 --2- 172.21.15.57:0/12883063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825c107270 0x7f825c10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 --2- 172.21.15.57:0/12883063 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c102b50 0x7f825c106b90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 --2- 172.21.15.57:0/12883063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c1021a0 0x7f825c102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 -- 172.21.15.57:0/12883063 >> 172.21.15.57:0/12883063 conn(0x7f825c0fda00 msgr2=0x7f825c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 -- 172.21.15.57:0/12883063 shutdown_connections 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 -- 172.21.15.57:0/12883063 wait complete. 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 Processor -- start 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.680+0000 7f8263b52640 1 -- start start 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8263b52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c1021a0 0x7f825c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8263b52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825c107270 0x7f825c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8263b52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 0x7f825c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.683 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8263b52640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1121b0 con 0x7f825c19a720 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8263b52640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f825c112030 con 0x7f825c107270 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8263b52640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f825c112330 con 0x7f825c1021a0 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 0x7f825c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82610ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825c107270 0x7f825c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 0x7f825c19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45176/0 (socket says 172.21.15.57:45176) 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 -- 172.21.15.57:0/838591088 learned_addr learned my addr 172.21.15.57:0/838591088 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82618ee640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c1021a0 0x7f825c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 -- 172.21.15.57:0/838591088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c1021a0 msgr2=0x7f825c1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c1021a0 0x7f825c1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 -- 172.21.15.57:0/838591088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825c107270 msgr2=0x7f825c1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825c107270 0x7f825c1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 -- 172.21.15.57:0/838591088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c19b320 con 0x7f825c19a720 2024-04-21T08:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f82620ef640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 0x7f825c19aba0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f825400eb10 tx=0x7f825400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8252ffd640 1 -- 172.21.15.57:0/838591088 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254004b20 con 0x7f825c19a720 2024-04-21T08:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.681+0000 7f8252ffd640 1 -- 172.21.15.57:0/838591088 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8254004cc0 con 0x7f825c19a720 2024-04-21T08:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.682+0000 7f8252ffd640 1 -- 172.21.15.57:0/838591088 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254010670 con 0x7f825c19a720 2024-04-21T08:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.682+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c1a7400 con 0x7f825c19a720 2024-04-21T08:47:25.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.682+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1a77b0 con 0x7f825c19a720 2024-04-21T08:47:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.685+0000 7f8252ffd640 1 -- 172.21.15.57:0/838591088 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8254005020 con 0x7f825c19a720 2024-04-21T08:47:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.685+0000 7f8266829740 1 Processor -- start 2024-04-21T08:47:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.685+0000 7f8266829740 1 -- start start 2024-04-21T08:47:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8266829740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x56527432c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8266829740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527432cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8266829740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274340ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8266829740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5652742918d0 con 0x56527433e8f0 2024-04-21T08:47:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8266829740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56527422ee80 con 0x56527432cb60 2024-04-21T08:47:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8266829740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56527428d890 con 0x56527432d8d0 2024-04-21T08:47:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x56527432c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x56527432c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45976/0 (socket says 172.21.15.57:45976) 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 -- 172.21.15.57:0/4158898722 learned_addr learned my addr 172.21.15.57:0/4158898722 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82618ee640 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527432cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274340ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8252ffd640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82480a3e40 0x7f82480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82620ef640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82480a3e40 0x7f82480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 -- 172.21.15.57:0/4158898722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 msgr2=0x56527432cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527432cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 -- 172.21.15.57:0/4158898722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 msgr2=0x565274340ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274340ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f8252ffd640 1 -- 172.21.15.57:0/838591088 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f82540d0fb0 con 0x7f825c19a720 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.686+0000 7f82610ed640 1 -- 172.21.15.57:0/4158898722 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565274250320 con 0x56527432d8d0 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.687+0000 7f82610ed640 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x56527432c620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f825c06b5d0 tx=0x7f824c005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e08ff08711b45481 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.687+0000 7f82620ef640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82480a3e40 0x7f82480a6300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f82540ce450 tx=0x7f825401bb30 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.687+0000 7f8247fff640 1 -- 172.21.15.57:0/4158898722 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c04a070 con 0x56527432d8d0 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.687+0000 7f8247fff640 1 -- 172.21.15.57:0/4158898722 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f824c02cc90 con 0x56527432d8d0 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.687+0000 7f8247fff640 1 -- 172.21.15.57:0/4158898722 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c004920 con 0x56527432d8d0 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 -- 172.21.15.57:0/4158898722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 msgr2=0x56527432c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x56527432c620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f825c06b5d0 tx=0x7f824c005e80 comp rx=0 tx=0).stop 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 -- 172.21.15.57:0/4158898722 shutdown_connections 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274340ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527432cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 --2- 172.21.15.57:0/4158898722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x56527432c620 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 -- 172.21.15.57:0/4158898722 >> 172.21.15.57:0/4158898722 conn(0x5652742f3c90 msgr2=0x565274327850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 -- 172.21.15.57:0/4158898722 shutdown_connections 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 -- 172.21.15.57:0/4158898722 wait complete. 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 -- start start 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.688+0000 7f8266829740 1 Processor -- start 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527433e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527432d8d0 0x56527433b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527433e8f0 0x56527433bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5652743415c0 con 0x56527432d8d0 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x565274219eb0 con 0x56527432cb60 2024-04-21T08:47:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x565274341740 con 0x56527433e8f0 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f8266829740 10 client.0 _ll_register_callbacks cb 0x565274246b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82618ee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527432d8d0 0x56527433b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527433e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527433e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:56802/0 (socket says 172.21.15.57:56802) 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 -- 172.21.15.57:0/3701481777 learned_addr learned my addr 172.21.15.57:0/3701481777 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527433e8f0 0x56527433bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 -- 172.21.15.57:0/3701481777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527433e8f0 msgr2=0x56527433bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527433e8f0 0x56527433bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 -- 172.21.15.57:0/3701481777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527432d8d0 msgr2=0x56527433b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527432d8d0 0x56527433b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.689+0000 7f82610ed640 1 -- 172.21.15.57:0/3701481777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565273fa4210 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82610ed640 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527433e270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f824c02c170 tx=0x7f824c0045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5766a48fc9e2a1e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82457fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82457fa640 1 -- 172.21.15.57:0/3701481777 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c04a040 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f824c02cc90 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82457fa640 1 -- 172.21.15.57:0/3701481777 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f824c004d10 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82457fa640 1 -- 172.21.15.57:0/3701481777 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f824c004f10 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 10 client.25655 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x565274070620 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82457fa640 1 -- 172.21.15.57:0/3701481777 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f824c02cc90 con 0x56527432cb60 2024-04-21T08:47:25.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f82457fa640 1 -- 172.21.15.57:0/3701481777 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f824c003cc0 con 0x56527432cb60 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 10 client.25655 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 20 client.25655 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.690+0000 7f8266829740 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56527434e740 0x565274350b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.691+0000 7f8266829740 4 client.25655 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.691+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652742a5880 con 0x56527434e740 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.691+0000 7f82618ee640 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56527434e740 0x565274350b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.691+0000 7f82457fa640 10 client.25655 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:25.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.691+0000 7f82618ee640 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56527434e740 0x565274350b30 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebda94c98755e2cf server_cookie=a452c8fadcc8fda7 in_seq=0 out_seq=0 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.692+0000 7f82457fa640 1 -- 172.21.15.57:0/3701481777 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652742a5880 con 0x56527434e740 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.692+0000 7f82457fa640 10 client.25655 handle_command_reply: tid=0 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 client.25655 shutdown 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 msgr2=0x56527433e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527433e270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f824c02c170 tx=0x7f824c0045d0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 shutdown_connections 2024-04-21T08:47:25.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56527434e740 0x565274350b30 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527433e8f0 0x56527433bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527432d8d0 0x56527433b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 --2- 172.21.15.57:0/3701481777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x56527433e270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f82457fa640 0 client.25655 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 >> 172.21.15.57:0/3701481777 conn(0x5652742f3c90 msgr2=0x5652743295a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 shutdown_connections 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 1 -- 172.21.15.57:0/3701481777 wait complete. 2024-04-21T08:47:25.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.693+0000 7f8266829740 20 client.25655 trim_cache size 0 max 16384 2024-04-21T08:47:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.711+0000 7f8266829740 1 Processor -- start 2024-04-21T08:47:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.711+0000 7f8266829740 1 -- start start 2024-04-21T08:47:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f8266829740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432cb60 0x56527433dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.716 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f8266829740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432d8d0 0x56527433e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f8266829740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274353b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f8266829740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1ab040 con 0x56527433e8f0 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f8266829740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1ad860 con 0x56527432d8d0 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f8266829740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1ab690 con 0x56527432cb60 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82610ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274353b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432d8d0 0x56527433e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82610ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274353b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45208/0 (socket says 172.21.15.57:45208) 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82610ed640 1 -- 172.21.15.57:0/3622177515 learned_addr learned my addr 172.21.15.57:0/3622177515 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82610ed640 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432cb60 0x56527433dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 -- 172.21.15.57:0/3622177515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432cb60 msgr2=0x56527433dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432cb60 0x56527433dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.717 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 -- 172.21.15.57:0/3622177515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 msgr2=0x565274353b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274353b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 -- 172.21.15.57:0/3622177515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527433bef0 con 0x56527432d8d0 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82610ed640 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274353b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82618ee640 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432d8d0 0x56527433e270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5652743417f0 tx=0x7f825800b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3962cfd426ea8f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.712+0000 7f82477fe640 1 -- 172.21.15.57:0/3622177515 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825800bc80 con 0x56527432d8d0 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f82477fe640 1 -- 172.21.15.57:0/3622177515 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f825800be20 con 0x56527432d8d0 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f82477fe640 1 -- 172.21.15.57:0/3622177515 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82580201b0 con 0x56527432d8d0 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 -- 172.21.15.57:0/3622177515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432d8d0 msgr2=0x56527433e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432d8d0 0x56527433e270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5652743417f0 tx=0x7f825800b390 comp rx=0 tx=0).stop 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 -- 172.21.15.57:0/3622177515 shutdown_connections 2024-04-21T08:47:25.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274353b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432d8d0 0x56527433e270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 --2- 172.21.15.57:0/3622177515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432cb60 0x56527433dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 -- 172.21.15.57:0/3622177515 >> 172.21.15.57:0/3622177515 conn(0x5652742f3c90 msgr2=0x565274327380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 -- 172.21.15.57:0/3622177515 shutdown_connections 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.713+0000 7f8266829740 1 -- 172.21.15.57:0/3622177515 wait complete. 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 -- start start 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 Processor -- start 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x565274353bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x565274350f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274351490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1abd20 con 0x56527433e8f0 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1ad190 con 0x56527432cb60 2024-04-21T08:47:25.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1ac3f0 con 0x56527432d8d0 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f8266829740 10 client.0 _ll_register_callbacks cb 0x7f824c048070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f82610ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x565274353bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f82610ed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x565274353bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:56830/0 (socket says 172.21.15.57:56830) 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f82610ed640 1 -- 172.21.15.57:0/2629244178 learned_addr learned my addr 172.21.15.57:0/2629244178 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f82610ed640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274351490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.714+0000 7f82618ee640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x565274350f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82610ed640 1 -- 172.21.15.57:0/2629244178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 msgr2=0x565274350f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82610ed640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x565274350f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82610ed640 1 -- 172.21.15.57:0/2629244178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 msgr2=0x565274351490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82610ed640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274351490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:25.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82610ed640 1 -- 172.21.15.57:0/2629244178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652743519d0 con 0x56527432cb60 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82610ed640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x565274353bc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f824c0438f0 tx=0x7f824c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3003b8c5c8a16fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82457fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82457fa640 1 -- 172.21.15.57:0/2629244178 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c04a070 con 0x56527432cb60 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825800be20 con 0x56527432cb60 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82457fa640 1 -- 172.21.15.57:0/2629244178 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f824c02c9f0 con 0x56527432cb60 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82457fa640 1 -- 172.21.15.57:0/2629244178 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f824c02cbf0 con 0x56527432cb60 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f8266829740 10 client.25661 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:25.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5652743b45e0 con 0x56527432cb60 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82457fa640 1 -- 172.21.15.57:0/2629244178 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f825800be20 con 0x56527432cb60 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.715+0000 7f82457fa640 1 -- 172.21.15.57:0/2629244178 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f824c0035f0 con 0x56527432cb60 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f8266829740 10 client.25661 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f8266829740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f8266829740 20 client.25661 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f8266829740 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5652743b4ac0 0x5652743b6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f8266829740 4 client.25661 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652742313d0 con 0x5652743b4ac0 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f82618ee640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5652743b4ac0 0x5652743b6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f82618ee640 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5652743b4ac0 0x5652743b6eb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1560929c98f0ce51 server_cookie=c9fa4cc477a61df8 in_seq=0 out_seq=0 2024-04-21T08:47:25.722 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:25.716+0000 7f82457fa640 10 client.25661 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:26 smithi129 ceph-mon[31272]: pgmap v63: 129 pgs: 129 active+clean; 218 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 1.11k op/s 2024-04-21T08:47:26.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:26 smithi057 ceph-mon[27968]: pgmap v63: 129 pgs: 129 active+clean; 218 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 1.11k op/s 2024-04-21T08:47:26.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:26 smithi136 ceph-mon[31791]: pgmap v63: 129 pgs: 129 active+clean; 218 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 1.11k op/s 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f82457fa640 1 -- 172.21.15.57:0/2629244178 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5652742313d0 con 0x5652743b4ac0 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f82457fa640 10 client.25661 handle_command_reply: tid=0 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 client.25661 shutdown 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 msgr2=0x565274353bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x565274353bc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f824c0438f0 tx=0x7f824c000bc0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 shutdown_connections 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5652743b4ac0 0x5652743b6eb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56527433e8f0 0x565274351490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56527432d8d0 0x565274350f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 --2- 172.21.15.57:0/2629244178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56527432cb60 0x565274353bc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f82457fa640 0 client.25661 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 >> 172.21.15.57:0/2629244178 conn(0x5652742f3c90 msgr2=0x565274349a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 shutdown_connections 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 1 -- 172.21.15.57:0/2629244178 wait complete. 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.382+0000 7f8266829740 20 client.25661 trim_cache size 0 max 16384 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:47:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:47:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:47:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82480a3e40 msgr2=0x7f82480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82480a3e40 0x7f82480a6300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f82540ce450 tx=0x7f825401bb30 comp rx=0 tx=0).stop 2024-04-21T08:47:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 msgr2=0x7f825c19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 0x7f825c19aba0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f825400eb10 tx=0x7f825400c2d0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 shutdown_connections 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f82480a3e40 0x7f82480a6300 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f825c19a720 0x7f825c19aba0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f825c107270 0x7f825c1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 --2- 172.21.15.57:0/838591088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c1021a0 0x7f825c1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 >> 172.21.15.57:0/838591088 conn(0x7f825c0fda00 msgr2=0x7f825c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 shutdown_connections 2024-04-21T08:47:28.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:28.383+0000 7f8263b52640 1 -- 172.21.15.57:0/838591088 wait complete. 2024-04-21T08:47:28.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:28 smithi136 ceph-mon[31791]: pgmap v64: 129 pgs: 129 active+clean; 221 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 1.13k op/s 2024-04-21T08:47:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:28 smithi129 ceph-mon[31272]: pgmap v64: 129 pgs: 129 active+clean; 221 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 1.13k op/s 2024-04-21T08:47:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:28 smithi057 ceph-mon[27968]: pgmap v64: 129 pgs: 129 active+clean; 221 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.8 MiB/s wr, 1.13k op/s 2024-04-21T08:47:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:30 smithi129 ceph-mon[31272]: pgmap v65: 129 pgs: 129 active+clean; 221 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 863 KiB/s wr, 28 op/s 2024-04-21T08:47:30.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:30 smithi057 ceph-mon[27968]: pgmap v65: 129 pgs: 129 active+clean; 221 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 863 KiB/s wr, 28 op/s 2024-04-21T08:47:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:30 smithi136 ceph-mon[31791]: pgmap v65: 129 pgs: 129 active+clean; 221 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 863 KiB/s wr, 28 op/s 2024-04-21T08:47:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:32 smithi129 ceph-mon[31272]: pgmap v66: 129 pgs: 129 active+clean; 217 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 986 KiB/s wr, 279 op/s 2024-04-21T08:47:32.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:32.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:32.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:32.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:32 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:47:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:32 smithi057 ceph-mon[27968]: pgmap v66: 129 pgs: 129 active+clean; 217 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 986 KiB/s wr, 279 op/s 2024-04-21T08:47:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:32 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:47:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:32 smithi136 ceph-mon[31791]: pgmap v66: 129 pgs: 129 active+clean; 217 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 986 KiB/s wr, 279 op/s 2024-04-21T08:47:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:47:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:47:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:47:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:32 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:47:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:34 smithi129 ceph-mon[31272]: pgmap v67: 129 pgs: 129 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 1.04k op/s 2024-04-21T08:47:34.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:34 smithi057 ceph-mon[27968]: pgmap v67: 129 pgs: 129 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 1.04k op/s 2024-04-21T08:47:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:34 smithi136 ceph-mon[31791]: pgmap v67: 129 pgs: 129 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 1.04k op/s 2024-04-21T08:47:36.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:36.774 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:36.800 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:36 smithi057 ceph-mon[27968]: pgmap v68: 129 pgs: 129 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 884 KiB/s wr, 1.03k op/s 2024-04-21T08:47:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:36 smithi129 ceph-mon[31272]: pgmap v68: 129 pgs: 129 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 884 KiB/s wr, 1.03k op/s 2024-04-21T08:47:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:36 smithi136 ceph-mon[31791]: pgmap v68: 129 pgs: 129 active+clean; 203 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 884 KiB/s wr, 1.03k op/s 2024-04-21T08:47:36.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/3392757453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e4107fd0 msgr2=0x7ff8e41083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/3392757453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e4107fd0 0x7ff8e41083b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4009900 tx=0x7ff8d402bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/3392757453 shutdown_connections 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/3392757453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e410ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/3392757453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8e41003e0 0x7ff8e4100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/3392757453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e4107fd0 0x7ff8e41083b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/3392757453 >> 172.21.15.57:0/3392757453 conn(0x7ff8e4077360 msgr2=0x7ff8e40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.893+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/3392757453 shutdown_connections 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.894+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/3392757453 wait complete. 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.894+0000 7ff8ea7b2640 1 Processor -- start 2024-04-21T08:47:36.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.894+0000 7ff8ea7b2640 1 -- start start 2024-04-21T08:47:36.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8ea7b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e41003e0 0x7ff8e4198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8ea7b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e4198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8ea7b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8e4107fd0 0x7ff8e419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8ea7b2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41121a0 con 0x7ff8e4100d60 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8ea7b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4112020 con 0x7ff8e41003e0 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8ea7b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4112320 con 0x7ff8e4107fd0 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e4198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e41003e0 0x7ff8e4198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e4198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43068/0 (socket says 172.21.15.57:43068) 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e37fe640 1 -- 172.21.15.57:0/674267238 learned_addr learned my addr 172.21.15.57:0/674267238 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8e4107fd0 0x7ff8e419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 -- 172.21.15.57:0/674267238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8e4107fd0 msgr2=0x7ff8e419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8e4107fd0 0x7ff8e419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 -- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 msgr2=0x7ff8e4198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e4198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 -- 172.21.15.57:0/674267238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e419d730 con 0x7ff8e41003e0 2024-04-21T08:47:36.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e4198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:36.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.895+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e41003e0 0x7ff8e4198740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4036040 tx=0x7ff8d402cd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.896+0000 7ff8e17fa640 1 -- 172.21.15.57:0/674267238 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4043020 con 0x7ff8e41003e0 2024-04-21T08:47:36.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.896+0000 7ff8e17fa640 1 -- 172.21.15.57:0/674267238 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d4004410 con 0x7ff8e41003e0 2024-04-21T08:47:36.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.896+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e419d9c0 con 0x7ff8e41003e0 2024-04-21T08:47:36.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.896+0000 7ff8e17fa640 1 -- 172.21.15.57:0/674267238 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4003b30 con 0x7ff8e41003e0 2024-04-21T08:47:36.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.896+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a52b0 con 0x7ff8e41003e0 2024-04-21T08:47:36.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 Processor -- start 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 -- start start 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8e17fa640 1 -- 172.21.15.57:0/674267238 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff8d4046020 con 0x7ff8e41003e0 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b68c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b695540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b695a80 0x55dd7b697e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b5f11e0 con 0x55dd7b68c580 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b58e390 con 0x55dd7b68cea0 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b5ed1a0 con 0x55dd7b695a80 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b68c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b68c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43084/0 (socket says 172.21.15.57:43084) 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.898+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3022685351 learned_addr learned my addr 172.21.15.57:0/3022685351 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b695a80 0x55dd7b697e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b695540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3022685351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b695a80 msgr2=0x55dd7b697e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b695a80 0x55dd7b697e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3022685351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 msgr2=0x55dd7b695540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b695540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3022685351 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7b3cd7d0 con 0x55dd7b68c580 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b695540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b68c960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009f50 tx=0x7ff8dc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd79f4d92d8ee2eb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e17fa640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff8cc0a3d20 0x7ff8cc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8ca7fc640 1 -- 172.21.15.57:0/3022685351 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc004d20 con 0x55dd7b68c580 2024-04-21T08:47:36.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff8cc0a3d20 0x7ff8cc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8ca7fc640 1 -- 172.21.15.57:0/3022685351 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8dc004050 con 0x55dd7b68c580 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8ca7fc640 1 -- 172.21.15.57:0/3022685351 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc005280 con 0x55dd7b68c580 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.899+0000 7ff8e17fa640 1 -- 172.21.15.57:0/674267238 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff8d40f26a0 con 0x7ff8e41003e0 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3022685351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 msgr2=0x55dd7b68c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b68c960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009f50 tx=0x7ff8dc0118d0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8e3fff640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff8cc0a3d20 0x7ff8cc0a61e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40efc50 tx=0x7ff8d404e000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3022685351 shutdown_connections 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b695a80 0x55dd7b697e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b695540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3022685351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b68c960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3022685351 >> 172.21.15.57:0/3022685351 conn(0x55dd7b686cb0 msgr2=0x55dd7b687090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3022685351 shutdown_connections 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3022685351 wait complete. 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.900+0000 7ff8ed3a0740 1 -- start start 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 Processor -- start 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b692a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b692f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b6934d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b698870 con 0x55dd7b695a80 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b5aa2f0 con 0x55dd7b68cea0 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55dd7b6989f0 con 0x55dd7b68c580 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8ed3a0740 10 client.0 _ll_register_callbacks cb 0x55dd7b5b7580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b6934d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b6934d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43098/0 (socket says 172.21.15.57:43098) 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b692f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/482976716 learned_addr learned my addr 172.21.15.57:0/482976716 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b692a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e37fe640 1 -- 172.21.15.57:0/482976716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 msgr2=0x55dd7b692a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b692a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e37fe640 1 -- 172.21.15.57:0/482976716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 msgr2=0x55dd7b6934d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b6934d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.901+0000 7ff8e37fe640 1 -- 172.21.15.57:0/482976716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7b3033f0 con 0x55dd7b68cea0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b692f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40fe350 tx=0x7ff8d800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f26d2c93f406a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8c37fe640 1 -- 172.21.15.57:0/482976716 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80120d0 con 0x55dd7b68cea0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8dc004050 con 0x55dd7b68cea0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8c37fe640 1 -- 172.21.15.57:0/482976716 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff8d8005020 con 0x55dd7b68cea0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8c37fe640 1 -- 172.21.15.57:0/482976716 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff8d8012270 con 0x55dd7b68cea0 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8ed3a0740 10 client.25673 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:36.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dd7b2c30f0 con 0x55dd7b68cea0 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8c37fe640 1 -- 172.21.15.57:0/482976716 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8dc004050 con 0x55dd7b68cea0 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.902+0000 7ff8c37fe640 1 -- 172.21.15.57:0/482976716 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff8d801c470 con 0x55dd7b68cea0 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8ed3a0740 10 client.25673 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8ed3a0740 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8ed3a0740 20 client.25673 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b6a9f80 0x55dd7b6ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8ed3a0740 4 client.25673 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd7b5fec00 con 0x55dd7b6a9f80 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b6a9f80 0x55dd7b6ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8c37fe640 10 client.25673 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.903+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b6a9f80 0x55dd7b6ac3a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3000a12232dda2f server_cookie=e22bf4fa5f147865 in_seq=0 out_seq=0 2024-04-21T08:47:36.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8c37fe640 1 -- 172.21.15.57:0/482976716 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd7b5fec00 con 0x55dd7b6a9f80 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8c37fe640 10 client.25673 handle_command_reply: tid=0 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 client.25673 shutdown 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 msgr2=0x55dd7b692f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b692f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40fe350 tx=0x7ff8d800f790 comp rx=0 tx=0).stop 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 shutdown_connections 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b6a9f80 0x55dd7b6ac3a0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b6934d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b692f90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8c37fe640 0 client.25673 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/482976716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b692a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 >> 172.21.15.57:0/482976716 conn(0x55dd7b686cb0 msgr2=0x55dd7b688a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.904+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 shutdown_connections 2024-04-21T08:47:36.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.905+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/482976716 wait complete. 2024-04-21T08:47:36.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.905+0000 7ff8ed3a0740 20 client.25673 trim_cache size 0 max 16384 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 Processor -- start 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 -- start start 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b691990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68cea0 0x55dd7b695180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b695a80 0x55dd7b6932f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41ab030 con 0x55dd7b68c580 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41ad890 con 0x55dd7b695a80 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.923+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41ab680 con 0x55dd7b68cea0 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b691990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b691990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43112/0 (socket says 172.21.15.57:43112) 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3376375008 learned_addr learned my addr 172.21.15.57:0/3376375008 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:36.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68cea0 0x55dd7b695180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b695a80 0x55dd7b6932f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3376375008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68cea0 msgr2=0x55dd7b695180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68cea0 0x55dd7b695180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3376375008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b695a80 msgr2=0x55dd7b6932f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b695a80 0x55dd7b6932f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3376375008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd7b693e40 con 0x55dd7b68c580 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b691990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc005fd0 tx=0x7ff8dc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40848d627aab4d1a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8c9ffb640 1 -- 172.21.15.57:0/3376375008 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc003a30 con 0x55dd7b68c580 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8c9ffb640 1 -- 172.21.15.57:0/3376375008 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8dc004e40 con 0x55dd7b68c580 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.924+0000 7ff8c9ffb640 1 -- 172.21.15.57:0/3376375008 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc005560 con 0x55dd7b68c580 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3376375008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 msgr2=0x55dd7b691990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b691990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc005fd0 tx=0x7ff8dc009500 comp rx=0 tx=0).stop 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3376375008 shutdown_connections 2024-04-21T08:47:36.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b695a80 0x55dd7b6932f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68cea0 0x55dd7b695180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3376375008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b68c580 0x55dd7b691990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3376375008 >> 172.21.15.57:0/3376375008 conn(0x55dd7b686cb0 msgr2=0x55dd7b685d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3376375008 shutdown_connections 2024-04-21T08:47:36.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3376375008 wait complete. 2024-04-21T08:47:36.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.925+0000 7ff8ed3a0740 1 -- start start 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 Processor -- start 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b6a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b69d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b69dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41abd50 con 0x55dd7b695a80 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41ad1c0 con 0x55dd7b68cea0 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e41ac420 con 0x55dd7b68c580 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8ed3a0740 10 client.0 _ll_register_callbacks cb 0x7ff8d8012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b69dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b69d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b69dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43116/0 (socket says 172.21.15.57:43116) 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3085972781 learned_addr learned my addr 172.21.15.57:0/3085972781 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b6a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3085972781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 msgr2=0x55dd7b6a24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b6a24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 msgr2=0x55dd7b69d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b69d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.926+0000 7ff8e8d4f640 1 -- 172.21.15.57:0/3085972781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd7b69e530 con 0x55dd7b695a80 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8e8d4f640 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b69dd00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0050e0 tx=0x7ff8dc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fd7531ae3cdf13 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8c37fe640 1 -- 172.21.15.57:0/3085972781 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc041060 con 0x55dd7b695a80 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8c37fe640 1 -- 172.21.15.57:0/3085972781 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff8dc004b90 con 0x55dd7b695a80 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dd7b5f8a70 con 0x55dd7b695a80 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8c37fe640 1 -- 172.21.15.57:0/3085972781 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff8dc033040 con 0x55dd7b695a80 2024-04-21T08:47:36.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 10 client.16170 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dd7b710500 con 0x55dd7b695a80 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8c37fe640 1 -- 172.21.15.57:0/3085972781 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55dd7b5f8a70 con 0x55dd7b695a80 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8c37fe640 1 -- 172.21.15.57:0/3085972781 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff8dc039070 con 0x55dd7b695a80 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 10 client.16170 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 20 client.16170 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b7109c0 0x55dd7b712de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 4 client.16170 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.927+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd7b574590 con 0x55dd7b7109c0 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.928+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b7109c0 0x55dd7b712de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.928+0000 7ff8e37fe640 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b7109c0 0x55dd7b712de0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec14136e83ebf58 server_cookie=1d685a96668592d2 in_seq=0 out_seq=0 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.928+0000 7ff8c37fe640 10 client.16170 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.931+0000 7ff8c37fe640 1 -- 172.21.15.57:0/3085972781 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dd7b574590 con 0x55dd7b7109c0 2024-04-21T08:47:36.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.931+0000 7ff8c37fe640 10 client.16170 handle_command_reply: tid=0 2024-04-21T08:47:36.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.931+0000 7ff8ed3a0740 1 client.16170 shutdown 2024-04-21T08:47:36.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.931+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 msgr2=0x55dd7b69dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.934 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.931+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b69dd00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0050e0 tx=0x7ff8dc000f40 comp rx=0 tx=0).stop 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 shutdown_connections 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dd7b7109c0 0x55dd7b712de0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dd7b695a80 0x55dd7b69dd00 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8c37fe640 0 client.16170 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dd7b68cea0 0x55dd7b69d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 --2- 172.21.15.57:0/3085972781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dd7b68c580 0x55dd7b6a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 >> 172.21.15.57:0/3085972781 conn(0x55dd7b686cb0 msgr2=0x55dd7b687fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 shutdown_connections 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 1 -- 172.21.15.57:0/3085972781 wait complete. 2024-04-21T08:47:36.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.932+0000 7ff8ed3a0740 20 client.16170 trim_cache size 0 max 16384 2024-04-21T08:47:36.936 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:47:36.936 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:47:36.936 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:47:36.936 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff8cc0a3d20 msgr2=0x7ff8cc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff8cc0a3d20 0x7ff8cc0a61e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40efc50 tx=0x7ff8d404e000 comp rx=0 tx=0).stop 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e41003e0 msgr2=0x7ff8e4198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e41003e0 0x7ff8e4198740 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4036040 tx=0x7ff8d402cd20 comp rx=0 tx=0).stop 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 shutdown_connections 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff8cc0a3d20 0x7ff8cc0a61e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8e4107fd0 0x7ff8e419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8e4100d60 0x7ff8e4198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 --2- 172.21.15.57:0/674267238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8e41003e0 0x7ff8e4198740 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 >> 172.21.15.57:0/674267238 conn(0x7ff8e4077360 msgr2=0x7ff8e40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 shutdown_connections 2024-04-21T08:47:36.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:36.933+0000 7ff8ea7b2640 1 -- 172.21.15.57:0/674267238 wait complete. 2024-04-21T08:47:36.944 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:47:36.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 -- 172.21.15.57:0/2556005542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa01011a0 msgr2=0x7f1aa0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/2556005542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa01011a0 0x7f1aa0101620 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009860 tx=0x7f1a9402bb70 comp rx=0 tx=0).stop 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 -- 172.21.15.57:0/2556005542 shutdown_connections 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/2556005542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/2556005542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa01011a0 0x7f1aa0101620 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/2556005542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa0105e30 0x7f1aa0106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 -- 172.21.15.57:0/2556005542 >> 172.21.15.57:0/2556005542 conn(0x7f1aa0077e30 msgr2=0x7f1aa0078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 -- 172.21.15.57:0/2556005542 shutdown_connections 2024-04-21T08:47:37.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.089+0000 7f1aa4b67640 1 -- 172.21.15.57:0/2556005542 wait complete. 2024-04-21T08:47:37.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 Processor -- start 2024-04-21T08:47:37.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 -- start start 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa01011a0 0x7f1aa01995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa0199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 0x7f1aa01a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa01110e0 con 0x7f1aa0101b60 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa0110f60 con 0x7f1aa0105e30 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1aa4b67640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa0111260 con 0x7f1aa01011a0 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1a9dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa0199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1a9ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 0x7f1aa01a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1a9ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 0x7f1aa01a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40196/0 (socket says 172.21.15.57:40196) 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1a9ed9d640 1 -- 172.21.15.57:0/3781977913 learned_addr learned my addr 172.21.15.57:0/3781977913 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1a9dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa0199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43132/0 (socket says 172.21.15.57:43132) 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.090+0000 7f1a9e59c640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa01011a0 0x7f1aa01995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9ed9d640 1 -- 172.21.15.57:0/3781977913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa01011a0 msgr2=0x7f1aa01995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9ed9d640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa01011a0 0x7f1aa01995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9ed9d640 1 -- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 msgr2=0x7f1aa0199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9ed9d640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa0199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9ed9d640 1 -- 172.21.15.57:0/3781977913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa01a4ba0 con 0x7f1aa0105e30 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9dd9b640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa0199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a9ed9d640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 0x7f1aa01a4660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800efc0 tx=0x7f1a9800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98003970 con 0x7f1aa0105e30 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa01a4d70 con 0x7f1aa0105e30 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a98004020 con 0x7f1aa0105e30 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800b940 con 0x7f1aa0105e30 2024-04-21T08:47:37.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.091+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa01a5230 con 0x7f1aa0105e30 2024-04-21T08:47:37.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.092+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa0068de0 con 0x7f1aa0105e30 2024-04-21T08:47:37.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.093+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1a9801d020 con 0x7f1aa0105e30 2024-04-21T08:47:37.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.094+0000 7f1a8f7fe640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1a880a3d20 0x7f1a880a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.094+0000 7f1a9e59c640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1a880a3d20 0x7f1a880a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.094+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1a980d10d0 con 0x7f1aa0105e30 2024-04-21T08:47:37.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.095+0000 7f1a9e59c640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1a880a3d20 0x7f1a880a61e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009ae0 tx=0x7f1a90005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.102+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a9809d3b0 con 0x7f1aa0105e30 2024-04-21T08:47:37.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.332+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aa0109350 con 0x7f1aa0105e30 2024-04-21T08:47:37.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.333+0000 7f1a8f7fe640 1 -- 172.21.15.57:0/3781977913 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f1a9809ccb0 con 0x7f1aa0105e30 2024-04-21T08:47:37.333 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:47:37.333 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:47:37.333 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:47:37.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1a880a3d20 msgr2=0x7f1a880a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1a880a3d20 0x7f1a880a61e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009ae0 tx=0x7f1a90005f90 comp rx=0 tx=0).stop 2024-04-21T08:47:37.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 msgr2=0x7f1aa01a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.335 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 0x7f1aa01a4660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800efc0 tx=0x7f1a9800c800 comp rx=0 tx=0).stop 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 shutdown_connections 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1a880a3d20 0x7f1a880a61e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa0105e30 0x7f1aa01a4660 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aa0101b60 0x7f1aa0199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 --2- 172.21.15.57:0/3781977913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa01011a0 0x7f1aa01995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 >> 172.21.15.57:0/3781977913 conn(0x7f1aa0077e30 msgr2=0x7f1aa00fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 shutdown_connections 2024-04-21T08:47:37.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.336+0000 7f1aa4b67640 1 -- 172.21.15.57:0/3781977913 wait complete. 2024-04-21T08:47:37.346 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:47:37.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/1822201775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad41031e0 msgr2=0x7f1ad41035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/1822201775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad41031e0 0x7f1ad41035c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009900 tx=0x7f1ac402bd60 comp rx=0 tx=0).stop 2024-04-21T08:47:37.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/1822201775 shutdown_connections 2024-04-21T08:47:37.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/1822201775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad410c520 0x7f1ad410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/1822201775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad4103b00 0x7f1ad410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/1822201775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad41031e0 0x7f1ad41035c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/1822201775 >> 172.21.15.57:0/1822201775 conn(0x7f1ad40771d0 msgr2=0x7f1ad40775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/1822201775 shutdown_connections 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.454+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/1822201775 wait complete. 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 Processor -- start 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 -- start start 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad41031e0 0x7f1ad4198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad4103b00 0x7f1ad4198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 0x7f1ad419cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad4112200 con 0x7f1ad410c520 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad4112080 con 0x7f1ad4103b00 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad9e3a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad4112380 con 0x7f1ad41031e0 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.455+0000 7f1ad3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 0x7f1ad419cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 0x7f1ad419cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43156/0 (socket says 172.21.15.57:43156) 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad41031e0 0x7f1ad4198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 -- 172.21.15.57:0/3179064473 learned_addr learned my addr 172.21.15.57:0/3179064473 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:37.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad4103b00 0x7f1ad4198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 -- 172.21.15.57:0/3179064473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad41031e0 msgr2=0x7f1ad4198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad41031e0 0x7f1ad4198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 -- 172.21.15.57:0/3179064473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad4103b00 msgr2=0x7f1ad4198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad4103b00 0x7f1ad4198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 -- 172.21.15.57:0/3179064473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad419d710 con 0x7f1ad410c520 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad3fff640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 0x7f1ad419cfe0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00eb10 tx=0x7f1acc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad0ff9640 1 -- 172.21.15.57:0/3179064473 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc004b20 con 0x7f1ad410c520 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad0ff9640 1 -- 172.21.15.57:0/3179064473 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1acc004cc0 con 0x7f1ad410c520 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad419da00 con 0x7f1ad410c520 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad0ff9640 1 -- 172.21.15.57:0/3179064473 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc010690 con 0x7f1ad410c520 2024-04-21T08:47:37.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.456+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad41a5290 con 0x7f1ad410c520 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad0ff9640 1 -- 172.21.15.57:0/3179064473 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1acc005020 con 0x7f1ad410c520 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 Processor -- start 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 -- start start 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 0x55f60b88f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88fd60 0x55f60b898890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8992c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55f60b7f41f0 con 0x55f60b898e60 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f60b7914b0 con 0x55f60b88f3b0 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1adca1f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55f60b7f01b0 con 0x55f60b88fd60 2024-04-21T08:47:37.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 0x55f60b88f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 0x55f60b88f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40222/0 (socket says 172.21.15.57:40222) 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/3971768592 learned_addr learned my addr 172.21.15.57:0/3971768592 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88fd60 0x55f60b898890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad0ff9640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1abc0a3d20 0x7f1abc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/3971768592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88fd60 msgr2=0x55f60b898890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.459+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88fd60 0x55f60b898890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/3971768592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 msgr2=0x55f60b8992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/3971768592 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f60b7831b0 con 0x55f60b88f3b0 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad3fff640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1abc0a3d20 0x7f1abc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad0ff9640 1 -- 172.21.15.57:0/3179064473 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1acc0d1030 con 0x7f1ad410c520 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 0x55f60b88f790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800b6c0 tx=0x7f1ac800b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bca58773bc2c523a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ab9ffb640 1 -- 172.21.15.57:0/3971768592 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8012d00 con 0x55f60b88f3b0 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ad3fff640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1abc0a3d20 0x7f1abc0a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0ce4d0 tx=0x7f1acc00efd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ab9ffb640 1 -- 172.21.15.57:0/3971768592 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac8010770 con 0x55f60b88f3b0 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.460+0000 7f1ab9ffb640 1 -- 172.21.15.57:0/3971768592 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8003850 con 0x55f60b88f3b0 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- 172.21.15.57:0/3971768592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 msgr2=0x55f60b88f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 0x55f60b88f790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800b6c0 tx=0x7f1ac800b650 comp rx=0 tx=0).stop 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- 172.21.15.57:0/3971768592 shutdown_connections 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88fd60 0x55f60b898890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- 172.21.15.57:0/3971768592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88f3b0 0x55f60b88f790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- 172.21.15.57:0/3971768592 >> 172.21.15.57:0/3971768592 conn(0x55f60b889ba0 msgr2=0x55f60b889f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- 172.21.15.57:0/3971768592 shutdown_connections 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- 172.21.15.57:0/3971768592 wait complete. 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- start start 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 Processor -- start 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b89f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 0x55f60b89c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b898e60 0x55f60b89cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55f60b89fda0 con 0x55f60b88fd60 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f60b7ad500 con 0x55f60b898e60 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.461+0000 7f1adca1f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55f60b89ff20 con 0x55f60b88f3b0 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1adca1f740 10 client.0 _ll_register_callbacks cb 0x55f60b77bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 0x55f60b89c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 0x55f60b89c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43174/0 (socket says 172.21.15.57:43174) 2024-04-21T08:47:37.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2809324454 learned_addr learned my addr 172.21.15.57:0/2809324454 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b898e60 0x55f60b89cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b89f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2809324454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 msgr2=0x55f60b89f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b89f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b898e60 msgr2=0x55f60b89cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b898e60 0x55f60b89cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2809324454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f60b5063f0 con 0x55f60b88fd60 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 0x55f60b89c630 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4006010 tx=0x7f1ac402db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a85e0460c78ffde server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ab2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2809324454 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4044030 con 0x55f60b88fd60 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2809324454 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1ac403a040 con 0x55f60b88fd60 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac8010770 con 0x55f60b88fd60 2024-04-21T08:47:37.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2809324454 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ac4051050 con 0x55f60b88fd60 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1adca1f740 10 client.16188 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.462+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f60b5d07d0 con 0x55f60b88fd60 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2809324454 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac8010770 con 0x55f60b88fd60 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2809324454 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1ac4004d00 con 0x55f60b88fd60 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1adca1f740 10 client.16188 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1adca1f740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1adca1f740 20 client.16188 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b8ad090 0x55f60b8af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1adca1f740 4 client.16188 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f60b793930 con 0x55f60b8ad090 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b8ad090 0x55f60b8af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1ab2ffd640 10 client.16188 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:37.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.463+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b8ad090 0x55f60b8af4a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f605a710023b227 server_cookie=27bb4d7854cd0fe in_seq=0 out_seq=0 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.464+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2809324454 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f60b793930 con 0x55f60b8ad090 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.464+0000 7f1ab2ffd640 10 client.16188 handle_command_reply: tid=0 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.464+0000 7f1adca1f740 1 client.16188 shutdown 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 msgr2=0x55f60b89c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 0x55f60b89c630 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4006010 tx=0x7f1ac402db50 comp rx=0 tx=0).stop 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 shutdown_connections 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b8ad090 0x55f60b8af4a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b898e60 0x55f60b89cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88fd60 0x55f60b89c630 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1ab2ffd640 0 client.16188 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2809324454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b89f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 >> 172.21.15.57:0/2809324454 conn(0x55f60b889ba0 msgr2=0x55f60b88b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 shutdown_connections 2024-04-21T08:47:37.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 1 -- 172.21.15.57:0/2809324454 wait complete. 2024-04-21T08:47:37.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.465+0000 7f1adca1f740 20 client.16188 trim_cache size 0 max 16384 2024-04-21T08:47:37.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 Processor -- start 2024-04-21T08:47:37.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 -- start start 2024-04-21T08:47:37.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 0x55f60b89a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8a8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b898e60 0x55f60b89b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad41ab090 con 0x55f60b88f3b0 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad41ad8f0 con 0x55f60b88fd60 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1adca1f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad41ab6e0 con 0x55f60b898e60 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1ad2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 0x55f60b89a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1ad2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 0x55f60b89a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43178/0 (socket says 172.21.15.57:43178) 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/4087514497 learned_addr learned my addr 172.21.15.57:0/4087514497 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8a8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b898e60 0x55f60b89b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.479+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/4087514497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b898e60 msgr2=0x55f60b89b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b898e60 0x55f60b89b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/4087514497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 msgr2=0x55f60b8a8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8a8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ad2ffd640 1 -- 172.21.15.57:0/4087514497 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f60b89cf00 con 0x55f60b88f3b0 2024-04-21T08:47:37.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8a8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 0x55f60b89a250 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800b6c0 tx=0x7f1ac8017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9650fe0ae5a39ef server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ab97fa640 1 -- 172.21.15.57:0/4087514497 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800baa0 con 0x55f60b88f3b0 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ab97fa640 1 -- 172.21.15.57:0/4087514497 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac8016070 con 0x55f60b88f3b0 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1ab97fa640 1 -- 172.21.15.57:0/4087514497 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8004280 con 0x55f60b88f3b0 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1adca1f740 1 -- 172.21.15.57:0/4087514497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 msgr2=0x55f60b89a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.480+0000 7f1adca1f740 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 0x55f60b89a250 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800b6c0 tx=0x7f1ac8017040 comp rx=0 tx=0).stop 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- 172.21.15.57:0/4087514497 shutdown_connections 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b898e60 0x55f60b89b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8a8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 --2- 172.21.15.57:0/4087514497 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b88f3b0 0x55f60b89a250 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- 172.21.15.57:0/4087514497 >> 172.21.15.57:0/4087514497 conn(0x55f60b889ba0 msgr2=0x55f60b89af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- 172.21.15.57:0/4087514497 shutdown_connections 2024-04-21T08:47:37.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- 172.21.15.57:0/4087514497 wait complete. 2024-04-21T08:47:37.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- start start 2024-04-21T08:47:37.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 Processor -- start 2024-04-21T08:47:37.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b8a5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8b4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad41abdb0 con 0x55f60b898e60 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad41ad220 con 0x55f60b88fd60 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad41ac480 con 0x55f60b88f3b0 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1adca1f740 10 client.0 _ll_register_callbacks cb 0x7f1ac4004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.481+0000 7f1ad2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40242/0 (socket says 172.21.15.57:40242) 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2514479233 learned_addr learned my addr 172.21.15.57:0/2514479233 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b8a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2514479233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 msgr2=0x55f60b8a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b8a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2514479233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 msgr2=0x55f60b8b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 -- 172.21.15.57:0/2514479233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f60b8b4f90 con 0x55f60b88fd60 2024-04-21T08:47:37.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ad37fe640 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8b4510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55f60b89dc00 tx=0x7f1ac4003f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a14d1b99157deb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ab2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2514479233 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4044070 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ac8016070 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2514479233 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1ac4005250 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2514479233 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ac405e050 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1adca1f740 10 client.25703 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.482+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55f60b917dd0 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2514479233 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac8016070 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2514479233 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1ac4004b70 con 0x55f60b88fd60 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1adca1f740 10 client.25703 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1adca1f740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1adca1f740 20 client.25703 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b918370 0x55f60b91a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1adca1f740 4 client.25703 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:47:37.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f60b801c10 con 0x55f60b918370 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b918370 0x55f60b91a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1ad2ffd640 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b918370 0x55f60b91a790 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2e2442c9e4dd2c1 server_cookie=a5c2f082abad56 in_seq=0 out_seq=0 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.483+0000 7f1ab2ffd640 10 client.25703 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1ab2ffd640 1 -- 172.21.15.57:0/2514479233 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f60b801c10 con 0x55f60b918370 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1ab2ffd640 10 client.25703 handle_command_reply: tid=0 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 client.25703 shutdown 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 msgr2=0x55f60b8b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8b4510 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55f60b89dc00 tx=0x7f1ac4003f40 comp rx=0 tx=0).stop 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 shutdown_connections 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55f60b918370 0x55f60b91a790 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55f60b898e60 0x55f60b8b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1ab2ffd640 0 client.25703 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f60b88fd60 0x55f60b8b4510 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 --2- 172.21.15.57:0/2514479233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55f60b88f3b0 0x55f60b8a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 >> 172.21.15.57:0/2514479233 conn(0x55f60b889ba0 msgr2=0x55f60b88af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 shutdown_connections 2024-04-21T08:47:37.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 1 -- 172.21.15.57:0/2514479233 wait complete. 2024-04-21T08:47:37.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.484+0000 7f1adca1f740 20 client.25703 trim_cache size 0 max 16384 2024-04-21T08:47:37.490 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1abc0a3d20 msgr2=0x7f1abc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1abc0a3d20 0x7f1abc0a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0ce4d0 tx=0x7f1acc00efd0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 msgr2=0x7f1ad419cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 0x7f1ad419cfe0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1acc00eb10 tx=0x7f1acc00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 shutdown_connections 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1abc0a3d20 0x7f1abc0a61e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ad410c520 0x7f1ad419cfe0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad4103b00 0x7f1ad4198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 --2- 172.21.15.57:0/3179064473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad41031e0 0x7f1ad4198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 >> 172.21.15.57:0/3179064473 conn(0x7f1ad40771d0 msgr2=0x7f1ad40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 shutdown_connections 2024-04-21T08:47:37.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:37.485+0000 7f1ad9e3a640 1 -- 172.21.15.57:0/3179064473 wait complete. 2024-04-21T08:47:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:37 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3781977913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:37.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:37 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3781977913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:37 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3781977913' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:38.496 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:47:38.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:47:38.496 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 -- 172.21.15.57:0/2797046529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c18102bc0 msgr2=0x7f3c18106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 --2- 172.21.15.57:0/2797046529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c18102bc0 0x7f3c18106c00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009900 tx=0x7f3c0802bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 -- 172.21.15.57:0/2797046529 shutdown_connections 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 --2- 172.21.15.57:0/2797046529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c181072e0 0x7f3c1810eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 --2- 172.21.15.57:0/2797046529 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c18102bc0 0x7f3c18106c00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 --2- 172.21.15.57:0/2797046529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c18102210 0x7f3c181025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 -- 172.21.15.57:0/2797046529 >> 172.21.15.57:0/2797046529 conn(0x7f3c180fdab0 msgr2=0x7f3c180ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 -- 172.21.15.57:0/2797046529 shutdown_connections 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.606+0000 7f3c20225640 1 -- 172.21.15.57:0/2797046529 wait complete. 2024-04-21T08:47:38.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 Processor -- start 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 -- start start 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c18102210 0x7f3c1819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 0x7f3c1819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c18198690 0x7f3c18198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18112250 con 0x7f3c181072e0 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181120d0 con 0x7f3c18198690 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c20225640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181123d0 con 0x7f3c18102210 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 0x7f3c1819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 0x7f3c1819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43204/0 (socket says 172.21.15.57:43204) 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 -- 172.21.15.57:0/870728431 learned_addr learned my addr 172.21.15.57:0/870728431 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1e7c2640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c18198690 0x7f3c18198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c18102210 0x7f3c1819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 -- 172.21.15.57:0/870728431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c18102210 msgr2=0x7f3c1819e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c18102210 0x7f3c1819e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.609 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 -- 172.21.15.57:0/870728431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c18198690 msgr2=0x7f3c18198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c18198690 0x7f3c18198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 -- 172.21.15.57:0/870728431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c18199290 con 0x7f3c181072e0 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.607+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 0x7f3c1819eac0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3c08000c00 tx=0x7f3c080045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.608+0000 7f3c0effd640 1 -- 172.21.15.57:0/870728431 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08043020 con 0x7f3c181072e0 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.608+0000 7f3c0effd640 1 -- 172.21.15.57:0/870728431 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0802cb70 con 0x7f3c181072e0 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.608+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c18199520 con 0x7f3c181072e0 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.608+0000 7f3c0effd640 1 -- 172.21.15.57:0/870728431 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08003b80 con 0x7f3c181072e0 2024-04-21T08:47:38.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.608+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c181a56c0 con 0x7f3c181072e0 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c0effd640 1 -- 172.21.15.57:0/870728431 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3c08046020 con 0x7f3c181072e0 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 Processor -- start 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 -- start start 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df0660 0x561824df0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 0x561824dfdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824e00630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561824d54f00 con 0x561824df1010 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561824cf2590 con 0x561824df0660 2024-04-21T08:47:38.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c22e11740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561824d50ec0 con 0x561824dfe220 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 0x561824dfdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c1e7c2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df0660 0x561824df0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 0x561824dfdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43218/0 (socket says 172.21.15.57:43218) 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3421487824 learned_addr learned my addr 172.21.15.57:0/3421487824 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.611+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824e00630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c0effd640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3c040a3da0 0x7f3c040a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3421487824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 msgr2=0x561824e00630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824e00630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3421487824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df0660 msgr2=0x561824df0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df0660 0x561824df0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3421487824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561824b2f7e0 con 0x561824df1010 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824e00630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1e7c2640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3c040a3da0 0x7f3c040a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c0effd640 1 -- 172.21.15.57:0/870728431 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f3c080f2a10 con 0x7f3c181072e0 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 0x561824dfdc50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3c140114f0 tx=0x7f3c140119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19e4dbbf98d0370 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c03fff640 1 -- 172.21.15.57:0/3421487824 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1401e070 con 0x561824df1010 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c1e7c2640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3c040a3da0 0x7f3c040a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3c10005fd0 tx=0x7f3c100046c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c03fff640 1 -- 172.21.15.57:0/3421487824 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c1400fad0 con 0x561824df1010 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.612+0000 7f3c03fff640 1 -- 172.21.15.57:0/3421487824 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14019b10 con 0x561824df1010 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 -- 172.21.15.57:0/3421487824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 msgr2=0x561824dfdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 0x561824dfdc50 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3c140114f0 tx=0x7f3c140119b0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 -- 172.21.15.57:0/3421487824 shutdown_connections 2024-04-21T08:47:38.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824e00630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df1010 0x561824dfdc50 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3421487824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df0660 0x561824df0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 -- 172.21.15.57:0/3421487824 >> 172.21.15.57:0/3421487824 conn(0x561824deafb0 msgr2=0x561824deb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 -- 172.21.15.57:0/3421487824 shutdown_connections 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 -- 172.21.15.57:0/3421487824 wait complete. 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 -- start start 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.613+0000 7f3c22e11740 1 Processor -- start 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824dfb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561824e01150 con 0x561824df0660 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561824d0e580 con 0x561824df1010 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561824e012d0 con 0x561824dfe220 2024-04-21T08:47:38.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c22e11740 10 client.0 _ll_register_callbacks cb 0x561824d14650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43228/0 (socket says 172.21.15.57:43228) 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1840281847 learned_addr learned my addr 172.21.15.57:0/1840281847 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824dfb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1e7c2640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1840281847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 msgr2=0x561824dfbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 msgr2=0x561824dfb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824dfb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1840281847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561824b8ff70 con 0x561824df0660 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfb1f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3c1401c040 tx=0x7f3c14009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ace1824d5ee98e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c017fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.614+0000 7f3c017fa640 1 -- 172.21.15.57:0/1840281847 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14028020 con 0x561824df0660 2024-04-21T08:47:38.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c017fa640 1 -- 172.21.15.57:0/1840281847 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f3c14004a40 con 0x561824df0660 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c1400fad0 con 0x561824df0660 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c017fa640 1 -- 172.21.15.57:0/1840281847 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3c14030050 con 0x561824df0660 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 10 client.16206 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561824a27170 con 0x561824df0660 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c017fa640 1 -- 172.21.15.57:0/1840281847 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c1400fad0 con 0x561824df0660 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c017fa640 1 -- 172.21.15.57:0/1840281847 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3c140050d0 con 0x561824df0660 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 10 client.16206 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 20 client.16206 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e0e720 0x561824e10b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 4 client.16206 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561824d62920 con 0x561824e0e720 2024-04-21T08:47:38.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.615+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e0e720 0x561824e10b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.616+0000 7f3c017fa640 10 client.16206 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.616+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e0e720 0x561824e10b10 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4380c342675df43 server_cookie=e6406a53050af34f in_seq=0 out_seq=0 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.616+0000 7f3c017fa640 1 -- 172.21.15.57:0/1840281847 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561824d62920 con 0x561824e0e720 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.616+0000 7f3c017fa640 10 client.16206 handle_command_reply: tid=0 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.616+0000 7f3c22e11740 1 client.16206 shutdown 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 msgr2=0x561824dfb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfb1f0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3c1401c040 tx=0x7f3c14009d00 comp rx=0 tx=0).stop 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 shutdown_connections 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e0e720 0x561824e10b10 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824dfb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c017fa640 0 client.16206 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1840281847 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfb1f0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 >> 172.21.15.57:0/1840281847 conn(0x561824deafb0 msgr2=0x561824decd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 shutdown_connections 2024-04-21T08:47:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 1 -- 172.21.15.57:0/1840281847 wait complete. 2024-04-21T08:47:38.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.617+0000 7f3c22e11740 20 client.16206 trim_cache size 0 max 16384 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.634+0000 7f3c22e11740 1 Processor -- start 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.634+0000 7f3c22e11740 1 -- start start 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824e0a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c22e11740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181aaff0 con 0x561824df0660 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c22e11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181ad850 con 0x561824df1010 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c22e11740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181ab640 con 0x561824dfe220 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43240/0 (socket says 172.21.15.57:43240) 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1332400427 learned_addr learned my addr 172.21.15.57:0/1332400427 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824e0a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1332400427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 msgr2=0x561824dfb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1332400427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 msgr2=0x561824e0a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824e0a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/1332400427 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561824def970 con 0x561824df0660 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfd8b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c14000f80 tx=0x7f3c140119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d699c5ccbb8ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c037fe640 1 -- 172.21.15.57:0/1332400427 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14028070 con 0x561824df0660 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.635+0000 7f3c037fe640 1 -- 172.21.15.57:0/1332400427 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c14004e50 con 0x561824df0660 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c037fe640 1 -- 172.21.15.57:0/1332400427 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1401e070 con 0x561824df0660 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 -- 172.21.15.57:0/1332400427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 msgr2=0x561824dfd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfd8b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3c14000f80 tx=0x7f3c140119b0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 -- 172.21.15.57:0/1332400427 shutdown_connections 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824dfe220 0x561824dfb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824df1010 0x561824e0a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 --2- 172.21.15.57:0/1332400427 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824dfd8b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 -- 172.21.15.57:0/1332400427 >> 172.21.15.57:0/1332400427 conn(0x561824deafb0 msgr2=0x561824dfb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 -- 172.21.15.57:0/1332400427 shutdown_connections 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.636+0000 7f3c22e11740 1 -- 172.21.15.57:0/1332400427 wait complete. 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 -- start start 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 Processor -- start 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824e02010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824df1010 0x561824e02550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824dfe220 0x561824e02a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181abd10 con 0x561824df0660 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181ad180 con 0x561824dfe220 2024-04-21T08:47:38.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3c181ac3e0 con 0x561824df1010 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c22e11740 10 client.0 _ll_register_callbacks cb 0x7f3c1401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824e02010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824e02010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43244/0 (socket says 172.21.15.57:43244) 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3060001443 learned_addr learned my addr 172.21.15.57:0/3060001443 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1e7c2640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824dfe220 0x561824e02a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824df1010 0x561824e02550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3060001443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824df1010 msgr2=0x561824e02550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824df1010 0x561824e02550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824dfe220 msgr2=0x561824e02a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.637+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824dfe220 0x561824e02a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c1dfc1640 1 -- 172.21.15.57:0/3060001443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561824e73bb0 con 0x561824df0660 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c1dfc1640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824e02010 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3c1401c040 tx=0x7f3c14009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8babcdf34a465e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c017fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c017fa640 1 -- 172.21.15.57:0/3060001443 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14028030 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c017fa640 1 -- 172.21.15.57:0/3060001443 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f3c14005000 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c14004e50 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c017fa640 1 -- 172.21.15.57:0/3060001443 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3c1401e070 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c22e11740 10 client.16212 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561824e74ef0 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c017fa640 1 -- 172.21.15.57:0/3060001443 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c14004e50 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.638+0000 7f3c017fa640 1 -- 172.21.15.57:0/3060001443 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3c14019c30 con 0x561824df0660 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c22e11740 10 client.16212 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c22e11740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c22e11740 20 client.16212 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e753b0 0x561824e777a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c22e11740 4 client.16212 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:47:38.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561824d5c790 con 0x561824e753b0 2024-04-21T08:47:38.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e753b0 0x561824e777a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c1d7c0640 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e753b0 0x561824e777a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee80247f03bb507d server_cookie=d74bd1c68846a435 in_seq=0 out_seq=0 2024-04-21T08:47:38.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.639+0000 7f3c017fa640 10 client.16212 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.640+0000 7f3c017fa640 1 -- 172.21.15.57:0/3060001443 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561824d5c790 con 0x561824e753b0 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.640+0000 7f3c017fa640 10 client.16212 handle_command_reply: tid=0 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.640+0000 7f3c22e11740 1 client.16212 shutdown 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 msgr2=0x561824e02010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824e02010 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3c1401c040 tx=0x7f3c14009c90 comp rx=0 tx=0).stop 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 shutdown_connections 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561824e753b0 0x561824e777a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561824dfe220 0x561824e02a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561824df1010 0x561824e02550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c017fa640 0 client.16212 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 --2- 172.21.15.57:0/3060001443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561824df0660 0x561824e02010 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 >> 172.21.15.57:0/3060001443 conn(0x561824deafb0 msgr2=0x561824dfba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 shutdown_connections 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 1 -- 172.21.15.57:0/3060001443 wait complete. 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c22e11740 20 client.16212 trim_cache size 0 max 16384 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:47:38.646 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "b926b3c8-226e-447f-98b1-7b78a97b9920", 2024-04-21T08:47:38.647 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:47:38.647 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:47:38.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3c040a3da0 msgr2=0x7f3c040a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.641+0000 7f3c20225640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3c040a3da0 0x7f3c040a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3c10005fd0 tx=0x7f3c100046c0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 msgr2=0x7f3c1819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 0x7f3c1819eac0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3c08000c00 tx=0x7f3c080045e0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 shutdown_connections 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3c040a3da0 0x7f3c040a6260 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c18198690 0x7f3c18198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c181072e0 0x7f3c1819eac0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 --2- 172.21.15.57:0/870728431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c18102210 0x7f3c1819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 >> 172.21.15.57:0/870728431 conn(0x7f3c180fdab0 msgr2=0x7f3c180fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 shutdown_connections 2024-04-21T08:47:38.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.642+0000 7f3c20225640 1 -- 172.21.15.57:0/870728431 wait complete. 2024-04-21T08:47:38.662 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:47:38.662 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:47:38.697 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:38 smithi057 ceph-mon[27968]: pgmap v69: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 3.4 MiB/s wr, 1.30k op/s 2024-04-21T08:47:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:38 smithi129 ceph-mon[31272]: pgmap v69: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 3.4 MiB/s wr, 1.30k op/s 2024-04-21T08:47:38.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.849+0000 7ff5e46fd640 1 -- 172.21.15.57:0/2775440546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 msgr2=0x7ff5dc106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.849+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/2775440546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc106760 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009900 tx=0x7ff5d802bd90 comp rx=0 tx=0).stop 2024-04-21T08:47:38.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.850+0000 7ff5e46fd640 1 -- 172.21.15.57:0/2775440546 shutdown_connections 2024-04-21T08:47:38.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.850+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/2775440546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc106760 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.850+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/2775440546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5dc1018d0 0x7ff5dc105d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.850+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/2775440546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc100fb0 0x7ff5dc101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.850+0000 7ff5e46fd640 1 -- 172.21.15.57:0/2775440546 >> 172.21.15.57:0/2775440546 conn(0x7ff5dc078000 msgr2=0x7ff5dc078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.855+0000 7ff5e46fd640 1 -- 172.21.15.57:0/2775440546 shutdown_connections 2024-04-21T08:47:38.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.855+0000 7ff5e46fd640 1 -- 172.21.15.57:0/2775440546 wait complete. 2024-04-21T08:47:38.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 Processor -- start 2024-04-21T08:47:38.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 -- start start 2024-04-21T08:47:38.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc100fb0 0x7ff5dc0737a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5dc1018d0 0x7ff5dc06c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc06ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.857 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc1188b0 con 0x7ff5dc1062d0 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc118730 con 0x7ff5dc100fb0 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.856+0000 7ff5e46fd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc118a30 con 0x7ff5dc1018d0 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc06ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc06ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43266/0 (socket says 172.21.15.57:43266) 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/3183168405 learned_addr learned my addr 172.21.15.57:0/3183168405 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/3183168405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5dc1018d0 msgr2=0x7ff5dc06c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5dc1018d0 0x7ff5dc06c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/3183168405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc100fb0 msgr2=0x7ff5dc0737a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc100fb0 0x7ff5dc0737a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/3183168405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc06d440 con 0x7ff5dc1062d0 2024-04-21T08:47:38.858 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.857+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc06ce00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff5d802c320 tx=0x7ff5d802db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.858+0000 7ff5d37fe640 1 -- 172.21.15.57:0/3183168405 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8043020 con 0x7ff5dc1062d0 2024-04-21T08:47:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.858+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5dc06d6d0 con 0x7ff5dc1062d0 2024-04-21T08:47:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.859+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5dc071610 con 0x7ff5dc1062d0 2024-04-21T08:47:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.859+0000 7ff5d37fe640 1 -- 172.21.15.57:0/3183168405 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d80052b0 con 0x7ff5dc1062d0 2024-04-21T08:47:38.859 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.859+0000 7ff5d37fe640 1 -- 172.21.15.57:0/3183168405 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8003b50 con 0x7ff5dc1062d0 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.869+0000 7ff5e73c8740 1 Processor -- start 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.869+0000 7ff5e73c8740 1 -- start start 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026580 0x556a8f026960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f02fa80 0x556a8f031e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556a8ef8b1e0 con 0x556a8f026ea0 2024-04-21T08:47:38.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556a8ef28390 con 0x556a8f02fa80 2024-04-21T08:47:38.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556a8ef871a0 con 0x556a8f026580 2024-04-21T08:47:38.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e2499640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e2499640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43280/0 (socket says 172.21.15.57:43280) 2024-04-21T08:47:38.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.870+0000 7ff5e2499640 1 -- 172.21.15.57:0/403972720 learned_addr learned my addr 172.21.15.57:0/403972720 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5d37fe640 1 -- 172.21.15.57:0/3183168405 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff5d8046020 con 0x7ff5dc1062d0 2024-04-21T08:47:38.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5e2499640 1 -- 172.21.15.57:0/403972720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026580 msgr2=0x556a8f026960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:47:38.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5e2499640 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026580 0x556a8f026960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5e2499640 1 -- 172.21.15.57:0/403972720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f02fa80 msgr2=0x556a8f031e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:47:38.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5e2499640 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f02fa80 0x556a8f031e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5e2499640 1 -- 172.21.15.57:0/403972720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a8ed677d0 con 0x556a8f026ea0 2024-04-21T08:47:38.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.871+0000 7ff5e2499640 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02f540 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009da0 tx=0x7ff5d400b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f29ca5204b1260 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.873+0000 7ff5d37fe640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5c80a3f20 0x7ff5c80a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.874+0000 7ff5d0ff9640 1 -- 172.21.15.57:0/403972720 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400bd40 con 0x556a8f026ea0 2024-04-21T08:47:38.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.874+0000 7ff5d37fe640 1 -- 172.21.15.57:0/3183168405 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff5d80f2c90 con 0x7ff5dc1062d0 2024-04-21T08:47:38.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.874+0000 7ff5d0ff9640 1 -- 172.21.15.57:0/403972720 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d4017070 con 0x556a8f026ea0 2024-04-21T08:47:38.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.875+0000 7ff5e73c8740 1 -- 172.21.15.57:0/403972720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 msgr2=0x556a8f02f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.875+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02f540 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009da0 tx=0x7ff5d400b420 comp rx=0 tx=0).stop 2024-04-21T08:47:38.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.876+0000 7ff5e1c98640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5c80a3f20 0x7ff5c80a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.878+0000 7ff5e73c8740 1 -- 172.21.15.57:0/403972720 shutdown_connections 2024-04-21T08:47:38.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.878+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f02fa80 0x556a8f031e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.878+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02f540 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.878+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/403972720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026580 0x556a8f026960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.878+0000 7ff5e73c8740 1 -- 172.21.15.57:0/403972720 >> 172.21.15.57:0/403972720 conn(0x556a8f020cb0 msgr2=0x556a8f021090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.880+0000 7ff5e1c98640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5c80a3f20 0x7ff5c80a63e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x556a8f021e40 tx=0x7ff5cc006bd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.879 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.880+0000 7ff5e73c8740 1 -- 172.21.15.57:0/403972720 shutdown_connections 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.881+0000 7ff5e73c8740 1 -- 172.21.15.57:0/403972720 wait complete. 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.881+0000 7ff5e73c8740 1 -- start start 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.881+0000 7ff5e73c8740 1 Processor -- start 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f02fa80 0x556a8f02d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556a8f0328c0 con 0x556a8f026ea0 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556a8ef442f0 con 0x556a8f026580 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556a8f032a40 con 0x556a8f02fa80 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.882+0000 7ff5e73c8740 10 client.0 _ll_register_callbacks cb 0x556a8ef51580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2c9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f02fa80 0x556a8f02d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2c9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f02fa80 0x556a8f02d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:37908/0 (socket says 172.21.15.57:37908) 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/3120628852 learned_addr learned my addr 172.21.15.57:0/3120628852 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2499640 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2499640 1 -- 172.21.15.57:0/3120628852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f02fa80 msgr2=0x556a8f02d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2499640 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f02fa80 0x556a8f02d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2499640 1 -- 172.21.15.57:0/3120628852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 msgr2=0x556a8f02cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:47:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2499640 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.883+0000 7ff5e2499640 1 -- 172.21.15.57:0/3120628852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a8ec9d3f0 con 0x556a8f026580 2024-04-21T08:47:38.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:38 smithi136 ceph-mon[31791]: pgmap v69: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 3.4 MiB/s wr, 1.30k op/s 2024-04-21T08:47:38.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.885+0000 7ff5e2499640 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02c960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4012f10 tx=0x7ff5d400bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef918098b51969 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.888+0000 7ff5c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:38.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.888+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/3120628852 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4022020 con 0x556a8f026580 2024-04-21T08:47:38.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.888+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/3120628852 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff5d4011d80 con 0x556a8f026580 2024-04-21T08:47:38.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.888+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d4017070 con 0x556a8f026580 2024-04-21T08:47:38.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.892+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/3120628852 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff5d4012430 con 0x556a8f026580 2024-04-21T08:47:38.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.895+0000 7ff5e73c8740 10 client.25724 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:38.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.895+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556a8ec5d0f0 con 0x556a8f026580 2024-04-21T08:47:38.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.899+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/3120628852 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d4017070 con 0x556a8f026580 2024-04-21T08:47:38.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.899+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/3120628852 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff5d4011070 con 0x556a8f026580 2024-04-21T08:47:38.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.903+0000 7ff5e73c8740 10 client.25724 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:38.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.903+0000 7ff5e73c8740 10 client.25724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:38.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.903+0000 7ff5e73c8740 20 client.25724 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:38.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.903+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f03fa20 0x556a8f041e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.903+0000 7ff5e73c8740 4 client.25724 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:38.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.903+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a8ef98c00 con 0x556a8f03fa20 2024-04-21T08:47:38.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.906+0000 7ff5e1c98640 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f03fa20 0x556a8f041e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.914+0000 7ff5e1c98640 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f03fa20 0x556a8f041e10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ab51cea6626b855 server_cookie=5b5fddf7784d345e in_seq=0 out_seq=0 2024-04-21T08:47:38.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.914+0000 7ff5c5ffb640 10 client.25724 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.918+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/3120628852 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a8ef98c00 con 0x556a8f03fa20 2024-04-21T08:47:38.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.918+0000 7ff5c5ffb640 10 client.25724 handle_command_reply: tid=0 2024-04-21T08:47:38.922 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.923+0000 7ff5e73c8740 1 client.25724 shutdown 2024-04-21T08:47:38.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.925+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 msgr2=0x556a8f02c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.925+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02c960 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4012f10 tx=0x7ff5d400bf70 comp rx=0 tx=0).stop 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 shutdown_connections 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f03fa20 0x556a8f041e10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f02fa80 0x556a8f02d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f026ea0 0x556a8f02cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/3120628852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02c960 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 >> 172.21.15.57:0/3120628852 conn(0x556a8f020cb0 msgr2=0x556a8f022a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.930+0000 7ff5c5ffb640 0 client.25724 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:38.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.935+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 shutdown_connections 2024-04-21T08:47:38.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.939+0000 7ff5e73c8740 1 -- 172.21.15.57:0/3120628852 wait complete. 2024-04-21T08:47:38.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.942+0000 7ff5e73c8740 20 client.25724 trim_cache size 0 max 16384 2024-04-21T08:47:38.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.981+0000 7ff5e73c8740 1 Processor -- start 2024-04-21T08:47:38.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.981+0000 7ff5e73c8740 1 -- start start 2024-04-21T08:47:38.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.981+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.981+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 0x556a8f03b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e2499640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e2499640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40348/0 (socket says 172.21.15.57:40348) 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f01fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e2499640 1 -- 172.21.15.57:0/213272614 learned_addr learned my addr 172.21.15.57:0/213272614 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c000df50 con 0x556a8f02fa80 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c00107b0 con 0x556a8f026580 2024-04-21T08:47:38.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.982+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c000e5a0 con 0x556a8f026ea0 2024-04-21T08:47:38.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.983+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f01fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.983+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/213272614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 msgr2=0x556a8f03b860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.983+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 0x556a8f03b860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.983+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/213272614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 msgr2=0x556a8f02f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.983+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.983+0000 7ff5e2c9a640 1 -- 172.21.15.57:0/213272614 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a8f020ae0 con 0x556a8f02fa80 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.984+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f01fd70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80c70c0 tx=0x7ff5d8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64db07543c11c9fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.984+0000 7ff5c7fff640 1 -- 172.21.15.57:0/213272614 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80bf080 con 0x556a8f02fa80 2024-04-21T08:47:38.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.984+0000 7ff5c7fff640 1 -- 172.21.15.57:0/213272614 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d80be980 con 0x556a8f02fa80 2024-04-21T08:47:38.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.984+0000 7ff5c7fff640 1 -- 172.21.15.57:0/213272614 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80f02a0 con 0x556a8f02fa80 2024-04-21T08:47:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.986+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 msgr2=0x556a8f01fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.986+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f01fd70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80c70c0 tx=0x7ff5d8009260 comp rx=0 tx=0).stop 2024-04-21T08:47:38.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.986+0000 7ff5e2499640 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.990+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 shutdown_connections 2024-04-21T08:47:38.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.990+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f01fd70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.990+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 0x556a8f03b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.990+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/213272614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f02f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.990+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 >> 172.21.15.57:0/213272614 conn(0x556a8f020cb0 msgr2=0x556a8f02b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:38.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.994+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 shutdown_connections 2024-04-21T08:47:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.997+0000 7ff5e73c8740 1 -- 172.21.15.57:0/213272614 wait complete. 2024-04-21T08:47:38.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.997+0000 7ff5e73c8740 1 -- start start 2024-04-21T08:47:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.997+0000 7ff5e73c8740 1 Processor -- start 2024-04-21T08:47:38.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f038040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 0x556a8f0333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f0338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c000ec70 con 0x556a8f02fa80 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c00100e0 con 0x556a8f026580 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c000f340 con 0x556a8f026ea0 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e73c8740 10 client.0 _ll_register_callbacks cb 0x7ff5d4009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e2499640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f038040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e2499640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f038040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40354/0 (socket says 172.21.15.57:40354) 2024-04-21T08:47:38.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.998+0000 7ff5e2499640 1 -- 172.21.15.57:0/1370894392 learned_addr learned my addr 172.21.15.57:0/1370894392 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.999+0000 7ff5e2c9a640 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f0338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.999+0000 7ff5e2499640 1 -- 172.21.15.57:0/1370894392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 msgr2=0x556a8f0333b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:47:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.999+0000 7ff5e2499640 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 0x556a8f0333b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.999+0000 7ff5e2499640 1 -- 172.21.15.57:0/1370894392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 msgr2=0x556a8f0338f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.999+0000 7ff5e2499640 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f0338f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:38.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:38.999+0000 7ff5e2499640 1 -- 172.21.15.57:0/1370894392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a8f033fe0 con 0x556a8f026580 2024-04-21T08:47:39.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.005+0000 7ff5e2499640 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f038040 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400b460 tx=0x7ff5d4009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f39c4daa66f07ece server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.005+0000 7ff5c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.005+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/1370894392 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d401f070 con 0x556a8f026580 2024-04-21T08:47:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.005+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d80be980 con 0x556a8f026580 2024-04-21T08:47:39.005 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.005+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/1370894392 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7ff5d4004190 con 0x556a8f026580 2024-04-21T08:47:39.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.007+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/1370894392 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff5d4022040 con 0x556a8f026580 2024-04-21T08:47:39.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.009+0000 7ff5e73c8740 10 client.25727 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:39.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.009+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556a8f0a5b60 con 0x556a8f026580 2024-04-21T08:47:39.009 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/1370894392 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5d80be980 con 0x556a8f026580 2024-04-21T08:47:39.009 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/1370894392 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff5d4011040 con 0x556a8f026580 2024-04-21T08:47:39.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5e73c8740 10 client.25727 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:39.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5e73c8740 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:39.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5e73c8740 20 client.25727 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:39.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f0a62b0 0x556a8f0a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5e73c8740 4 client.25727 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:47:39.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.010+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556a8ef92a70 con 0x556a8f0a62b0 2024-04-21T08:47:39.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.015+0000 7ff5e1c98640 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f0a62b0 0x556a8f0a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.022 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.022+0000 7ff5e1c98640 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f0a62b0 0x556a8f0a86d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=731e8fdfea76f514 server_cookie=dfcc4f653db359e4 in_seq=0 out_seq=0 2024-04-21T08:47:39.022 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.023+0000 7ff5c5ffb640 10 client.25727 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:39.026 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.026+0000 7ff5c5ffb640 1 -- 172.21.15.57:0/1370894392 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x556a8ef92a70 con 0x556a8f0a62b0 2024-04-21T08:47:39.026 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.026+0000 7ff5c5ffb640 10 client.25727 handle_command_reply: tid=0 2024-04-21T08:47:39.026 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.026+0000 7ff5e73c8740 1 client.25727 shutdown 2024-04-21T08:47:39.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.028+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 msgr2=0x556a8f038040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.028+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f038040 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400b460 tx=0x7ff5d4009b20 comp rx=0 tx=0).stop 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.030+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 shutdown_connections 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.030+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556a8f0a62b0 0x556a8f0a86d0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.030+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556a8f02fa80 0x556a8f0338f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.030+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556a8f026ea0 0x556a8f0333b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.030+0000 7ff5e73c8740 1 --2- 172.21.15.57:0/1370894392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556a8f026580 0x556a8f038040 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.030+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 >> 172.21.15.57:0/1370894392 conn(0x556a8f020cb0 msgr2=0x556a8f01fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.032+0000 7ff5c5ffb640 0 client.25727 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:39.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.033+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 shutdown_connections 2024-04-21T08:47:39.035 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.035+0000 7ff5e73c8740 1 -- 172.21.15.57:0/1370894392 wait complete. 2024-04-21T08:47:39.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.036+0000 7ff5e73c8740 20 client.25727 trim_cache size 0 max 16384 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (15 inodes in the stack)", 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: "b926b3c8-226e-447f-98b1-7b78a97b9920": { 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "b926b3c8-226e-447f-98b1-7b78a97b9920", 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:47:39.038 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:47:39.039 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:47:39.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.041+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5c80a3f20 msgr2=0x7ff5c80a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.041+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5c80a3f20 0x7ff5c80a63e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x556a8f021e40 tx=0x7ff5cc006bd0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.041+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 msgr2=0x7ff5dc06ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.041+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc06ce00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7ff5d802c320 tx=0x7ff5d802db80 comp rx=0 tx=0).stop 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 shutdown_connections 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5c80a3f20 0x7ff5c80a63e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5dc1062d0 0x7ff5dc06ce00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5dc1018d0 0x7ff5dc06c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 --2- 172.21.15.57:0/3183168405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5dc100fb0 0x7ff5dc0737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 >> 172.21.15.57:0/3183168405 conn(0x7ff5dc078000 msgr2=0x7ff5dc103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 shutdown_connections 2024-04-21T08:47:39.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.044+0000 7ff5e46fd640 1 -- 172.21.15.57:0/3183168405 wait complete. 2024-04-21T08:47:39.063 INFO:tasks.cephfs.filesystem:scrub status for tag:b926b3c8-226e-447f-98b1-7b78a97b9920 - {'path': '/', 'tag': 'b926b3c8-226e-447f-98b1-7b78a97b9920', 'options': 'recursive,force'} 2024-04-21T08:47:39.298 INFO:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.0 2024-04-21T08:47:39.298 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:47:39.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 -- 172.21.15.57:0/4262673694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200100e30 msgr2=0x7fa200101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 --2- 172.21.15.57:0/4262673694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200100e30 0x7fa200101290 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa1f801acb0 tx=0x7fa1f803d130 comp rx=0 tx=0).stop 2024-04-21T08:47:39.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 -- 172.21.15.57:0/4262673694 shutdown_connections 2024-04-21T08:47:39.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 --2- 172.21.15.57:0/4262673694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200100e30 0x7fa200101290 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 --2- 172.21.15.57:0/4262673694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa200106570 0x7fa2001008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 --2- 172.21.15.57:0/4262673694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.464+0000 7fa206d40640 1 -- 172.21.15.57:0/4262673694 >> 172.21.15.57:0/4262673694 conn(0x7fa200076170 msgr2=0x7fa200076580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.465+0000 7fa206d40640 1 -- 172.21.15.57:0/4262673694 shutdown_connections 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.465+0000 7fa206d40640 1 -- 172.21.15.57:0/4262673694 wait complete. 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 Processor -- start 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 -- start start 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa200100e30 0x7fa200072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200106570 0x7fa2001a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010a050 con 0x7fa200100e30 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa200109ed0 con 0x7fa200105c50 2024-04-21T08:47:39.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa206d40640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010a1d0 con 0x7fa200106570 2024-04-21T08:47:39.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200072f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40366/0 (socket says 172.21.15.57:40366) 2024-04-21T08:47:39.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa1fffff640 1 -- 172.21.15.57:0/3651466333 learned_addr learned my addr 172.21.15.57:0/3651466333 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:39.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.466+0000 7fa2052dd640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200106570 0x7fa2001a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa204adc640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa200100e30 0x7fa200072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa2052dd640 1 -- 172.21.15.57:0/3651466333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 msgr2=0x7fa200072f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa2052dd640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200072f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa2052dd640 1 -- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa200100e30 msgr2=0x7fa200072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa2052dd640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa200100e30 0x7fa200072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa2052dd640 1 -- 172.21.15.57:0/3651466333 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2001a3dd0 con 0x7fa200106570 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa1fffff640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa2052dd640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200106570 0x7fa2001a36a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1f801ac00 tx=0x7fa1f80438e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f804a040 con 0x7fa200106570 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2001a4060 con 0x7fa200106570 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1f8055070 con 0x7fa200106570 2024-04-21T08:47:39.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.467+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8045440 con 0x7fa200106570 2024-04-21T08:47:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.468+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2001ab930 con 0x7fa200106570 2024-04-21T08:47:39.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.469+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa200069230 con 0x7fa200106570 2024-04-21T08:47:39.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.470+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa1f80455e0 con 0x7fa200106570 2024-04-21T08:47:39.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.471+0000 7fa1fdffb640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1f00a3e20 0x7fa1f00a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.471+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa1f8103b20 con 0x7fa200106570 2024-04-21T08:47:39.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.471+0000 7fa204adc640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1f00a3e20 0x7fa1f00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.472+0000 7fa204adc640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1f00a3e20 0x7fa1f00a62e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00bbe0 tx=0x7fa1ec00b390 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.476+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1f80cff10 con 0x7fa200106570 2024-04-21T08:47:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:39 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:47:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:39 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:39 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:47:39.710 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.711+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2001abb00 con 0x7fa200106570 2024-04-21T08:47:39.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.711+0000 7fa1fdffb640 1 -- 172.21.15.57:0/3651466333 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fa1f80cf810 con 0x7fa200106570 2024-04-21T08:47:39.711 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:47:39.711 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:47:39.712 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1f00a3e20 msgr2=0x7fa1f00a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1f00a3e20 0x7fa1f00a62e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec00bbe0 tx=0x7fa1ec00b390 comp rx=0 tx=0).stop 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200106570 msgr2=0x7fa2001a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200106570 0x7fa2001a36a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1f801ac00 tx=0x7fa1f80438e0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 shutdown_connections 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1f00a3e20 0x7fa1f00a62e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa200106570 0x7fa2001a36a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa200105c50 0x7fa200072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 --2- 172.21.15.57:0/3651466333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa200100e30 0x7fa200072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 >> 172.21.15.57:0/3651466333 conn(0x7fa200076170 msgr2=0x7fa200102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 shutdown_connections 2024-04-21T08:47:39.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.714+0000 7fa206d40640 1 -- 172.21.15.57:0/3651466333 wait complete. 2024-04-21T08:47:39.724 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db / --timeout 30 --expiration 74.0 --await' 2024-04-21T08:47:39.725 DEBUG:teuthology.orchestra.run.smithi057:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db / --timeout 30 --expiration 74.0 --await 2024-04-21T08:47:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:39 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:47:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:39 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:39 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:47:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 -- 172.21.15.57:0/3831366908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102b40 msgr2=0x7f7b64106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 --2- 172.21.15.57:0/3831366908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102b40 0x7f7b64106b80 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009900 tx=0x7f7b5002bc30 comp rx=0 tx=0).stop 2024-04-21T08:47:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 -- 172.21.15.57:0/3831366908 shutdown_connections 2024-04-21T08:47:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 --2- 172.21.15.57:0/3831366908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b64107260 0x7f7b6410ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 --2- 172.21.15.57:0/3831366908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102b40 0x7f7b64106b80 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 --2- 172.21.15.57:0/3831366908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b64102190 0x7f7b64102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 -- 172.21.15.57:0/3831366908 >> 172.21.15.57:0/3831366908 conn(0x7f7b64079310 msgr2=0x7f7b64079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 -- 172.21.15.57:0/3831366908 shutdown_connections 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.837+0000 7f7b6b043640 1 -- 172.21.15.57:0/3831366908 wait complete. 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 Processor -- start 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 -- start start 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 0x7f7b6419e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b64107260 0x7f7b6419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b641985e0 0x7f7b64198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64112180 con 0x7f7b64102190 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64112000 con 0x7f7b64107260 2024-04-21T08:47:39.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b6b043640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64112300 con 0x7f7b641985e0 2024-04-21T08:47:39.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b68ddf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 0x7f7b6419e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b68ddf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 0x7f7b6419e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43344/0 (socket says 172.21.15.57:43344) 2024-04-21T08:47:39.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b68ddf640 1 -- 172.21.15.57:0/881917868 learned_addr learned my addr 172.21.15.57:0/881917868 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:39.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.838+0000 7f7b63fff640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b64107260 0x7f7b6419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b695e0640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b641985e0 0x7f7b64198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b68ddf640 1 -- 172.21.15.57:0/881917868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b641985e0 msgr2=0x7f7b64198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b641985e0 0x7f7b64198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b68ddf640 1 -- 172.21.15.57:0/881917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b64107260 msgr2=0x7f7b6419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b64107260 0x7f7b6419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b68ddf640 1 -- 172.21.15.57:0/881917868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b64199100 con 0x7f7b64102190 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 0x7f7b6419e3e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800e9f0 tx=0x7f7b5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b61ffb640 1 -- 172.21.15.57:0/881917868 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800cd80 con 0x7f7b64102190 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b61ffb640 1 -- 172.21.15.57:0/881917868 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b580108e0 con 0x7f7b64102190 2024-04-21T08:47:39.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b641993f0 con 0x7f7b64102190 2024-04-21T08:47:39.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b61ffb640 1 -- 172.21.15.57:0/881917868 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58018980 con 0x7f7b64102190 2024-04-21T08:47:39.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.839+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b641a55f0 con 0x7f7b64102190 2024-04-21T08:47:39.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b61ffb640 1 -- 172.21.15.57:0/881917868 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7b58010430 con 0x7f7b64102190 2024-04-21T08:47:39.849 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 Processor -- start 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 -- start start 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 0x56009e733da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7342e0 0x56009e73c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e73f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56009e6940b0 con 0x56009e7339c0 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56009e645680 con 0x56009e7342e0 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56009e651c60 con 0x56009e73cec0 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b695e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 0x56009e733da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b61ffb640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7b540a3d20 0x7f7b540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b695e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 0x56009e733da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43348/0 (socket says 172.21.15.57:43348) 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b695e0640 1 -- 172.21.15.57:0/4158578505 learned_addr learned my addr 172.21.15.57:0/4158578505 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.842+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7342e0 0x56009e73c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b63fff640 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e73f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.850 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7b540a3d20 0x7f7b540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b61ffb640 1 -- 172.21.15.57:0/881917868 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f7b580d0ef0 con 0x7f7b64102190 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 -- 172.21.15.57:0/4158578505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 msgr2=0x56009e73f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e73f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 -- 172.21.15.57:0/4158578505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7342e0 msgr2=0x56009e73c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7342e0 0x56009e73c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 -- 172.21.15.57:0/4158578505 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56009e476c90 con 0x56009e7339c0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b695e0640 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 0x56009e733da0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0139f0 tx=0x7f7b5c013eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92bb742256d6ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b4effd640 1 -- 172.21.15.57:0/4158578505 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0134c0 con 0x56009e7339c0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b4effd640 1 -- 172.21.15.57:0/4158578505 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b5c022070 con 0x56009e7339c0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.843+0000 7f7b4effd640 1 -- 172.21.15.57:0/4158578505 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c01cc20 con 0x56009e7339c0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4158578505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 msgr2=0x56009e733da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b695e0640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7b540a3d20 0x7f7b540a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c021000 tx=0x7f7b5c01f000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 0x56009e733da0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c0139f0 tx=0x7f7b5c013eb0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4158578505 shutdown_connections 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e73f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7342e0 0x56009e73c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4158578505 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7339c0 0x56009e733da0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4158578505 >> 172.21.15.57:0/4158578505 conn(0x56009e72e0f0 msgr2=0x56009e72e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4158578505 shutdown_connections 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4158578505 wait complete. 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 -- start start 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.844+0000 7f7b6dc3c740 1 Processor -- start 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e746e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e7473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e741270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56009e697400 con 0x56009e7342e0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56009e653840 con 0x56009e7339c0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56009e69c720 con 0x56009e73cec0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b6dc3c740 10 client.0 _ll_register_callbacks cb 0x56009e6a4ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e7473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e7473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43354/0 (socket says 172.21.15.57:43354) 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e746e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b63fff640 1 -- 172.21.15.57:0/3887029445 learned_addr learned my addr 172.21.15.57:0/3887029445 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b695e0640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e741270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b68ddf640 1 -- 172.21.15.57:0/3887029445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 msgr2=0x56009e741270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e741270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b68ddf640 1 -- 172.21.15.57:0/3887029445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 msgr2=0x56009e7473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e7473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b68ddf640 1 -- 172.21.15.57:0/3887029445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56009e4d7420 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.845+0000 7f7b695e0640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e741270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e746e60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b580ce2e0 tx=0x7f7b5806b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1aa61492b0bd13e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b43fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b43fff640 1 -- 172.21.15.57:0/3887029445 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580a0ee0 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b43fff640 1 -- 172.21.15.57:0/3887029445 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f7b580991a0 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5c022070 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b43fff640 1 -- 172.21.15.57:0/3887029445 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b580a57a0 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 10 client.25745 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56009e36a580 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b43fff640 1 -- 172.21.15.57:0/3887029445 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b5c022070 con 0x56009e7339c0 2024-04-21T08:47:39.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b43fff640 1 -- 172.21.15.57:0/3887029445 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7b58018b20 con 0x56009e7339c0 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 10 client.25745 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 10 client.25745 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 20 client.25745 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7510a0 0x56009e753510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 4 client.25745 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.846+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56009e6a5b10 con 0x56009e7510a0 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.847+0000 7f7b63fff640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7510a0 0x56009e753510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.847+0000 7f7b63fff640 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7510a0 0x56009e753510 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f30efce8edeadac server_cookie=97c6bf2293444eb3 in_seq=0 out_seq=0 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.847+0000 7f7b43fff640 10 client.25745 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b43fff640 1 -- 172.21.15.57:0/3887029445 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56009e6a5b10 con 0x56009e7510a0 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b43fff640 10 client.25745 handle_command_reply: tid=0 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 client.25745 shutdown 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 msgr2=0x56009e746e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e746e60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7b580ce2e0 tx=0x7f7b5806b5a0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 shutdown_connections 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7510a0 0x56009e753510 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e741270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e7473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3887029445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e746e60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 >> 172.21.15.57:0/3887029445 conn(0x56009e72e0f0 msgr2=0x56009e72fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b43fff640 0 client.25745 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 shutdown_connections 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3887029445 wait complete. 2024-04-21T08:47:39.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.849+0000 7f7b6dc3c740 20 client.25745 trim_cache size 0 max 16384 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 Processor -- start 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 -- start start 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e7470c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e7342e0 0x56009e758de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 0x56009e759320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641ad900 con 0x56009e73cec0 2024-04-21T08:47:39.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641ab6f0 con 0x56009e7339c0 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641ad230 con 0x56009e7342e0 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.867+0000 7f7b695e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 0x56009e759320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b68ddf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e7470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 0x56009e759320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43360/0 (socket says 172.21.15.57:43360) 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 -- 172.21.15.57:0/4088787865 learned_addr learned my addr 172.21.15.57:0/4088787865 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 -- 172.21.15.57:0/4088787865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e7342e0 msgr2=0x56009e758de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b63fff640 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e7342e0 0x56009e758de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e7342e0 0x56009e758de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 -- 172.21.15.57:0/4088787865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 msgr2=0x56009e7470c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e7470c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 -- 172.21.15.57:0/4088787865 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56009e741240 con 0x56009e73cec0 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e7470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:39.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b63fff640 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e7342e0 0x56009e758de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b695e0640 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 0x56009e759320 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c011750 tx=0x7f7b5c028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7174442eca817f35 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b4e7fc640 1 -- 172.21.15.57:0/4088787865 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c003970 con 0x56009e73cec0 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b4e7fc640 1 -- 172.21.15.57:0/4088787865 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b5c035070 con 0x56009e73cec0 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.868+0000 7f7b4e7fc640 1 -- 172.21.15.57:0/4088787865 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c01c350 con 0x56009e73cec0 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4088787865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 msgr2=0x56009e759320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 0x56009e759320 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c011750 tx=0x7f7b5c028000 comp rx=0 tx=0).stop 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4088787865 shutdown_connections 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e73cec0 0x56009e759320 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e7342e0 0x56009e758de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/4088787865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e7470c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4088787865 >> 172.21.15.57:0/4088787865 conn(0x56009e72e0f0 msgr2=0x56009e7469b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4088787865 shutdown_connections 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/4088787865 wait complete. 2024-04-21T08:47:39.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 -- start start 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.869+0000 7f7b6dc3c740 1 Processor -- start 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e74b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e74b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e7b6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641ac490 con 0x56009e7342e0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641abdc0 con 0x56009e7339c0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641acb60 con 0x56009e73cec0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 10 client.0 _ll_register_callbacks cb 0x7f7b5809c700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e74b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b68ddf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e74b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e74b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43374/0 (socket says 172.21.15.57:43374) 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 -- 172.21.15.57:0/3980685342 learned_addr learned my addr 172.21.15.57:0/3980685342 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b695e0640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e7b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 -- 172.21.15.57:0/3980685342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 msgr2=0x56009e7b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e7b6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 -- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 msgr2=0x56009e74b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e74b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 -- 172.21.15.57:0/3980685342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56009e73f620 con 0x56009e7342e0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e74b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b695e0640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e7b6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b63fff640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e74b950 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7b500041d0 tx=0x7f7b50004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955a9473062826fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b43fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b43fff640 1 -- 172.21.15.57:0/3980685342 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50050060 con 0x56009e7342e0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b43fff640 1 -- 172.21.15.57:0/3980685342 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f7b5004bbd0 con 0x56009e7342e0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.870+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5c035070 con 0x56009e7342e0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b43fff640 1 -- 172.21.15.57:0/3980685342 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b50057050 con 0x56009e7342e0 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 10 client.16248 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:39.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56009e7b7ae0 con 0x56009e7342e0 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b43fff640 1 -- 172.21.15.57:0/3980685342 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b5c035070 con 0x56009e7342e0 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b43fff640 1 -- 172.21.15.57:0/3980685342 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7b500046c0 con 0x56009e7342e0 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 10 client.16248 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 10 client.16248 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 20 client.16248 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7b82d0 0x56009e7ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 4 client.16248 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 74.0, "await": true}] 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 30.0, "expiration": 74.0, "await": true}) -- 0x56009e69f980 con 0x56009e7b82d0 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7b82d0 0x56009e7ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b43fff640 10 client.16248 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:39.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:47:39.871+0000 7f7b68ddf640 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7b82d0 0x56009e7ba6f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa489d12e160dc5a server_cookie=c6be69a6e31e19fa in_seq=0 out_seq=0 2024-04-21T08:47:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:39 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:47:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:39 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:47:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:39 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:47:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:40 smithi129 ceph-mon[31272]: pgmap v70: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 3.0 MiB/s wr, 1.28k op/s 2024-04-21T08:47:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:40 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3651466333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:40.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:40 smithi057 ceph-mon[27968]: pgmap v70: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 3.0 MiB/s wr, 1.28k op/s 2024-04-21T08:47:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:40 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3651466333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:40 smithi136 ceph-mon[31791]: pgmap v70: 129 pgs: 129 active+clean; 231 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 3.0 MiB/s wr, 1.28k op/s 2024-04-21T08:47:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:40 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3651466333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 -- 172.21.15.57:0/1719277964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1061b0 msgr2=0x7f43ac10dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 --2- 172.21.15.57:0/1719277964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1061b0 0x7f43ac10dd80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009ca0 tx=0x7f43a402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 -- 172.21.15.57:0/1719277964 shutdown_connections 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 --2- 172.21.15.57:0/1719277964 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1061b0 0x7f43ac10dd80 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 --2- 172.21.15.57:0/1719277964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43ac101a90 0x7f43ac105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 --2- 172.21.15.57:0/1719277964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43ac1010e0 0x7f43ac1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 -- 172.21.15.57:0/1719277964 >> 172.21.15.57:0/1719277964 conn(0x7f43ac0771d0 msgr2=0x7f43ac0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 -- 172.21.15.57:0/1719277964 shutdown_connections 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 -- 172.21.15.57:0/1719277964 wait complete. 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.485+0000 7f43b2255640 1 Processor -- start 2024-04-21T08:47:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 -- start start 2024-04-21T08:47:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 0x7f43ac198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43ac1061b0 0x7f43ac198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43ac19cee0 0x7f43ac19d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac111120 con 0x7f43ac1010e0 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac110fa0 con 0x7f43ac19cee0 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43b2255640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1112a0 con 0x7f43ac1061b0 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 0x7f43ac198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 0x7f43ac198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43388/0 (socket says 172.21.15.57:43388) 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 -- 172.21.15.57:0/58616649 learned_addr learned my addr 172.21.15.57:0/58616649 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43abfff640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43ac19cee0 0x7f43ac19d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43aaffd640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43ac1061b0 0x7f43ac198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 -- 172.21.15.57:0/58616649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43ac1061b0 msgr2=0x7f43ac198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43ac1061b0 0x7f43ac198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 -- 172.21.15.57:0/58616649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43ac19cee0 msgr2=0x7f43ac19d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43ac19cee0 0x7f43ac19d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 -- 172.21.15.57:0/58616649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac19d9d0 con 0x7f43ac1010e0 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.486+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 0x7f43ac198670 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f439c00d770 tx=0x7f439c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.487+0000 7f43a8ff9640 1 -- 172.21.15.57:0/58616649 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c017070 con 0x7f43ac1010e0 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.487+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ac19dcc0 con 0x7f43ac1010e0 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.487+0000 7f43a8ff9640 1 -- 172.21.15.57:0/58616649 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f439c0054d0 con 0x7f43ac1010e0 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.487+0000 7f43a8ff9640 1 -- 172.21.15.57:0/58616649 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c004c30 con 0x7f43ac1010e0 2024-04-21T08:47:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.487+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ac1a54b0 con 0x7f43ac1010e0 2024-04-21T08:47:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43a8ff9640 1 -- 172.21.15.57:0/58616649 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f439c004e10 con 0x7f43ac1010e0 2024-04-21T08:47:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 Processor -- start 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 -- start start 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a4d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a4dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 0x557102a594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5571029b28d0 con 0x557102a570e0 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55710294fe80 con 0x557102a4e8d0 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43b4e48740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5571029ae890 con 0x557102a4db60 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 0x557102a594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43abfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a4d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 0x557102a594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43404/0 (socket says 172.21.15.57:43404) 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 -- 172.21.15.57:0/3384574154 learned_addr learned my addr 172.21.15.57:0/3384574154 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43abfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a4d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40430/0 (socket says 172.21.15.57:40430) 2024-04-21T08:47:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43a8ff9640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f43940a3dc0 0x7f43940a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a4dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43abfff640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f43940a3dc0 0x7f43940a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43a8ff9640 1 -- 172.21.15.57:0/58616649 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f439c0d0580 con 0x7f43ac1010e0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 -- 172.21.15.57:0/3384574154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 msgr2=0x557102a4dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a4dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 -- 172.21.15.57:0/3384574154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 msgr2=0x557102a4d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a4d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.490+0000 7f43aaffd640 1 -- 172.21.15.57:0/3384574154 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557102971320 con 0x557102a570e0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.491+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a4dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.491+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 0x557102a594d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009c20 tx=0x7f43a0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373710dea38db966 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.491+0000 7f4391ffb640 1 -- 172.21.15.57:0/3384574154 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0004050 con 0x557102a570e0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.491+0000 7f4391ffb640 1 -- 172.21.15.57:0/3384574154 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a00054d0 con 0x557102a570e0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.491+0000 7f4391ffb640 1 -- 172.21.15.57:0/3384574154 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0004d40 con 0x557102a570e0 2024-04-21T08:47:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 -- 172.21.15.57:0/3384574154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 msgr2=0x557102a594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43abfff640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f43940a3dc0 0x7f43940a6280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f43a4005e30 tx=0x7f43a402e040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 0x557102a594d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009c20 tx=0x7f43a0011810 comp rx=0 tx=0).stop 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 -- 172.21.15.57:0/3384574154 shutdown_connections 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a570e0 0x557102a594d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a4dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3384574154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a4d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 -- 172.21.15.57:0/3384574154 >> 172.21.15.57:0/3384574154 conn(0x557102a14c90 msgr2=0x557102a48850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 -- 172.21.15.57:0/3384574154 shutdown_connections 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 -- 172.21.15.57:0/3384574154 wait complete. 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 -- start start 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 Processor -- start 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a56af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.492+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 0x557102a53e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a570e0 0x557102a54490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43b4e48740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557102a59e10 con 0x557102a4e8d0 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43b4e48740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55710293aeb0 con 0x557102a570e0 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43b4e48740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557102a59f90 con 0x557102a4db60 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43b4e48740 10 client.0 _ll_register_callbacks cb 0x557102967b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 0x557102a53e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 0x557102a53e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43406/0 (socket says 172.21.15.57:43406) 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 -- 172.21.15.57:0/3848445697 learned_addr learned my addr 172.21.15.57:0/3848445697 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43abfff640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a570e0 0x557102a54490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a56af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 -- 172.21.15.57:0/3848445697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 msgr2=0x557102a56af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a56af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 -- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a570e0 msgr2=0x557102a54490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a570e0 0x557102a54490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 -- 172.21.15.57:0/3848445697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571026c5210 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a56af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 0x557102a53e80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f43a0000c00 tx=0x7f43a0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866b166e27711b38 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f438affd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f438affd640 1 -- 172.21.15.57:0/3848445697 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a0029020 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f438affd640 1 -- 172.21.15.57:0/3848445697 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f43a0003c50 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.493+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43a00054d0 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f438affd640 1 -- 172.21.15.57:0/3848445697 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f43a0034050 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 10 client.16260 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557102791620 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f438affd640 1 -- 172.21.15.57:0/3848445697 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a00054d0 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f438affd640 1 -- 172.21.15.57:0/3848445697 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f43a00051a0 con 0x557102a4e8d0 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 10 client.16260 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:41.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 20 client.16260 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102a67170 0x557102a69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 4 client.16260 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571029c6880 con 0x557102a67170 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.494+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102a67170 0x557102a69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.495+0000 7f438affd640 10 client.16260 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:41.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.495+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102a67170 0x557102a69560 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c1a147718037fef server_cookie=962b4c3890875769 in_seq=0 out_seq=0 2024-04-21T08:47:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f438affd640 1 -- 172.21.15.57:0/3848445697 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571029c6880 con 0x557102a67170 2024-04-21T08:47:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f438affd640 10 client.16260 handle_command_reply: tid=0 2024-04-21T08:47:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 client.16260 shutdown 2024-04-21T08:47:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 msgr2=0x557102a53e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 0x557102a53e80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f43a0000c00 tx=0x7f43a0003820 comp rx=0 tx=0).stop 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 shutdown_connections 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102a67170 0x557102a69560 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a570e0 0x557102a54490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f438affd640 0 client.16260 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4e8d0 0x557102a53e80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3848445697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a4db60 0x557102a56af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 >> 172.21.15.57:0/3848445697 conn(0x557102a14c90 msgr2=0x557102a4a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 shutdown_connections 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 1 -- 172.21.15.57:0/3848445697 wait complete. 2024-04-21T08:47:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.519+0000 7f43b4e48740 20 client.16260 trim_cache size 0 max 16384 2024-04-21T08:47:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 Processor -- start 2024-04-21T08:47:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 -- start start 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a62d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a565b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a56af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1ad100 con 0x557102a4db60 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1af960 con 0x557102a4e8d0 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43b4e48740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1ad750 con 0x557102a570e0 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a62d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a62d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43410/0 (socket says 172.21.15.57:43410) 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43ab7fe640 1 -- 172.21.15.57:0/1271459344 learned_addr learned my addr 172.21.15.57:0/1271459344 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43aaffd640 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.535+0000 7f43abfff640 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a56af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43ab7fe640 1 -- 172.21.15.57:0/1271459344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 msgr2=0x557102a56af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a56af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43ab7fe640 1 -- 172.21.15.57:0/1271459344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 msgr2=0x557102a565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43ab7fe640 1 -- 172.21.15.57:0/1271459344 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557102a4ce30 con 0x557102a4db60 2024-04-21T08:47:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a62d10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f439c00d740 tx=0x7f439c06d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce463ece1274f59f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43917fa640 1 -- 172.21.15.57:0/1271459344 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c0a4e30 con 0x557102a4db60 2024-04-21T08:47:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43917fa640 1 -- 172.21.15.57:0/1271459344 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f439c01fe50 con 0x557102a4db60 2024-04-21T08:47:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.536+0000 7f43917fa640 1 -- 172.21.15.57:0/1271459344 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c0d01d0 con 0x557102a4db60 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 -- 172.21.15.57:0/1271459344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 msgr2=0x557102a62d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a62d10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f439c00d740 tx=0x7f439c06d040 comp rx=0 tx=0).stop 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 -- 172.21.15.57:0/1271459344 shutdown_connections 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a56af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 --2- 172.21.15.57:0/1271459344 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a62d10 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 -- 172.21.15.57:0/1271459344 >> 172.21.15.57:0/1271459344 conn(0x557102a14c90 msgr2=0x557102a48380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 -- 172.21.15.57:0/1271459344 shutdown_connections 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 -- 172.21.15.57:0/1271459344 wait complete. 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 -- start start 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.537+0000 7f43b4e48740 1 Processor -- start 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a574c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a57a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a57f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1ade20 con 0x557102a4db60 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1af290 con 0x557102a4e8d0 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac1ae4f0 con 0x557102a570e0 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 10 client.0 _ll_register_callbacks cb 0x7f43a0009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43426/0 (socket says 172.21.15.57:43426) 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 -- 172.21.15.57:0/3704307003 learned_addr learned my addr 172.21.15.57:0/3704307003 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a57a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43abfff640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a57f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 -- 172.21.15.57:0/3704307003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 msgr2=0x557102a57f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a57f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.542 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 -- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 msgr2=0x557102a57a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a57a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 -- 172.21.15.57:0/3704307003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557102a5e320 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43ab7fe640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a574c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f439c0cda20 tx=0x7f439c005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f915d0c7aa288373 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f438affd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f438affd640 1 -- 172.21.15.57:0/3704307003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439c0a4e30 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f438affd640 1 -- 172.21.15.57:0/3704307003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f439c0cdbf0 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f439c01fe50 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f438affd640 1 -- 172.21.15.57:0/3704307003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f439c0988e0 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 10 client.16266 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.538+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557102a5f5b0 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f438affd640 1 -- 172.21.15.57:0/3704307003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f439c01fe50 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f438affd640 1 -- 172.21.15.57:0/3704307003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f439c0d01d0 con 0x557102a4db60 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43b4e48740 10 client.16266 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43b4e48740 10 client.16266 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43b4e48740 20 client.16266 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102acd540 0x557102acf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43b4e48740 4 client.16266 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5571029523d0 con 0x557102acd540 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.539+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102acd540 0x557102acf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.540+0000 7f438affd640 10 client.16266 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:41.540+0000 7f43aaffd640 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102acd540 0x557102acf930 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0122cec1240719a server_cookie=ccf8eecfa22c9938 in_seq=0 out_seq=0 2024-04-21T08:47:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:42 smithi129 ceph-mon[31272]: pgmap v71: 129 pgs: 129 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 1.32k op/s 2024-04-21T08:47:42.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:42 smithi057 ceph-mon[27968]: pgmap v71: 129 pgs: 129 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 1.32k op/s 2024-04-21T08:47:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:42 smithi136 ceph-mon[31791]: pgmap v71: 129 pgs: 129 active+clean; 245 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 1.32k op/s 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.655+0000 7f438affd640 1 -- 172.21.15.57:0/3704307003 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5571029523d0 con 0x557102acd540 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.655+0000 7f438affd640 10 client.16266 handle_command_reply: tid=0 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.655+0000 7f43b4e48740 1 client.16266 shutdown 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.655+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 msgr2=0x557102a574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.655+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a574c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f439c0cda20 tx=0x7f439c005680 comp rx=0 tx=0).stop 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 shutdown_connections 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557102acd540 0x557102acf930 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557102a570e0 0x557102a57f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557102a4e8d0 0x557102a57a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f438affd640 0 client.16266 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 --2- 172.21.15.57:0/3704307003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557102a4db60 0x557102a574c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 >> 172.21.15.57:0/3704307003 conn(0x557102a14c90 msgr2=0x557102a4eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 shutdown_connections 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 1 -- 172.21.15.57:0/3704307003 wait complete. 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b4e48740 20 client.16266 trim_cache size 0 max 16384 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:47:44.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f43940a3dc0 msgr2=0x7f43940a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f43940a3dc0 0x7f43940a6280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f43a4005e30 tx=0x7f43a402e040 comp rx=0 tx=0).stop 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 msgr2=0x7f43ac198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 0x7f43ac198670 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f439c00d770 tx=0x7f439c00dc30 comp rx=0 tx=0).stop 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 shutdown_connections 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f43940a3dc0 0x7f43940a6280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.658 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43ac19cee0 0x7f43ac19d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.659 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43ac1061b0 0x7f43ac198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.660 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 --2- 172.21.15.57:0/58616649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f43ac1010e0 0x7f43ac198670 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:44.660 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.656+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 >> 172.21.15.57:0/58616649 conn(0x7f43ac0771d0 msgr2=0x7f43ac06a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:44.660 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.657+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 shutdown_connections 2024-04-21T08:47:44.660 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:44.657+0000 7f43b2255640 1 -- 172.21.15.57:0/58616649 wait complete. 2024-04-21T08:47:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:44 smithi129 ceph-mon[31272]: pgmap v72: 129 pgs: 129 active+clean; 257 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 1.24k op/s 2024-04-21T08:47:44.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:44 smithi057 ceph-mon[27968]: pgmap v72: 129 pgs: 129 active+clean; 257 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 1.24k op/s 2024-04-21T08:47:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:44 smithi136 ceph-mon[31791]: pgmap v72: 129 pgs: 129 active+clean; 257 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 1.24k op/s 2024-04-21T08:47:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:46 smithi129 ceph-mon[31272]: pgmap v73: 129 pgs: 129 active+clean; 257 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 481 op/s 2024-04-21T08:47:46.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:46 smithi057 ceph-mon[27968]: pgmap v73: 129 pgs: 129 active+clean; 257 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 481 op/s 2024-04-21T08:47:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:46 smithi136 ceph-mon[31791]: pgmap v73: 129 pgs: 129 active+clean; 257 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 481 op/s 2024-04-21T08:47:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:48 smithi129 ceph-mon[31272]: pgmap v74: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 1.72k op/s 2024-04-21T08:47:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:48 smithi057 ceph-mon[27968]: pgmap v74: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 1.72k op/s 2024-04-21T08:47:48.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:48 smithi136 ceph-mon[31791]: pgmap v74: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 5.4 MiB/s wr, 1.72k op/s 2024-04-21T08:47:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:50 smithi129 ceph-mon[31272]: pgmap v75: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 1.45k op/s 2024-04-21T08:47:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:50 smithi057 ceph-mon[27968]: pgmap v75: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 1.45k op/s 2024-04-21T08:47:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:50 smithi136 ceph-mon[31791]: pgmap v75: 129 pgs: 129 active+clean; 239 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 1.45k op/s 2024-04-21T08:47:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:52 smithi129 ceph-mon[31272]: pgmap v76: 129 pgs: 129 active+clean; 243 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 1.61k op/s 2024-04-21T08:47:52.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:52 smithi057 ceph-mon[27968]: pgmap v76: 129 pgs: 129 active+clean; 243 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 1.61k op/s 2024-04-21T08:47:52.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:52 smithi136 ceph-mon[31791]: pgmap v76: 129 pgs: 129 active+clean; 243 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.2 MiB/s wr, 1.61k op/s 2024-04-21T08:47:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:54 smithi129 ceph-mon[31272]: pgmap v77: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 2.2 MiB/s wr, 1.80k op/s 2024-04-21T08:47:54.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:54 smithi057 ceph-mon[27968]: pgmap v77: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 2.2 MiB/s wr, 1.80k op/s 2024-04-21T08:47:54.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:54 smithi136 ceph-mon[31791]: pgmap v77: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 2.2 MiB/s wr, 1.80k op/s 2024-04-21T08:47:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:56 smithi129 ceph-mon[31272]: pgmap v78: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.1 MiB/s wr, 1.63k op/s 2024-04-21T08:47:56.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:56 smithi057 ceph-mon[27968]: pgmap v78: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.1 MiB/s wr, 1.63k op/s 2024-04-21T08:47:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:56 smithi136 ceph-mon[31791]: pgmap v78: 129 pgs: 129 active+clean; 244 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.1 MiB/s wr, 1.63k op/s 2024-04-21T08:47:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 -- 172.21.15.57:0/2830529490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f200810c5a0 msgr2=0x7f200810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 --2- 172.21.15.57:0/2830529490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f200810c5a0 0x7f200810e9d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2000009be0 tx=0x7f200002bf80 comp rx=0 tx=0).stop 2024-04-21T08:47:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 -- 172.21.15.57:0/2830529490 shutdown_connections 2024-04-21T08:47:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 --2- 172.21.15.57:0/2830529490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f200810c5a0 0x7f200810e9d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 --2- 172.21.15.57:0/2830529490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2008103af0 0x7f200810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 --2- 172.21.15.57:0/2830529490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f20081035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 -- 172.21.15.57:0/2830529490 >> 172.21.15.57:0/2830529490 conn(0x7f2008077fa0 msgr2=0x7f20080783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.772+0000 7f200d7d5640 1 -- 172.21.15.57:0/2830529490 shutdown_connections 2024-04-21T08:47:57.774 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.773+0000 7f200d7d5640 1 -- 172.21.15.57:0/2830529490 wait complete. 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.773+0000 7f200d7d5640 1 Processor -- start 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 -- start start 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f200819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2008103af0 0x7f200819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f200819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200810c5a0 0x7f200819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f200819a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:46322/0 (socket says 172.21.15.57:46322) 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f2006ffd640 1 -- 172.21.15.57:0/1638062824 learned_addr learned my addr 172.21.15.57:0/1638062824 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2008102090 con 0x7f2008103af0 2024-04-21T08:47:57.776 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2008101f10 con 0x7f200810c5a0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2008102210 con 0x7f20081031d0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.774+0000 7f20067fc640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2008103af0 0x7f200819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f20077fe640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200810c5a0 0x7f200819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f2006ffd640 1 -- 172.21.15.57:0/1638062824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200810c5a0 msgr2=0x7f200819eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f2006ffd640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200810c5a0 0x7f200819eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f2006ffd640 1 -- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2008103af0 msgr2=0x7f200819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f2006ffd640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2008103af0 0x7f200819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f2006ffd640 1 -- 172.21.15.57:0/1638062824 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200819f600 con 0x7f20081031d0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f20077fe640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200810c5a0 0x7f200819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f2006ffd640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f200819a610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800ea60 tx=0x7f1ff800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f1feffff640 1 -- 172.21.15.57:0/1638062824 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff800cd20 con 0x7f20081031d0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f1feffff640 1 -- 172.21.15.57:0/1638062824 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff80035f0 con 0x7f20081031d0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.775+0000 7f1feffff640 1 -- 172.21.15.57:0/1638062824 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8010690 con 0x7f20081031d0 2024-04-21T08:47:57.777 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.776+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200819f8f0 con 0x7f20081031d0 2024-04-21T08:47:57.778 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.776+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20081a7180 con 0x7f20081031d0 2024-04-21T08:47:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 Processor -- start 2024-04-21T08:47:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 -- start start 2024-04-21T08:47:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 0x555b61856620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f1feffff640 1 -- 172.21.15.57:0/1638062824 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1ff8004020 con 0x7f20081031d0 2024-04-21T08:47:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61856fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.779 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b618624d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x555b617bb8d0 con 0x555b618578d0 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.778+0000 7f20103c9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555b61758e80 con 0x555b618600e0 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f20103c9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x555b617b7890 con 0x555b61856b60 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f20077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 0x555b61856620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61856fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f20077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 0x555b61856620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43820/0 (socket says 172.21.15.57:43820) 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f20077fe640 1 -- 172.21.15.57:0/1026511682 learned_addr learned my addr 172.21.15.57:0/1026511682 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:57.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f20067fc640 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b618624d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.782 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 -- 172.21.15.57:0/1026511682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 msgr2=0x555b618624d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b618624d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 -- 172.21.15.57:0/1026511682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 msgr2=0x555b61856620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 0x555b61856620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 -- 172.21.15.57:0/1026511682 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b6177a320 con 0x555b61856b60 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f20077fe640 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 0x555b61856620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f1feffff640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1ff00a3e90 0x7f1ff00a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.779+0000 7f2006ffd640 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61856fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80ce270 tx=0x7f1ff80d94b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a618c982b26214d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.780+0000 7f20077fe640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1ff00a3e90 0x7f1ff00a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.780+0000 7f1feffff640 1 -- 172.21.15.57:0/1638062824 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1ff80d0ee0 con 0x7f20081031d0 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.780+0000 7f1fed7fa640 1 -- 172.21.15.57:0/1026511682 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80e4070 con 0x555b61856b60 2024-04-21T08:47:57.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.780+0000 7f1fed7fa640 1 -- 172.21.15.57:0/1026511682 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff8004310 con 0x555b61856b60 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.780+0000 7f1fed7fa640 1 -- 172.21.15.57:0/1026511682 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff809c890 con 0x555b61856b60 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 -- 172.21.15.57:0/1026511682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 msgr2=0x555b61856fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61856fe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80ce270 tx=0x7f1ff80d94b0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20077fe640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1ff00a3e90 0x7f1ff00a6350 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2000009b30 tx=0x7f2000005e30 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 -- 172.21.15.57:0/1026511682 shutdown_connections 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b618624d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61856fe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 --2- 172.21.15.57:0/1026511682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618578d0 0x555b61856620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 -- 172.21.15.57:0/1026511682 >> 172.21.15.57:0/1026511682 conn(0x555b6181dc90 msgr2=0x555b61851850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 -- 172.21.15.57:0/1026511682 shutdown_connections 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.781+0000 7f20103c9740 1 -- 172.21.15.57:0/1026511682 wait complete. 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.782+0000 7f20103c9740 1 -- start start 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.782+0000 7f20103c9740 1 Processor -- start 2024-04-21T08:47:57.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.782+0000 7f20103c9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b6185fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.782+0000 7f20103c9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b6185ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.787 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.782+0000 7f20103c9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b6185d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b6185fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20103c9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x555b61862e10 con 0x555b618600e0 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20103c9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x555b61743eb0 con 0x555b618578d0 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20103c9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x555b61862f90 con 0x555b61856b60 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b6185fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:46346/0 (socket says 172.21.15.57:46346) 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 -- 172.21.15.57:0/3561649456 learned_addr learned my addr 172.21.15.57:0/3561649456 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20103c9740 10 client.0 _ll_register_callbacks cb 0x555b61770b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20067fc640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b6185ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20077fe640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b6185d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 -- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 msgr2=0x555b6185ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b6185ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 -- 172.21.15.57:0/3561649456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 msgr2=0x555b6185d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b6185d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 -- 172.21.15.57:0/3561649456 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b614ce210 con 0x555b61856b60 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20067fc640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b6185ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20077fe640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b6185d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f2006ffd640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b6185fb00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8000c00 tx=0x7f1ff800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=302755158be94d68 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f1feaff5640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f1feaff5640 1 -- 172.21.15.57:0/3561649456 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff809c420 con 0x555b61856b60 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b617cf880 con 0x555b61856b60 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.783+0000 7f1feaff5640 1 -- 172.21.15.57:0/3561649456 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f1ff8003ac0 con 0x555b61856b60 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.784+0000 7f1feaff5640 1 -- 172.21.15.57:0/3561649456 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1ff809c600 con 0x555b61856b60 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.784+0000 7f20103c9740 10 client.25690 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.784+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555b6159a620 con 0x555b61856b60 2024-04-21T08:47:57.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.784+0000 7f1feaff5640 1 -- 172.21.15.57:0/3561649456 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x555b617cf880 con 0x555b61856b60 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.784+0000 7f1feaff5640 1 -- 172.21.15.57:0/3561649456 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1ff80ce4f0 con 0x555b61856b60 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20103c9740 10 client.25690 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20103c9740 10 client.25690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20103c9740 20 client.25690 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20103c9740 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20103c9740 4 client.25690 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b6175b3d0 con 0x555b61870000 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.785+0000 7f20067fc640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.786+0000 7f20067fc640 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618723f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94dec44906d87c59 server_cookie=84418bec458e896c in_seq=0 out_seq=0 2024-04-21T08:47:57.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.786+0000 7f1feaff5640 10 client.25690 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:57.791 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f1feaff5640 1 -- 172.21.15.57:0/3561649456 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555b6175b3d0 con 0x555b61870000 2024-04-21T08:47:57.791 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f1feaff5640 10 client.25690 handle_command_reply: tid=0 2024-04-21T08:47:57.791 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 client.25690 shutdown 2024-04-21T08:47:57.791 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 msgr2=0x555b6185fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b6185fb00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8000c00 tx=0x7f1ff800ef90 comp rx=0 tx=0).stop 2024-04-21T08:47:57.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 shutdown_connections 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618723f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f1feaff5640 0 client.25690 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b6185d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b6185ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 --2- 172.21.15.57:0/3561649456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b6185fb00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 >> 172.21.15.57:0/3561649456 conn(0x555b6181dc90 msgr2=0x555b618535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.791+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 shutdown_connections 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.792+0000 7f20103c9740 1 -- 172.21.15.57:0/3561649456 wait complete. 2024-04-21T08:47:57.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.792+0000 7f20103c9740 20 client.25690 trim_cache size 0 max 16384 2024-04-21T08:47:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 Processor -- start 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 -- start start 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 0x555b6185f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b618578d0 0x555b6185fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b61875410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b3630 con 0x555b61856b60 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b5e70 con 0x555b618600e0 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.813+0000 7f20103c9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b3c80 con 0x555b618578d0 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 0x555b6185f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 0x555b6185f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:43852/0 (socket says 172.21.15.57:43852) 2024-04-21T08:47:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 -- 172.21.15.57:0/3129758913 learned_addr learned my addr 172.21.15.57:0/3129758913 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f20067fc640 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b618578d0 0x555b6185fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f20077fe640 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b61875410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 -- 172.21.15.57:0/3129758913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b618578d0 msgr2=0x555b6185fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b618578d0 0x555b6185fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 -- 172.21.15.57:0/3129758913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 msgr2=0x555b61875410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b61875410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 -- 172.21.15.57:0/3129758913 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b61855e30 con 0x555b61856b60 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f2006ffd640 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 0x555b6185f5c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80d11d0 tx=0x7f1ff80a5730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc41d15059262530 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f1fecff9640 1 -- 172.21.15.57:0/3129758913 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff809c420 con 0x555b61856b60 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f1fecff9640 1 -- 172.21.15.57:0/3129758913 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff809d1c0 con 0x555b61856b60 2024-04-21T08:47:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.814+0000 7f1fecff9640 1 -- 172.21.15.57:0/3129758913 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff809c890 con 0x555b61856b60 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 -- 172.21.15.57:0/3129758913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 msgr2=0x555b6185f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 0x555b6185f5c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80d11d0 tx=0x7f1ff80a5730 comp rx=0 tx=0).stop 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 -- 172.21.15.57:0/3129758913 shutdown_connections 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618600e0 0x555b61875410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b618578d0 0x555b6185fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 --2- 172.21.15.57:0/3129758913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b61856b60 0x555b6185f5c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 -- 172.21.15.57:0/3129758913 >> 172.21.15.57:0/3129758913 conn(0x555b6181dc90 msgr2=0x555b61851380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 -- 172.21.15.57:0/3129758913 shutdown_connections 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 -- 172.21.15.57:0/3129758913 wait complete. 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 -- start start 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.815+0000 7f20103c9740 1 Processor -- start 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61875400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b61875940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b61872990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b4330 con 0x555b618600e0 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b57a0 con 0x555b618578d0 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f20081b4a00 con 0x555b61856b60 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20103c9740 10 client.0 _ll_register_callbacks cb 0x7f1ff80e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b61872990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61875400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f2006ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61875400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:46358/0 (socket says 172.21.15.57:46358) 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f2006ffd640 1 -- 172.21.15.57:0/2875333136 learned_addr learned my addr 172.21.15.57:0/2875333136 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20067fc640 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b61875940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 -- 172.21.15.57:0/2875333136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 msgr2=0x555b61875400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61875400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 -- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 msgr2=0x555b61875940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:47:57.823 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b61875940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 -- 172.21.15.57:0/2875333136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b61873190 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f20077fe640 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b61872990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f20000099a0 tx=0x7f2000005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82206da4357cee6d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f1feaff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f1feaff5640 1 -- 172.21.15.57:0/2875333136 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200004d630 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.816+0000 7f1feaff5640 1 -- 172.21.15.57:0/2875333136 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f200004d7d0 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ff809d1c0 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f1feaff5640 1 -- 172.21.15.57:0/2875333136 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f200005e050 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 10 client.16281 fetch_fsmap learned FSMap version 21 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x555b618d5e20 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f1feaff5640 1 -- 172.21.15.57:0/2875333136 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ff809d1c0 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f1feaff5640 1 -- 172.21.15.57:0/2875333136 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f200003fb30 con 0x555b618600e0 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 10 client.16281 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 10 client.16281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:47:57.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 20 client.16281 populate_metadata read hostname 'smithi057' 2024-04-21T08:47:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618d85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:47:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 4 client.16281 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:47:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555b617c92f0 con 0x555b61870000 2024-04-21T08:47:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.817+0000 7f2006ffd640 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618d85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:47:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.818+0000 7f1feaff5640 10 client.16281 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:47:57.825 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:47:57.818+0000 7f2006ffd640 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618d85b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31bc42e736a84216 server_cookie=a1b7f94163bd02dc in_seq=0 out_seq=0 2024-04-21T08:47:58.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:47:58 smithi129 ceph-mon[31272]: pgmap v79: 129 pgs: 129 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 2.18k op/s 2024-04-21T08:47:58.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:47:58 smithi057 ceph-mon[27968]: pgmap v79: 129 pgs: 129 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 2.18k op/s 2024-04-21T08:47:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:47:58 smithi136 ceph-mon[31791]: pgmap v79: 129 pgs: 129 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 2.18k op/s 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b43fff640 1 -- 172.21.15.57:0/3980685342 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x56009e69f980 con 0x56009e7b82d0 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b43fff640 10 client.16248 handle_command_reply: tid=0 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 client.16248 shutdown 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 msgr2=0x56009e74b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e74b950 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7b500041d0 tx=0x7f7b50004430 comp rx=0 tx=0).stop 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 shutdown_connections 2024-04-21T08:48:00.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56009e7b82d0 0x56009e7ba6f0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56009e73cec0 0x56009e7b6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56009e7342e0 0x56009e74b950 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 --2- 172.21.15.57:0/3980685342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56009e7339c0 0x56009e74b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b43fff640 0 client.16248 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 >> 172.21.15.57:0/3980685342 conn(0x56009e72e0f0 msgr2=0x56009e72c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 shutdown_connections 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 1 -- 172.21.15.57:0/3980685342 wait complete. 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.023+0000 7f7b6dc3c740 20 client.16248 trim_cache size 0 max 16384 2024-04-21T08:48:00.025 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "epoch": 21, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "leader": 24503, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "set_version": 2, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "sets": { 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "bad5ac53": { 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "version": 2, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "age_ref": 0.0, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "state": { 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "QUIESCED", 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 0.0 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "timeout": 30.0, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "expiration": 74.0, 2024-04-21T08:48:00.026 INFO:teuthology.orchestra.run.smithi057.stdout: "members": { 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: "file:/": { 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: "excluded": false, 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: "state": { 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "QUIESCED", 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 0.0 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:48:00.027 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7b540a3d20 msgr2=0x7f7b540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7b540a3d20 0x7f7b540a61e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c021000 tx=0x7f7b5c01f000 comp rx=0 tx=0).stop 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 msgr2=0x7f7b6419e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 0x7f7b6419e3e0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800e9f0 tx=0x7f7b5800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 shutdown_connections 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7b540a3d20 0x7f7b540a61e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7b641985e0 0x7f7b64198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b64107260 0x7f7b6419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 --2- 172.21.15.57:0/881917868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b64102190 0x7f7b6419e3e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:00.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 >> 172.21.15.57:0/881917868 conn(0x7f7b64079310 msgr2=0x7f7b64100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:00.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 shutdown_connections 2024-04-21T08:48:00.029 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:00.024+0000 7f7b6b043640 1 -- 172.21.15.57:0/881917868 wait complete. 2024-04-21T08:48:00.039 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced, set_id: bad5ac53, quiesce duration: 14.0 2024-04-21T08:48:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:00 smithi129 ceph-mon[31272]: pgmap v80: 129 pgs: 129 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 882 KiB/s wr, 939 op/s 2024-04-21T08:48:00.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:00 smithi057 ceph-mon[27968]: pgmap v80: 129 pgs: 129 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 882 KiB/s wr, 939 op/s 2024-04-21T08:48:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:00 smithi136 ceph-mon[31791]: pgmap v80: 129 pgs: 129 active+clean; 248 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 882 KiB/s wr, 939 op/s 2024-04-21T08:48:02.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:02 smithi129 ceph-mon[31272]: pgmap v81: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 1.10k op/s 2024-04-21T08:48:02.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:02 smithi057 ceph-mon[27968]: pgmap v81: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 1.10k op/s 2024-04-21T08:48:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:02 smithi136 ceph-mon[31791]: pgmap v81: 129 pgs: 129 active+clean; 251 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 1.10k op/s 2024-04-21T08:48:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:04 smithi129 ceph-mon[31272]: pgmap v82: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 1.31k op/s 2024-04-21T08:48:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:04 smithi057 ceph-mon[27968]: pgmap v82: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 1.31k op/s 2024-04-21T08:48:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:04 smithi136 ceph-mon[31791]: pgmap v82: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 1.31k op/s 2024-04-21T08:48:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:06 smithi129 ceph-mon[31272]: pgmap v83: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.08k op/s 2024-04-21T08:48:06.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:06 smithi057 ceph-mon[27968]: pgmap v83: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.08k op/s 2024-04-21T08:48:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:06 smithi136 ceph-mon[31791]: pgmap v83: 129 pgs: 129 active+clean; 252 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.08k op/s 2024-04-21T08:48:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:08 smithi129 ceph-mon[31272]: pgmap v84: 129 pgs: 129 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.09k op/s 2024-04-21T08:48:08.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:08 smithi057 ceph-mon[27968]: pgmap v84: 129 pgs: 129 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.09k op/s 2024-04-21T08:48:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:08 smithi136 ceph-mon[31791]: pgmap v84: 129 pgs: 129 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.09k op/s 2024-04-21T08:48:09.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:48:09.065 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 -- 172.21.15.57:0/2278849157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806aed0 msgr2=0x7f383810e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 --2- 172.21.15.57:0/2278849157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806aed0 0x7f383810e710 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f382c009930 tx=0x7f382c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 -- 172.21.15.57:0/2278849157 shutdown_connections 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 --2- 172.21.15.57:0/2278849157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383810ec50 0x7f3838111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 --2- 172.21.15.57:0/2278849157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806aed0 0x7f383810e710 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 --2- 172.21.15.57:0/2278849157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f383806a5b0 0x7f383806a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 -- 172.21.15.57:0/2278849157 >> 172.21.15.57:0/2278849157 conn(0x7f3838077c40 msgr2=0x7f3838078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 -- 172.21.15.57:0/2278849157 shutdown_connections 2024-04-21T08:48:09.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 -- 172.21.15.57:0/2278849157 wait complete. 2024-04-21T08:48:09.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.182+0000 7f383fac1640 1 Processor -- start 2024-04-21T08:48:09.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 -- start start 2024-04-21T08:48:09.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 0x7f38381a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383806aed0 0x7f38381a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f383810ec50 0x7f383819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f38381143d0 con 0x7f383806a5b0 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3838114250 con 0x7f383806aed0 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383fac1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3838114550 con 0x7f383810ec50 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 0x7f38381a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 0x7f38381a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50956/0 (socket says 172.21.15.57:50956) 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 -- 172.21.15.57:0/2619130349 learned_addr learned my addr 172.21.15.57:0/2619130349 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383e05e640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f383810ec50 0x7f383819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d05c640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383806aed0 0x7f38381a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 -- 172.21.15.57:0/2619130349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f383810ec50 msgr2=0x7f383819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f383810ec50 0x7f383819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 -- 172.21.15.57:0/2619130349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383806aed0 msgr2=0x7f38381a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383806aed0 0x7f38381a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 -- 172.21.15.57:0/2619130349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383819af90 con 0x7f383806a5b0 2024-04-21T08:48:09.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.183+0000 7f383d85d640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 0x7f38381a0610 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f382800eab0 tx=0x7f382800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.184+0000 7f3836ffd640 1 -- 172.21.15.57:0/2619130349 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382800ce40 con 0x7f383806a5b0 2024-04-21T08:48:09.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.184+0000 7f3836ffd640 1 -- 172.21.15.57:0/2619130349 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3828011920 con 0x7f383806a5b0 2024-04-21T08:48:09.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.184+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383819b280 con 0x7f383806a5b0 2024-04-21T08:48:09.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.184+0000 7f3836ffd640 1 -- 172.21.15.57:0/2619130349 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3828010a30 con 0x7f383806a5b0 2024-04-21T08:48:09.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.184+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38381a7590 con 0x7f383806a5b0 2024-04-21T08:48:09.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.187+0000 7f3836ffd640 1 -- 172.21.15.57:0/2619130349 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3828010bd0 con 0x7f383806a5b0 2024-04-21T08:48:09.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.187+0000 7f384278c740 1 Processor -- start 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.187+0000 7f384278c740 1 -- start start 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.187+0000 7f384278c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69da580 0x55d9f69da960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.187+0000 7f384278c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69e3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.187+0000 7f384278c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69e3a80 0x55d9f69e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f384278c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55d9f693f1e0 con 0x55d9f69daea0 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f384278c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d9f68dc390 con 0x55d9f69e3a80 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f384278c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d9f693b1a0 con 0x55d9f69da580 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69e3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69e3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50960/0 (socket says 172.21.15.57:50960) 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69e3a80 0x55d9f69e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d85d640 1 -- 172.21.15.57:0/2724020067 learned_addr learned my addr 172.21.15.57:0/2724020067 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:09.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f3836ffd640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f38200a3d20 0x7f38200a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383e05e640 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69da580 0x55d9f69da960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383e05e640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f38200a3d20 0x7f38200a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f3836ffd640 1 -- 172.21.15.57:0/2619130349 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f38280d1120 con 0x7f383806a5b0 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 -- 172.21.15.57:0/2724020067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69da580 msgr2=0x55d9f69da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69da580 0x55d9f69da960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 -- 172.21.15.57:0/2724020067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 msgr2=0x55d9f69e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69e3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 -- 172.21.15.57:0/2724020067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f671b7d0 con 0x55d9f69e3a80 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383e05e640 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69da580 0x55d9f69da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383d05c640 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69e3a80 0x55d9f69e5e70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f382c009230 tx=0x7f382c005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b548cfcaab3d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.188+0000 7f383e05e640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f38200a3d20 0x7f38200a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f383806a380 tx=0x7f383000b2e0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f381ffff640 1 -- 172.21.15.57:0/2724020067 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c045020 con 0x55d9f69e3a80 2024-04-21T08:48:09.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f381ffff640 1 -- 172.21.15.57:0/2724020067 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c02c7e0 con 0x55d9f69e3a80 2024-04-21T08:48:09.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f381ffff640 1 -- 172.21.15.57:0/2724020067 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382c005450 con 0x55d9f69e3a80 2024-04-21T08:48:09.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 -- 172.21.15.57:0/2724020067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69e3a80 msgr2=0x55d9f69e5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:09.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69e3a80 0x55d9f69e5e70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f382c009230 tx=0x7f382c005c00 comp rx=0 tx=0).stop 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 -- 172.21.15.57:0/2724020067 shutdown_connections 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69e3a80 0x55d9f69e5e70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69e3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 --2- 172.21.15.57:0/2724020067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69da580 0x55d9f69da960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.189+0000 7f384278c740 1 -- 172.21.15.57:0/2724020067 >> 172.21.15.57:0/2724020067 conn(0x55d9f69d4cb0 msgr2=0x55d9f69d5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 -- 172.21.15.57:0/2724020067 shutdown_connections 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 -- 172.21.15.57:0/2724020067 wait complete. 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 -- start start 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 Processor -- start 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69e09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55d9f69e6810 con 0x55d9f69da580 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d9f68f82f0 con 0x55d9f69daea0 2024-04-21T08:48:09.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.190+0000 7f384278c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d9f69e6990 con 0x55d9f69e3a80 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f384278c740 10 client.0 _ll_register_callbacks cb 0x55d9f6905580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69e09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69e09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50974/0 (socket says 172.21.15.57:50974) 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 -- 172.21.15.57:0/695800024 learned_addr learned my addr 172.21.15.57:0/695800024 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d05c640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383e05e640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 -- 172.21.15.57:0/695800024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 msgr2=0x55d9f69e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 -- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 msgr2=0x55d9f69e0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 -- 172.21.15.57:0/695800024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f66513f0 con 0x55d9f69da580 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f383d85d640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69e09d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f38280a5440 tx=0x7f382809c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46dcfd25116c6afb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:09.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f381d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f381d7fa640 1 -- 172.21.15.57:0/695800024 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382809cd00 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f381d7fa640 1 -- 172.21.15.57:0/695800024 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f3828098a50 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382c02c7e0 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f381d7fa640 1 -- 172.21.15.57:0/695800024 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f38280a10b0 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f384278c740 10 client.16293 fetch_fsmap learned FSMap version 21 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.191+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d9f66110f0 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f381d7fa640 1 -- 172.21.15.57:0/695800024 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382c02c7e0 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f381d7fa640 1 -- 172.21.15.57:0/695800024 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f382800f040 con 0x55d9f69da580 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f384278c740 10 client.16293 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f384278c740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f384278c740 20 client.16293 populate_metadata read hostname 'smithi057' 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f384278c740 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f69f39c0 0x55d9f69f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f384278c740 4 client.16293 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9f694cc00 con 0x55d9f69f39c0 2024-04-21T08:48:09.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.192+0000 7f383d05c640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f69f39c0 0x55d9f69f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:09.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.193+0000 7f381d7fa640 10 client.16293 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:48:09.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:09.193+0000 7f383d05c640 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f69f39c0 0x55d9f69f5e50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14a293cd28a0f713 server_cookie=faf1b1d1b7dfb03f in_seq=0 out_seq=0 2024-04-21T08:48:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:10 smithi129 ceph-mon[31272]: pgmap v85: 129 pgs: 129 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 701 KiB/s wr, 538 op/s 2024-04-21T08:48:10.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:10 smithi057 ceph-mon[27968]: pgmap v85: 129 pgs: 129 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 701 KiB/s wr, 538 op/s 2024-04-21T08:48:10.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:10 smithi136 ceph-mon[31791]: pgmap v85: 129 pgs: 129 active+clean; 253 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 701 KiB/s wr, 538 op/s 2024-04-21T08:48:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:12 smithi129 ceph-mon[31272]: pgmap v86: 129 pgs: 129 active+clean; 254 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 847 KiB/s wr, 554 op/s 2024-04-21T08:48:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:12 smithi057 ceph-mon[27968]: pgmap v86: 129 pgs: 129 active+clean; 254 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 847 KiB/s wr, 554 op/s 2024-04-21T08:48:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:12 smithi136 ceph-mon[31791]: pgmap v86: 129 pgs: 129 active+clean; 254 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 847 KiB/s wr, 554 op/s 2024-04-21T08:48:14.040 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 -- 172.21.15.57:0/1211851330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7110102b50 msgr2=0x7f7110106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 --2- 172.21.15.57:0/1211851330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7110102b50 0x7f7110106b90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7104009900 tx=0x7f710402bc30 comp rx=0 tx=0).stop 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 -- 172.21.15.57:0/1211851330 shutdown_connections 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 --2- 172.21.15.57:0/1211851330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7110107270 0x7f711010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 --2- 172.21.15.57:0/1211851330 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7110102b50 0x7f7110106b90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 --2- 172.21.15.57:0/1211851330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71101021a0 0x7f7110102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 -- 172.21.15.57:0/1211851330 >> 172.21.15.57:0/1211851330 conn(0x7f71100793a0 msgr2=0x7f71100797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:14.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 -- 172.21.15.57:0/1211851330 shutdown_connections 2024-04-21T08:48:14.156 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.155+0000 7f7115481640 1 -- 172.21.15.57:0/1211851330 wait complete. 2024-04-21T08:48:14.156 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 Processor -- start 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 -- start start 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71101021a0 0x7f7110198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7110107270 0x7f7110198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 0x7f711019d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f71101121b0 con 0x7f711019cfd0 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7110112030 con 0x7f7110107270 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f7115481640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7110112330 con 0x7f71101021a0 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f710f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 0x7f711019d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f710f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 0x7f711019d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51358/0 (socket says 172.21.15.57:51358) 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f710f7fe640 1 -- 172.21.15.57:0/265209087 learned_addr learned my addr 172.21.15.57:0/265209087 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f710e7fc640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7110107270 0x7f7110198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.156+0000 7f710effd640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71101021a0 0x7f7110198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f710f7fe640 1 -- 172.21.15.57:0/265209087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71101021a0 msgr2=0x7f7110198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f710f7fe640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71101021a0 0x7f7110198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f710f7fe640 1 -- 172.21.15.57:0/265209087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7110107270 msgr2=0x7f7110198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f710f7fe640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7110107270 0x7f7110198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f710f7fe640 1 -- 172.21.15.57:0/265209087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711019dac0 con 0x7f711019cfd0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f710f7fe640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 0x7f711019d480 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f710000eaa0 tx=0x7f710000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710000cdb0 con 0x7f711019cfd0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7100005020 con 0x7f711019cfd0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711019ddb0 con 0x7f711019cfd0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000116d0 con 0x7f711019cfd0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.157+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71101a5520 con 0x7f711019cfd0 2024-04-21T08:48:14.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.158+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7110069230 con 0x7f711019cfd0 2024-04-21T08:48:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.159+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f71000051d0 con 0x7f711019cfd0 2024-04-21T08:48:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.160+0000 7f70f7fff640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70f80a3d20 0x7f70f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.160+0000 7f710effd640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70f80a3d20 0x7f70f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.160+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f71000d0c50 con 0x7f711019cfd0 2024-04-21T08:48:14.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.160+0000 7f710effd640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70f80a3d20 0x7f70f80a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7108005fd0 tx=0x7f710800b390 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:14.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.165+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f71000d6020 con 0x7f711019cfd0 2024-04-21T08:48:14.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.390+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7110102b50 con 0x7f711019cfd0 2024-04-21T08:48:14.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.391+0000 7f70f7fff640 1 -- 172.21.15.57:0/265209087 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f7100016030 con 0x7f711019cfd0 2024-04-21T08:48:14.390 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:48:14.391 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:48:14.391 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70f80a3d20 msgr2=0x7f70f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70f80a3d20 0x7f70f80a61e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7108005fd0 tx=0x7f710800b390 comp rx=0 tx=0).stop 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 msgr2=0x7f711019d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 0x7f711019d480 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f710000eaa0 tx=0x7f710000ef60 comp rx=0 tx=0).stop 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 shutdown_connections 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70f80a3d20 0x7f70f80a61e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f711019cfd0 0x7f711019d480 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7110107270 0x7f7110198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 --2- 172.21.15.57:0/265209087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71101021a0 0x7f7110198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 >> 172.21.15.57:0/265209087 conn(0x7f71100793a0 msgr2=0x7f71100fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 shutdown_connections 2024-04-21T08:48:14.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.393+0000 7f7115481640 1 -- 172.21.15.57:0/265209087 wait complete. 2024-04-21T08:48:14.403 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --set-id bad5ac53 --release --await' 2024-04-21T08:48:14.404 DEBUG:teuthology.orchestra.run.smithi057:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --set-id bad5ac53 --release --await 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 -- 172.21.15.57:0/2709970281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c106160 msgr2=0x7f220c10dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 --2- 172.21.15.57:0/2709970281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c106160 0x7f220c10dd30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2204009ca0 tx=0x7f220402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 -- 172.21.15.57:0/2709970281 shutdown_connections 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 --2- 172.21.15.57:0/2709970281 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c106160 0x7f220c10dd30 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 --2- 172.21.15.57:0/2709970281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f220c101a40 0x7f220c105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 --2- 172.21.15.57:0/2709970281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f220c101090 0x7f220c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 -- 172.21.15.57:0/2709970281 >> 172.21.15.57:0/2709970281 conn(0x7f220c077f50 msgr2=0x7f220c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 -- 172.21.15.57:0/2709970281 shutdown_connections 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.517+0000 7f2211115640 1 -- 172.21.15.57:0/2709970281 wait complete. 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 Processor -- start 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 -- start start 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f220c101090 0x7f220c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f220c106160 0x7f220c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c198680 0x7f220c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f220c111140 con 0x7f220c198680 2024-04-21T08:48:14.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f220c110fc0 con 0x7f220c106160 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.518+0000 7f2211115640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1112c0 con 0x7f220c101090 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c198680 0x7f220c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c198680 0x7f220c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51368/0 (socket says 172.21.15.57:51368) 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 -- 172.21.15.57:0/1723912019 learned_addr learned my addr 172.21.15.57:0/1723912019 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220ad9d640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f220c101090 0x7f220c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 -- 172.21.15.57:0/1723912019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f220c101090 msgr2=0x7f220c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220a59c640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f220c106160 0x7f220c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f220c101090 0x7f220c19e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 -- 172.21.15.57:0/1723912019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f220c106160 msgr2=0x7f220c19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f220c106160 0x7f220c19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 -- 172.21.15.57:0/1723912019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220c199260 con 0x7f220c198680 2024-04-21T08:48:14.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220ad9d640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f220c101090 0x7f220c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220a59c640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f220c106160 0x7f220c19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f220b59e640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f220c198680 0x7f220c198b30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f22040098d0 tx=0x7f2204035d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f21f3fff640 1 -- 172.21.15.57:0/1723912019 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204039040 con 0x7f220c198680 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f21f3fff640 1 -- 172.21.15.57:0/1723912019 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2204044070 con 0x7f220c198680 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f2211115640 1 -- 172.21.15.57:0/1723912019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f220c1a5340 con 0x7f220c198680 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f21f3fff640 1 -- 172.21.15.57:0/1723912019 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22040042a0 con 0x7f220c198680 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.519+0000 7f2211115640 1 -- 172.21.15.57:0/1723912019 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f220c1a56f0 con 0x7f220c198680 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.522+0000 7f2213de8740 1 Processor -- start 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.522+0000 7f21f3fff640 1 -- 172.21.15.57:0/1723912019 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2204004440 con 0x7f220c198680 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.522+0000 7f2213de8740 1 -- start start 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f2213de8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc1f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f2213de8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc282d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f2213de8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc2ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f2213de8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56223db84500 con 0x56223dc1f310 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f2213de8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56223db2a6a0 con 0x56223dc1fc30 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f2213de8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56223db804c0 con 0x56223dc28810 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc1f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc1f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51376/0 (socket says 172.21.15.57:51376) 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 -- 172.21.15.57:0/1855531927 learned_addr learned my addr 172.21.15.57:0/1855531927 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f21f3fff640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f21f40a3d20 0x7f21f40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc2ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220ad9d640 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 -- 172.21.15.57:0/1855531927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 msgr2=0x56223dc2ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc2ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 -- 172.21.15.57:0/1855531927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 msgr2=0x56223dc282d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc282d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 -- 172.21.15.57:0/1855531927 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56223db495e0 con 0x56223dc1f310 2024-04-21T08:48:14.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220b59e640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f21f40a3d20 0x7f21f40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f21f3fff640 1 -- 172.21.15.57:0/1723912019 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f22040f3050 con 0x7f220c198680 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f220a59c640 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc1f6f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f220000f360 tx=0x7f220000b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0e300cf2b2f86d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.523+0000 7f21f17fa640 1 -- 172.21.15.57:0/1855531927 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200012860 con 0x56223dc1f310 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f21f17fa640 1 -- 172.21.15.57:0/1855531927 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2200011440 con 0x56223dc1f310 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f21f17fa640 1 -- 172.21.15.57:0/1855531927 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220000d630 con 0x56223dc1f310 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 -- 172.21.15.57:0/1855531927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 msgr2=0x56223dc1f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f220b59e640 1 --2- 172.21.15.57:0/1723912019 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f21f40a3d20 0x7f21f40a61e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f22040f0a40 tx=0x7f22040df040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc1f6f0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f220000f360 tx=0x7f220000b4e0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 -- 172.21.15.57:0/1855531927 shutdown_connections 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc2ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc282d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 --2- 172.21.15.57:0/1855531927 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc1f6f0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 -- 172.21.15.57:0/1855531927 >> 172.21.15.57:0/1855531927 conn(0x56223dc19a40 msgr2=0x56223dc19e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 -- 172.21.15.57:0/1855531927 shutdown_connections 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.524+0000 7f2213de8740 1 -- 172.21.15.57:0/1855531927 wait complete. 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 -- start start 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 Processor -- start 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1f310 0x56223dc25670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 0x56223dc25bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc260f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56223dc2b600 con 0x56223dc1fc30 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56223db3da60 con 0x56223dc28810 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56223dc2b780 con 0x56223dc1f310 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f2213de8740 10 client.0 _ll_register_callbacks cb 0x56223d896110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 0x56223dc25bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 0x56223dc25bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:51384/0 (socket says 172.21.15.57:51384) 2024-04-21T08:48:14.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 -- 172.21.15.57:0/2169761518 learned_addr learned my addr 172.21.15.57:0/2169761518 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220a59c640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc260f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220a59c640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1f310 0x56223dc25670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 -- 172.21.15.57:0/2169761518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1f310 msgr2=0x56223dc25670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1f310 0x56223dc25670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 -- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 msgr2=0x56223dc260f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc260f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 -- 172.21.15.57:0/2169761518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56223d896920 con 0x56223dc1fc30 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220a59c640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1f310 0x56223dc25670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f220ad9d640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 0x56223dc25bb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00bfb0 tx=0x7f21fc00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1decaebac6644b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.525+0000 7f21eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f21eeff5640 1 -- 172.21.15.57:0/2169761518 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21fc018020 con 0x56223dc1fc30 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f21eeff5640 1 -- 172.21.15.57:0/2169761518 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f21fc014870 con 0x56223dc1fc30 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2200011440 con 0x56223dc1fc30 2024-04-21T08:48:14.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f21eeff5640 1 -- 172.21.15.57:0/2169761518 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f21fc023050 con 0x56223dc1fc30 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 10 client.16311 fetch_fsmap learned FSMap version 21 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56223d962d30 con 0x56223dc1fc30 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f21eeff5640 1 -- 172.21.15.57:0/2169761518 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2200011440 con 0x56223dc1fc30 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f21eeff5640 1 -- 172.21.15.57:0/2169761518 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f21fc013a50 con 0x56223dc1fc30 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 10 client.16311 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 10 client.16311 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 20 client.16311 populate_metadata read hostname 'smithi057' 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dc38a40 0x56223dc3ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 4 client.16311 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.526+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56223db57a10 con 0x56223dc38a40 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.527+0000 7f220a59c640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dc38a40 0x56223dc3ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.527+0000 7f21eeff5640 10 client.16311 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:48:14.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:48:14.527+0000 7f220a59c640 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dc38a40 0x56223dc3ae80 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=458ac5bf1d4f3424 server_cookie=906677862ccdd3e in_seq=0 out_seq=0 2024-04-21T08:48:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:14 smithi129 ceph-mon[31272]: pgmap v87: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 759 KiB/s wr, 446 op/s 2024-04-21T08:48:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/265209087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:14 smithi057 ceph-mon[27968]: pgmap v87: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 759 KiB/s wr, 446 op/s 2024-04-21T08:48:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/265209087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:14 smithi136 ceph-mon[31791]: pgmap v87: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 759 KiB/s wr, 446 op/s 2024-04-21T08:48:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/265209087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:48:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:16 smithi129 ceph-mon[31272]: pgmap v88: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 449 KiB/s wr, 66 op/s 2024-04-21T08:48:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:16 smithi057 ceph-mon[27968]: pgmap v88: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 449 KiB/s wr, 66 op/s 2024-04-21T08:48:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:16 smithi136 ceph-mon[31791]: pgmap v88: 129 pgs: 129 active+clean; 257 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 449 KiB/s wr, 66 op/s 2024-04-21T08:48:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:18 smithi129 ceph-mon[31272]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 550 KiB/s wr, 68 op/s 2024-04-21T08:48:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:18 smithi057 ceph-mon[27968]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 550 KiB/s wr, 68 op/s 2024-04-21T08:48:18.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:18 smithi136 ceph-mon[31791]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 550 KiB/s wr, 68 op/s 2024-04-21T08:48:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:20 smithi129 ceph-mon[31272]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 547 KiB/s wr, 67 op/s 2024-04-21T08:48:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:20 smithi057 ceph-mon[27968]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 547 KiB/s wr, 67 op/s 2024-04-21T08:48:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:20 smithi136 ceph-mon[31791]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 547 KiB/s wr, 67 op/s 2024-04-21T08:48:22.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:22 smithi129 ceph-mon[31272]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 547 KiB/s wr, 67 op/s 2024-04-21T08:48:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:22 smithi057 ceph-mon[27968]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 547 KiB/s wr, 67 op/s 2024-04-21T08:48:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:22 smithi136 ceph-mon[31791]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 547 KiB/s wr, 67 op/s 2024-04-21T08:48:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:24 smithi129 ceph-mon[31272]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 415 KiB/s wr, 54 op/s 2024-04-21T08:48:24.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:24 smithi057 ceph-mon[27968]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 415 KiB/s wr, 54 op/s 2024-04-21T08:48:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:24 smithi136 ceph-mon[31791]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 415 KiB/s wr, 54 op/s 2024-04-21T08:48:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:26 smithi129 ceph-mon[31272]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 5 op/s 2024-04-21T08:48:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:26 smithi057 ceph-mon[27968]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 5 op/s 2024-04-21T08:48:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:26 smithi136 ceph-mon[31791]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 5 op/s 2024-04-21T08:48:27.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:27 smithi129 ceph-mon[31272]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 6 op/s 2024-04-21T08:48:27.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:27 smithi057 ceph-mon[27968]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 6 op/s 2024-04-21T08:48:27.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:27 smithi136 ceph-mon[31791]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 6 op/s 2024-04-21T08:48:30.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:30 smithi129 ceph-mon[31272]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-21T08:48:30.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:30 smithi057 ceph-mon[27968]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-21T08:48:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:30 smithi136 ceph-mon[31791]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 3 op/s 2024-04-21T08:48:32.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:32 smithi129 ceph-mon[31272]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 4 op/s 2024-04-21T08:48:32.338 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:32 smithi057 ceph-mon[27968]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 4 op/s 2024-04-21T08:48:32.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:32 smithi136 ceph-mon[31791]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 4 op/s 2024-04-21T08:48:33.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:33 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:33.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:33 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:33.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:33 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:33.324 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:33 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:48:33.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:33 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:33.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:33 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:33.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:33 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:33.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:33 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:48:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:33 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:48:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:33 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:48:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:33 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:48:33.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:33 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:48:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:34 smithi129 ceph-mon[31272]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 5 op/s 2024-04-21T08:48:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:34 smithi057 ceph-mon[27968]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 5 op/s 2024-04-21T08:48:34.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:34 smithi136 ceph-mon[31791]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 5 op/s 2024-04-21T08:48:36.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:36 smithi129 ceph-mon[31272]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T08:48:36.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:36 smithi057 ceph-mon[27968]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T08:48:36.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:36 smithi136 ceph-mon[31791]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T08:48:38.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:38 smithi129 ceph-mon[31272]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.9 KiB/s wr, 3 op/s 2024-04-21T08:48:38.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:38 smithi057 ceph-mon[27968]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.9 KiB/s wr, 3 op/s 2024-04-21T08:48:38.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:38 smithi136 ceph-mon[31791]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.9 KiB/s wr, 3 op/s 2024-04-21T08:48:40.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:40 smithi129 ceph-mon[31272]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T08:48:40.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:40 smithi057 ceph-mon[27968]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T08:48:40.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:40 smithi136 ceph-mon[31791]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-04-21T08:48:42.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:42 smithi129 ceph-mon[31272]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.0 KiB/s wr, 3 op/s 2024-04-21T08:48:42.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:42 smithi057 ceph-mon[27968]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.0 KiB/s wr, 3 op/s 2024-04-21T08:48:42.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:42 smithi136 ceph-mon[31791]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.0 KiB/s wr, 3 op/s 2024-04-21T08:48:44.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:44 smithi129 ceph-mon[31272]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.2 KiB/s wr, 3 op/s 2024-04-21T08:48:44.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:44 smithi057 ceph-mon[27968]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.2 KiB/s wr, 3 op/s 2024-04-21T08:48:44.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:44 smithi136 ceph-mon[31791]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.2 KiB/s wr, 3 op/s 2024-04-21T08:48:46.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:46 smithi129 ceph-mon[31272]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-04-21T08:48:46.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:46 smithi057 ceph-mon[27968]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-04-21T08:48:46.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:46 smithi136 ceph-mon[31791]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-04-21T08:48:48.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:48 smithi129 ceph-mon[31272]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:48:48.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:48 smithi057 ceph-mon[27968]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:48:48.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:48 smithi136 ceph-mon[31791]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:48:50.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:50 smithi129 ceph-mon[31272]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.3 KiB/s wr, 2 op/s 2024-04-21T08:48:50.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:50 smithi057 ceph-mon[27968]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.3 KiB/s wr, 2 op/s 2024-04-21T08:48:50.386 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:50 smithi136 ceph-mon[31791]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.3 KiB/s wr, 2 op/s 2024-04-21T08:48:52.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:52 smithi057 ceph-mon[27968]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:48:52.384 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:52 smithi136 ceph-mon[31791]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:48:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:52 smithi129 ceph-mon[31272]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-04-21T08:48:54.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:54 smithi057 ceph-mon[27968]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 3 op/s 2024-04-21T08:48:54.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:54 smithi136 ceph-mon[31791]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 3 op/s 2024-04-21T08:48:54.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:54 smithi129 ceph-mon[31272]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.7 KiB/s wr, 3 op/s 2024-04-21T08:48:56.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:56 smithi057 ceph-mon[27968]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2024-04-21T08:48:56.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:56 smithi136 ceph-mon[31791]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2024-04-21T08:48:56.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:56 smithi129 ceph-mon[31272]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2024-04-21T08:48:58.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:48:58 smithi136 ceph-mon[31791]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.6 KiB/s wr, 3 op/s 2024-04-21T08:48:58.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:48:58 smithi129 ceph-mon[31272]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.6 KiB/s wr, 3 op/s 2024-04-21T08:48:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:48:58 smithi057 ceph-mon[27968]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.6 KiB/s wr, 3 op/s 2024-04-21T08:49:00.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:00 smithi129 ceph-mon[31272]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:49:00.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:00 smithi136 ceph-mon[31791]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:49:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:00 smithi057 ceph-mon[27968]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-04-21T08:49:02.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:02 smithi136 ceph-mon[31791]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2024-04-21T08:49:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:02 smithi129 ceph-mon[31272]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2024-04-21T08:49:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:02 smithi057 ceph-mon[27968]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.6 KiB/s wr, 2 op/s 2024-04-21T08:49:04.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:04 smithi136 ceph-mon[31791]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-04-21T08:49:04.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:04 smithi129 ceph-mon[31272]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-04-21T08:49:04.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:04 smithi057 ceph-mon[27968]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-04-21T08:49:06.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:06 smithi136 ceph-mon[31791]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-21T08:49:06.391 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:06 smithi057 ceph-mon[27968]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-21T08:49:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:06 smithi129 ceph-mon[31272]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-21T08:49:08.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:08 smithi136 ceph-mon[31791]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:08.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:08 smithi129 ceph-mon[31272]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:08 smithi057 ceph-mon[27968]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:10.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:10 smithi136 ceph-mon[31791]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-21T08:49:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:10 smithi129 ceph-mon[31272]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-21T08:49:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:10 smithi057 ceph-mon[27968]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 16 op/s 2024-04-21T08:49:12.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:12 smithi136 ceph-mon[31791]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:12 smithi129 ceph-mon[31272]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:12 smithi057 ceph-mon[27968]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:14 smithi136 ceph-mon[31791]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:14 smithi129 ceph-mon[31272]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:14 smithi057 ceph-mon[27968]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f1feaff5640 1 -- 172.21.15.57:0/2875333136 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555b617c92f0 con 0x555b61870000 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f1feaff5640 10 client.16281 handle_command_reply: tid=0 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f20103c9740 1 client.16281 shutdown 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 msgr2=0x555b61872990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f20103c9740 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b61872990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f20000099a0 tx=0x7f2000005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 shutdown_connections 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f20103c9740 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x555b61870000 0x555b618d85b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.914+0000 7f20103c9740 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x555b618600e0 0x555b61872990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f20103c9740 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x555b618578d0 0x555b61875940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f20103c9740 1 --2- 172.21.15.57:0/2875333136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x555b61856b60 0x555b61875400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.915 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 >> 172.21.15.57:0/2875333136 conn(0x555b6181dc90 msgr2=0x555b61857ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f1feaff5640 0 client.16281 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 shutdown_connections 2024-04-21T08:49:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f20103c9740 1 -- 172.21.15.57:0/2875333136 wait complete. 2024-04-21T08:49:14.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f381d7fa640 1 -- 172.21.15.57:0/695800024 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9f694cc00 con 0x55d9f69f39c0 2024-04-21T08:49:14.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f381d7fa640 10 client.16293 handle_command_reply: tid=0 2024-04-21T08:49:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f20103c9740 20 client.16281 trim_cache size 0 max 16384 2024-04-21T08:49:14.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 client.16293 shutdown 2024-04-21T08:49:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:49:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:49:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:49:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 msgr2=0x55d9f69e09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69e09d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f38280a5440 tx=0x7f382809c840 comp rx=0 tx=0).stop 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 shutdown_connections 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f69f39c0 0x55d9f69f5e50 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 --2- 172.21.15.57:0/695800024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69e09d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 >> 172.21.15.57:0/695800024 conn(0x55d9f69d4cb0 msgr2=0x55d9f69d6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f21eeff5640 1 -- 172.21.15.57:0/2169761518 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56223db57a10 con 0x56223dc38a40 2024-04-21T08:49:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f21eeff5640 10 client.16311 handle_command_reply: tid=0 2024-04-21T08:49:14.919 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f2213de8740 1 client.16311 shutdown 2024-04-21T08:49:14.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.915+0000 7f381d7fa640 0 client.16293 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 shutdown_connections 2024-04-21T08:49:14.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f384278c740 1 -- 172.21.15.57:0/695800024 wait complete. 2024-04-21T08:49:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1ff00a3e90 msgr2=0x7f1ff00a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1ff00a3e90 0x7f1ff00a6350 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2000009b30 tx=0x7f2000005e30 comp rx=0 tx=0).stop 2024-04-21T08:49:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 msgr2=0x7f200819a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f200819a610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ff800ea60 tx=0x7f1ff800ee60 comp rx=0 tx=0).stop 2024-04-21T08:49:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 msgr2=0x56223dc25bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 0x56223dc25bb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00bfb0 tx=0x7f21fc00f670 comp rx=0 tx=0).stop 2024-04-21T08:49:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f384278c740 20 client.16293 trim_cache size 0 max 16384 2024-04-21T08:49:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 shutdown_connections 2024-04-21T08:49:14.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dc38a40 0x56223dc3ae80 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc260f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f21eeff5640 0 client.16311 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1fc30 0x56223dc25bb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.922 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 --2- 172.21.15.57:0/2169761518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1f310 0x56223dc25670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 shutdown_connections 2024-04-21T08:49:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f1ff00a3e90 0x7f1ff00a6350 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f200810c5a0 0x7f200819eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2008103af0 0x7f200819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 --2- 172.21.15.57:0/1638062824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20081031d0 0x7f200819a610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 >> 172.21.15.57:0/1638062824 conn(0x7f2008077fa0 msgr2=0x7f20081005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 >> 172.21.15.57:0/2169761518 conn(0x56223dc19a40 msgr2=0x56223dc1b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 shutdown_connections 2024-04-21T08:49:14.923 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 1 -- 172.21.15.57:0/2169761518 wait complete. 2024-04-21T08:49:14.924 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 shutdown_connections 2024-04-21T08:49:14.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f2213de8740 20 client.16311 trim_cache size 0 max 16384 2024-04-21T08:49:14.924 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:14.916+0000 7f200d7d5640 1 -- 172.21.15.57:0/1638062824 wait complete. 2024-04-21T08:49:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.937+0000 7f384278c740 1 Processor -- start 2024-04-21T08:49:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.937+0000 7f384278c740 1 -- start start 2024-04-21T08:49:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.937+0000 7f384278c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69ef700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f384278c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f384278c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f384278c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f38381af340 con 0x55d9f69da580 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f384278c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f38381b1ba0 con 0x55d9f69daea0 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f384278c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f38381af990 con 0x55d9f69e3a80 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69ef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d05c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69ef700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44868/0 (socket says 172.21.15.57:44868) 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 -- 172.21.15.57:0/3831540152 learned_addr learned my addr 172.21.15.57:0/3831540152 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383e05e640 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 -- 172.21.15.57:0/3831540152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 msgr2=0x55d9f69e0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 -- 172.21.15.57:0/3831540152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 msgr2=0x55d9f69e3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 -- 172.21.15.57:0/3831540152 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9f69dc5d0 con 0x55d9f69da580 2024-04-21T08:49:14.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383e05e640 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.938+0000 7f383d85d640 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69ef700 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f38280996d0 tx=0x7f38280b4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea71f4d4156d7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.939+0000 7f381f7fe640 1 -- 172.21.15.57:0/3831540152 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382809c660 con 0x55d9f69da580 2024-04-21T08:49:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.939+0000 7f381f7fe640 1 -- 172.21.15.57:0/3831540152 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382809cd00 con 0x55d9f69da580 2024-04-21T08:49:14.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.939+0000 7f381f7fe640 1 -- 172.21.15.57:0/3831540152 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38280a5bc0 con 0x55d9f69da580 2024-04-21T08:49:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 -- 172.21.15.57:0/3831540152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 msgr2=0x55d9f69ef700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69ef700 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f38280996d0 tx=0x7f38280b4000 comp rx=0 tx=0).stop 2024-04-21T08:49:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 -- 172.21.15.57:0/3831540152 shutdown_connections 2024-04-21T08:49:14.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69daea0 0x55d9f69e3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 --2- 172.21.15.57:0/3831540152 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69da580 0x55d9f69ef700 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 -- 172.21.15.57:0/3831540152 >> 172.21.15.57:0/3831540152 conn(0x55d9f69d4cb0 msgr2=0x55d9f69d6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 -- 172.21.15.57:0/3831540152 shutdown_connections 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 -- 172.21.15.57:0/3831540152 wait complete. 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 -- start start 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.940+0000 7f384278c740 1 Processor -- start 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 0x55d9f69ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69ec150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69dcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f38381b0060 con 0x55d9f69daea0 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f38381b14d0 con 0x55d9f69da580 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f38381b0730 con 0x55d9f69e3a80 2024-04-21T08:49:14.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f384278c740 10 client.0 _ll_register_callbacks cb 0x7f382800ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d05c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 0x55d9f69ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383e05e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69dcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 0x55d9f69ebc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:56012/0 (socket says 172.21.15.57:56012) 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d05c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69ec150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44876/0 (socket says 172.21.15.57:44876) 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 -- 172.21.15.57:0/2612502404 learned_addr learned my addr 172.21.15.57:0/2612502404 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 -- 172.21.15.57:0/2612502404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 msgr2=0x55d9f69dcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69dcc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 -- 172.21.15.57:0/2612502404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 msgr2=0x55d9f69ec150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69ec150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d85d640 1 -- 172.21.15.57:0/2612502404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9f69dd180 con 0x55d9f69da580 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.941+0000 7f383d05c640 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:14.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f383d85d640 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 0x55d9f69ebc10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f38280a5440 tx=0x7f38280094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a467ebf233c9bfc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f381d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f381d7fa640 1 -- 172.21.15.57:0/2612502404 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3828086020 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f381d7fa640 1 -- 172.21.15.57:0/2612502404 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f38280bd0d0 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f382809cd00 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f381d7fa640 1 -- 172.21.15.57:0/2612502404 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f382809c660 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f384278c740 10 client.25811 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d9f69dd180 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f381d7fa640 1 -- 172.21.15.57:0/2612502404 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f382809cd00 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.942+0000 7f381d7fa640 1 -- 172.21.15.57:0/2612502404 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3828058020 con 0x55d9f69da580 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f384278c740 10 client.25811 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f384278c740 10 client.25811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f384278c740 20 client.25811 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:14.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f384278c740 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f6a59a00 0x55d9f6a5be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f384278c740 4 client.25811 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9f6946a70 con 0x55d9f6a59a00 2024-04-21T08:49:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f383d05c640 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f6a59a00 0x55d9f6a5be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f383d05c640 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f6a59a00 0x55d9f6a5be20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=195ca722be4f94c9 server_cookie=1243c282f41f9741 in_seq=0 out_seq=0 2024-04-21T08:49:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.943+0000 7f381d7fa640 10 client.25811 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.944+0000 7f381d7fa640 1 -- 172.21.15.57:0/2612502404 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55d9f6946a70 con 0x55d9f6a59a00 2024-04-21T08:49:14.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.944+0000 7f381d7fa640 10 client.25811 handle_command_reply: tid=0 2024-04-21T08:49:14.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.944+0000 7f384278c740 1 client.25811 shutdown 2024-04-21T08:49:14.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 Processor -- start 2024-04-21T08:49:14.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 -- start start 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 msgr2=0x55d9f69ebc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 0x55d9f69ebc10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f38280a5440 tx=0x7f38280094a0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 shutdown_connections 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d9f6a59a00 0x55d9f6a5be20 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d9f69e3a80 0x55d9f69dcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d9f69daea0 0x55d9f69ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f381d7fa640 0 client.25811 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 --2- 172.21.15.57:0/2612502404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d9f69da580 0x55d9f69ebc10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 >> 172.21.15.57:0/2612502404 conn(0x55d9f69d4cb0 msgr2=0x55d9f69e0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 shutdown_connections 2024-04-21T08:49:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc27da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 1 -- 172.21.15.57:0/2612502404 wait complete. 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc40750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1af480 con 0x56223dc1f310 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b1ce0 con 0x56223dc1fc30 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f2213de8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1afad0 con 0x56223dc28810 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc27da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc27da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44888/0 (socket says 172.21.15.57:44888) 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 -- 172.21.15.57:0/3496014089 learned_addr learned my addr 172.21.15.57:0/3496014089 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:14.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f384278c740 20 client.25811 trim_cache size 0 max 16384 2024-04-21T08:49:14.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc40750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.952 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:49:14.952 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (6100 inodes in the stack)", 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: "b926b3c8-226e-447f-98b1-7b78a97b9920": { 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "b926b3c8-226e-447f-98b1-7b78a97b9920", 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.953 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 -- 172.21.15.57:0/3496014089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 msgr2=0x56223dc40750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc40750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.945+0000 7f220a59c640 1 -- 172.21.15.57:0/3496014089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 msgr2=0x56223dc344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f220a59c640 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f220a59c640 1 -- 172.21.15.57:0/3496014089 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56223dc13c50 con 0x56223dc1f310 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f220ad9d640 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:14.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f220a59c640 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc27da0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2200011ea0 tx=0x7f220000b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90ed67df6128a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f38200a3d20 msgr2=0x7f38200a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f38200a3d20 0x7f38200a61e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f383806a380 tx=0x7f383000b2e0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 msgr2=0x7f38381a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 0x7f38381a0610 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f382800eab0 tx=0x7f382800ef70 comp rx=0 tx=0).stop 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 shutdown_connections 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f38200a3d20 0x7f38200a61e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f383810ec50 0x7f383819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383806aed0 0x7f38381a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 --2- 172.21.15.57:0/2619130349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f383806a5b0 0x7f38381a0610 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f21f0ff9640 1 -- 172.21.15.57:0/3496014089 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220002a720 con 0x56223dc1f310 2024-04-21T08:49:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 >> 172.21.15.57:0/2619130349 conn(0x7f3838077c40 msgr2=0x7f38381101f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f21f0ff9640 1 -- 172.21.15.57:0/3496014089 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f220002a8c0 con 0x56223dc1f310 2024-04-21T08:49:14.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f21f0ff9640 1 -- 172.21.15.57:0/3496014089 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22000128c0 con 0x56223dc1f310 2024-04-21T08:49:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 shutdown_connections 2024-04-21T08:49:14.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.946+0000 7f383fac1640 1 -- 172.21.15.57:0/2619130349 wait complete. 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 -- 172.21.15.57:0/3496014089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 msgr2=0x56223dc27da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc27da0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f2200011ea0 tx=0x7f220000b4e0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 -- 172.21.15.57:0/3496014089 shutdown_connections 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc28810 0x56223dc40750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc1fc30 0x56223dc344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 --2- 172.21.15.57:0/3496014089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc27da0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 -- 172.21.15.57:0/3496014089 >> 172.21.15.57:0/3496014089 conn(0x56223dc19a40 msgr2=0x56223dc25ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 -- 172.21.15.57:0/3496014089 shutdown_connections 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 -- 172.21.15.57:0/3496014089 wait complete. 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.947+0000 7f2213de8740 1 -- start start 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 Processor -- start 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc30c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1fc30 0x56223dc2b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc2bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b01a0 con 0x56223dc1f310 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b1610 con 0x56223dc28810 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f220c1b0870 con 0x56223dc1fc30 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f2213de8740 10 client.0 _ll_register_callbacks cb 0x7f21fc00b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f220a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc30c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f220a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc30c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:44904/0 (socket says 172.21.15.57:44904) 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f220a59c640 1 -- 172.21.15.57:0/3973713637 learned_addr learned my addr 172.21.15.57:0/3973713637 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:14.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f220ad9d640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1fc30 0x56223dc2b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.948+0000 7f220a59c640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc2bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220a59c640 1 -- 172.21.15.57:0/3973713637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1fc30 msgr2=0x56223dc2b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220a59c640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1fc30 0x56223dc2b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220a59c640 1 -- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 msgr2=0x56223dc2bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220a59c640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc2bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220a59c640 1 -- 172.21.15.57:0/3973713637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56223dc2c4c0 con 0x56223dc1f310 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220ad9d640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1fc30 0x56223dc2b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f220a59c640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc30c00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2200005b80 tx=0x7f220000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0140f2f099421fa server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f21eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f21eeff5640 1 -- 172.21.15.57:0/3973713637 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200011440 con 0x56223dc1f310 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f21eeff5640 1 -- 172.21.15.57:0/3973713637 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f22000115e0 con 0x56223dc1f310 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f220002a8c0 con 0x56223dc1f310 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.949+0000 7f21eeff5640 1 -- 172.21.15.57:0/3973713637 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f22000117e0 con 0x56223dc1f310 2024-04-21T08:49:14.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 10 client.16323 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56223dc9fdb0 con 0x56223dc1f310 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f21eeff5640 1 -- 172.21.15.57:0/3973713637 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f220002a8c0 con 0x56223dc1f310 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f21eeff5640 1 -- 172.21.15.57:0/3973713637 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2200005140 con 0x56223dc1f310 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 10 client.16323 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 10 client.16323 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 20 client.16323 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dca0230 0x56223dca2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 4 client.16323 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bad5ac53", "release": true, "await": true}] 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bad5ac53", "release": true, "await": true}) -- 0x56223db91f20 con 0x56223dca0230 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.950+0000 7f220ad9d640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dca0230 0x56223dca2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.951+0000 7f21eeff5640 10 client.16323 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.951+0000 7f220ad9d640 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dca0230 0x56223dca2620 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc66b705255bd899 server_cookie=abedc2dedd752f02 in_seq=0 out_seq=0 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f21eeff5640 1 -- 172.21.15.57:0/3973713637 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: -1 ) ==== 8+0+588 (crc 0 0 0) 0x56223db91f20 con 0x56223dca0230 2024-04-21T08:49:14.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f21eeff5640 10 client.16323 handle_command_reply: tid=0 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 client.16323 shutdown 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 msgr2=0x56223dc30c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc30c00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2200005b80 tx=0x7f220000bd30 comp rx=0 tx=0).stop 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 shutdown_connections 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56223dca0230 0x56223dca2620 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56223dc28810 0x56223dc2bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56223dc1fc30 0x56223dc2b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f21eeff5640 0 client.16323 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 --2- 172.21.15.57:0/3973713637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56223dc1f310 0x56223dc30c00 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 >> 172.21.15.57:0/3973713637 conn(0x56223dc19a40 msgr2=0x56223dc1ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 shutdown_connections 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 1 -- 172.21.15.57:0/3973713637 wait complete. 2024-04-21T08:49:14.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:14.952+0000 7f2213de8740 20 client.16323 trim_cache size 0 max 16384 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "epoch": 21, 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "leader": 24503, 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "set_version": 3, 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "sets": { 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "bad5ac53": { 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "version": 3, 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "age_ref": 0.9, 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "state": { 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "EXPIRED", 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 0.0 2024-04-21T08:49:14.966 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "timeout": 30.0, 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "expiration": 74.0, 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "members": { 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "file:/": { 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "excluded": false, 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "state": { 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "QUIESCED", 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: "age": 74.0 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:14.967 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:49:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-21T08:49:14.968 INFO:teuthology.orchestra.run.smithi057.stderr:Error EPERM: 2024-04-21T08:49:14.969 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't release root with rc: -124 (Unknown), stdout: { "epoch": 21, "leader": 24503, "set_version": 3, "sets": { "bad5ac53": { "version": 3, "age_ref": 0.9, "state": { "name": "EXPIRED", "age": 0.0 }, "timeout": 30.0, "expiration": 74.0, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCED", "age": 74.0 } } } } } } 2024-04-21T08:49:14.969 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 240, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 216, in do_quiesce raise RuntimeError(f"Error releasing root: {rcinfo}") RuntimeError: Error releasing root: -124 (Unknown) 2024-04-21T08:49:14.970 INFO:tasks.cephfs.filesystem:scrub status for tag:b926b3c8-226e-447f-98b1-7b78a97b9920 - {'path': '/', 'tag': 'b926b3c8-226e-447f-98b1-7b78a97b9920', 'options': 'recursive,force'} 2024-04-21T08:49:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:16 smithi136 ceph-mon[31791]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:49:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:16 smithi129 ceph-mon[31272]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:49:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:16 smithi057 ceph-mon[27968]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-04-21T08:49:17.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:17 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:17.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:17 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:49:17.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:17 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:17.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:17 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:49:17.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:17 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:17.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:17 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:49:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:18 smithi129 ceph-mon[31272]: pgmap v119: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 20 KiB/s wr, 634 op/s 2024-04-21T08:49:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:18 smithi057 ceph-mon[27968]: pgmap v119: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 20 KiB/s wr, 634 op/s 2024-04-21T08:49:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:18 smithi136 ceph-mon[31791]: pgmap v119: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 20 KiB/s wr, 634 op/s 2024-04-21T08:49:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:20 smithi129 ceph-mon[31272]: pgmap v120: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 20 KiB/s wr, 633 op/s 2024-04-21T08:49:20.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:20 smithi057 ceph-mon[27968]: pgmap v120: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 20 KiB/s wr, 633 op/s 2024-04-21T08:49:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:20 smithi136 ceph-mon[31791]: pgmap v120: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 20 KiB/s wr, 633 op/s 2024-04-21T08:49:22.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:22 smithi129 ceph-mon[31272]: pgmap v121: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 23 KiB/s wr, 635 op/s 2024-04-21T08:49:22.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:22 smithi057 ceph-mon[27968]: pgmap v121: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 23 KiB/s wr, 635 op/s 2024-04-21T08:49:22.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:22 smithi136 ceph-mon[31791]: pgmap v121: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 23 KiB/s wr, 635 op/s 2024-04-21T08:49:24.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:24 smithi129 ceph-mon[31272]: pgmap v122: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 93 KiB/s wr, 636 op/s 2024-04-21T08:49:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:24 smithi057 ceph-mon[27968]: pgmap v122: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 93 KiB/s wr, 636 op/s 2024-04-21T08:49:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:24 smithi136 ceph-mon[31791]: pgmap v122: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 93 KiB/s wr, 636 op/s 2024-04-21T08:49:26.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:26 smithi129 ceph-mon[31272]: pgmap v123: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 93 KiB/s wr, 636 op/s 2024-04-21T08:49:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:26 smithi057 ceph-mon[27968]: pgmap v123: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 93 KiB/s wr, 636 op/s 2024-04-21T08:49:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:26 smithi136 ceph-mon[31791]: pgmap v123: 129 pgs: 129 active+clean; 248 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 93 KiB/s wr, 636 op/s 2024-04-21T08:49:28.025 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 -- 172.21.15.57:0/211046050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 msgr2=0x7f64301014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:28.025 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 --2- 172.21.15.57:0/211046050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f64301014d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6420009900 tx=0x7f642002bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:28.025 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 -- 172.21.15.57:0/211046050 shutdown_connections 2024-04-21T08:49:28.025 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 --2- 172.21.15.57:0/211046050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6430106170 0x7f643010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 --2- 172.21.15.57:0/211046050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430101aa0 0x7f6430105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 --2- 172.21.15.57:0/211046050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f64301014d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 -- 172.21.15.57:0/211046050 >> 172.21.15.57:0/211046050 conn(0x7f6430077fd0 msgr2=0x7f64300783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 -- 172.21.15.57:0/211046050 shutdown_connections 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 -- 172.21.15.57:0/211046050 wait complete. 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.023+0000 7f6435fa4640 1 Processor -- start 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 -- start start 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f6430102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430106170 0x7f6430103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64301035b0 0x7f64301a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f64301110e0 con 0x7f64301010f0 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6430110f60 con 0x7f6430106170 2024-04-21T08:49:28.026 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f6435fa4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6430111260 con 0x7f64301035b0 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f6430102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f6430102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41596/0 (socket says 172.21.15.57:41596) 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 -- 172.21.15.57:0/3930351381 learned_addr learned my addr 172.21.15.57:0/3930351381 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642effd640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430106170 0x7f6430103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642ffff640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64301035b0 0x7f64301a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 -- 172.21.15.57:0/3930351381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64301035b0 msgr2=0x7f64301a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64301035b0 0x7f64301a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 -- 172.21.15.57:0/3930351381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430106170 msgr2=0x7f6430103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430106170 0x7f6430103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642f7fe640 1 -- 172.21.15.57:0/3930351381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64301a6f70 con 0x7f64301010f0 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642ffff640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64301035b0 0x7f64301a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.024+0000 7f642effd640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430106170 0x7f6430103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.025+0000 7f642f7fe640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f6430102b30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6420005940 tx=0x7f64200045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.025+0000 7f642cff9640 1 -- 172.21.15.57:0/3930351381 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420043020 con 0x7f64301010f0 2024-04-21T08:49:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.025+0000 7f642cff9640 1 -- 172.21.15.57:0/3930351381 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f642002caf0 con 0x7f64301010f0 2024-04-21T08:49:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.025+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64301a7200 con 0x7f64301010f0 2024-04-21T08:49:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.025+0000 7f642cff9640 1 -- 172.21.15.57:0/3930351381 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420003af0 con 0x7f64301010f0 2024-04-21T08:49:28.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.025+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64301a7750 con 0x7f64301010f0 2024-04-21T08:49:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.027+0000 7f6438b96740 1 Processor -- start 2024-04-21T08:49:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642cff9640 1 -- 172.21.15.57:0/3930351381 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6420046020 con 0x7f64301010f0 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 -- start start 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63bd0e0 0x562cc63bf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562cc63188d0 con 0x562cc63b48d0 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562cc62b5e80 con 0x562cc63b3b60 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f6438b96740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562cc6314890 con 0x562cc63bd0e0 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41598/0 (socket says 172.21.15.57:41598) 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 -- 172.21.15.57:0/2703023137 learned_addr learned my addr 172.21.15.57:0/2703023137 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642effd640 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63bd0e0 0x562cc63bf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 -- 172.21.15.57:0/2703023137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63bd0e0 msgr2=0x562cc63bf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63bd0e0 0x562cc63bf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 -- 172.21.15.57:0/2703023137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 msgr2=0x562cc63b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642cff9640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f64180a3dc0 0x7f64180a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642ffff640 1 -- 172.21.15.57:0/2703023137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cc62d7320 con 0x562cc63b48d0 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.028+0000 7f642f7fe640 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.029+0000 7f642effd640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f64180a3dc0 0x7f64180a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.029+0000 7f642ffff640 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b3620 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6428009f30 tx=0x7f6428010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60b404e412c3ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.029+0000 7f642cff9640 1 -- 172.21.15.57:0/3930351381 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f64200f2c00 con 0x7f64301010f0 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.029+0000 7f6415ffb640 1 -- 172.21.15.57:0/2703023137 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428016070 con 0x562cc63b48d0 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.029+0000 7f6415ffb640 1 -- 172.21.15.57:0/2703023137 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f642801bb10 con 0x562cc63b48d0 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.029+0000 7f6415ffb640 1 -- 172.21.15.57:0/2703023137 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428012c10 con 0x562cc63b48d0 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 -- 172.21.15.57:0/2703023137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 msgr2=0x562cc63b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b3620 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f6428009f30 tx=0x7f6428010800 comp rx=0 tx=0).stop 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f642effd640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f64180a3dc0 0x7f64180a6280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x562cc63af9e0 tx=0x7f6424005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 -- 172.21.15.57:0/2703023137 shutdown_connections 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63bd0e0 0x562cc63bf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 --2- 172.21.15.57:0/2703023137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b3620 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 -- 172.21.15.57:0/2703023137 >> 172.21.15.57:0/2703023137 conn(0x562cc637ac90 msgr2=0x562cc63ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:28.036 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 -- 172.21.15.57:0/2703023137 shutdown_connections 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 -- 172.21.15.57:0/2703023137 wait complete. 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 -- start start 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.030+0000 7f6438b96740 1 Processor -- start 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63bca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b48d0 0x562cc63b9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63ba330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562cc63bfe10 con 0x562cc63bd0e0 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562cc62a0eb0 con 0x562cc63b48d0 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562cc63bff90 con 0x562cc63b3b60 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f6438b96740 10 client.0 _ll_register_callbacks cb 0x562cc62cdb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41604/0 (socket says 172.21.15.57:41604) 2024-04-21T08:49:28.037 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 -- 172.21.15.57:0/2940241946 learned_addr learned my addr 172.21.15.57:0/2940241946 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b48d0 0x562cc63b9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642f7fe640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63bca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 -- 172.21.15.57:0/2940241946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 msgr2=0x562cc63bca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63bca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 -- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b48d0 msgr2=0x562cc63b9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b48d0 0x562cc63b9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 -- 172.21.15.57:0/2940241946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cc602b210 con 0x562cc63bd0e0 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642effd640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b48d0 0x562cc63b9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f642ffff640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63ba330 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6428009f00 tx=0x7f6428009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584973a61b6ac124 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.031+0000 7f640effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f640effd640 1 -- 172.21.15.57:0/2940241946 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428016040 con 0x562cc63bd0e0 2024-04-21T08:49:28.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f640effd640 1 -- 172.21.15.57:0/2940241946 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6428004c40 con 0x562cc63bd0e0 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f642801bb10 con 0x562cc63bd0e0 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f640effd640 1 -- 172.21.15.57:0/2940241946 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6428031050 con 0x562cc63bd0e0 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 10 client.16335 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562cc60f7620 con 0x562cc63bd0e0 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f640effd640 1 -- 172.21.15.57:0/2940241946 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f642801bb10 con 0x562cc63bd0e0 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f640effd640 1 -- 172.21.15.57:0/2940241946 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f642802c030 con 0x562cc63bd0e0 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 10 client.16335 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 10 client.16335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 20 client.16335 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc63cd020 0x562cc63cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 4 client.16335 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.032+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562cc632c880 con 0x562cc63cd020 2024-04-21T08:49:28.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.033+0000 7f642f7fe640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc63cd020 0x562cc63cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.033+0000 7f642f7fe640 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc63cd020 0x562cc63cf410 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=436642c04fb8d73b server_cookie=eda8bea68c5a158b in_seq=0 out_seq=0 2024-04-21T08:49:28.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.033+0000 7f640effd640 10 client.16335 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f640effd640 1 -- 172.21.15.57:0/2940241946 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562cc632c880 con 0x562cc63cd020 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f640effd640 10 client.16335 handle_command_reply: tid=0 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 client.16335 shutdown 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 msgr2=0x562cc63ba330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63ba330 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f6428009f00 tx=0x7f6428009c70 comp rx=0 tx=0).stop 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 shutdown_connections 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc63cd020 0x562cc63cf410 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63ba330 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b48d0 0x562cc63b9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 --2- 172.21.15.57:0/2940241946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63bca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f640effd640 0 client.16335 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:28.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 >> 172.21.15.57:0/2940241946 conn(0x562cc637ac90 msgr2=0x562cc63b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 shutdown_connections 2024-04-21T08:49:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 1 -- 172.21.15.57:0/2940241946 wait complete. 2024-04-21T08:49:28.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.035+0000 7f6438b96740 20 client.16335 trim_cache size 0 max 16384 2024-04-21T08:49:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.051+0000 7f6438b96740 1 Processor -- start 2024-04-21T08:49:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.051+0000 7f6438b96740 1 -- start start 2024-04-21T08:49:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f6438b96740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f6438b96740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f6438b96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63bd0e0 0x562cc63bc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f6438b96740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f64301af3c0 con 0x562cc63b48d0 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f6438b96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64301b1c20 con 0x562cc63bd0e0 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f6438b96740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f64301afa10 con 0x562cc63b3b60 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41614/0 (socket says 172.21.15.57:41614) 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63bd0e0 0x562cc63bc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642effd640 1 -- 172.21.15.57:0/869837308 learned_addr learned my addr 172.21.15.57:0/869837308 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642f7fe640 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 -- 172.21.15.57:0/869837308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 msgr2=0x562cc63c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 -- 172.21.15.57:0/869837308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 msgr2=0x562cc63b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 -- 172.21.15.57:0/869837308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cc63b4220 con 0x562cc63bd0e0 2024-04-21T08:49:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642effd640 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.052+0000 7f642ffff640 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63bd0e0 0x562cc63bc500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f642801b990 tx=0x7f64280096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf28fcabdb2872b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f64157fa640 1 -- 172.21.15.57:0/869837308 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64280052c0 con 0x562cc63bd0e0 2024-04-21T08:49:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f64157fa640 1 -- 172.21.15.57:0/869837308 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f642801b460 con 0x562cc63bd0e0 2024-04-21T08:49:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f64157fa640 1 -- 172.21.15.57:0/869837308 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6428004a00 con 0x562cc63bd0e0 2024-04-21T08:49:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 -- 172.21.15.57:0/869837308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63bd0e0 msgr2=0x562cc63bc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63bd0e0 0x562cc63bc500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f642801b990 tx=0x7f64280096e0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 -- 172.21.15.57:0/869837308 shutdown_connections 2024-04-21T08:49:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63bd0e0 0x562cc63bc500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63b48d0 0x562cc63b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 --2- 172.21.15.57:0/869837308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b3b60 0x562cc63c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 -- 172.21.15.57:0/869837308 >> 172.21.15.57:0/869837308 conn(0x562cc637ac90 msgr2=0x562cc63afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 -- 172.21.15.57:0/869837308 shutdown_connections 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.053+0000 7f6438b96740 1 -- 172.21.15.57:0/869837308 wait complete. 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 -- start start 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 Processor -- start 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b48d0 0x562cc63c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63b6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f64301b00e0 con 0x562cc63bd0e0 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64301b1550 con 0x562cc63b3b60 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f64301b07b0 con 0x562cc63b48d0 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f6438b96740 10 client.0 _ll_register_callbacks cb 0x7f64200f6620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f642ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f642f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f642ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63b6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41624/0 (socket says 172.21.15.57:41624) 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f642ffff640 1 -- 172.21.15.57:0/3844265985 learned_addr learned my addr 172.21.15.57:0/3844265985 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.054+0000 7f642f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44610/0 (socket says 172.21.15.57:44610) 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642effd640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b48d0 0x562cc63c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642ffff640 1 -- 172.21.15.57:0/3844265985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b48d0 msgr2=0x562cc63c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642ffff640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b48d0 0x562cc63c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642ffff640 1 -- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 msgr2=0x562cc63c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642ffff640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642ffff640 1 -- 172.21.15.57:0/3844265985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cc63b6580 con 0x562cc63bd0e0 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642effd640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b48d0 0x562cc63c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642f7fe640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f642ffff640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63b6040 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6428009f30 tx=0x7f6428005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888429e197a4d742 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f640effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:28.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f640effd640 1 -- 172.21.15.57:0/3844265985 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64280050c0 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f640effd640 1 -- 172.21.15.57:0/3844265985 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6428005260 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f642801b460 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f640effd640 1 -- 172.21.15.57:0/3844265985 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f642804c050 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f6438b96740 10 client.16341 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.055+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562cc6432a40 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f640effd640 1 -- 172.21.15.57:0/3844265985 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f642801b460 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f640effd640 1 -- 172.21.15.57:0/3844265985 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f642802c040 con 0x562cc63bd0e0 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f6438b96740 10 client.16341 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f6438b96740 10 client.16341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f6438b96740 20 client.16341 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f6438b96740 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc6433100 0x562cc6435520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f6438b96740 4 client.16341 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562cc62b83d0 con 0x562cc6433100 2024-04-21T08:49:28.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f642f7fe640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc6433100 0x562cc6435520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f642f7fe640 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc6433100 0x562cc6435520 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21f3d8a51859b4d2 server_cookie=bd87bbf03e6b349f in_seq=0 out_seq=0 2024-04-21T08:49:28.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:28.056+0000 7f640effd640 10 client.16341 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:28 smithi129 ceph-mon[31272]: pgmap v124: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 898 op/s 2024-04-21T08:49:28.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:28 smithi057 ceph-mon[27968]: pgmap v124: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 898 op/s 2024-04-21T08:49:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:28 smithi136 ceph-mon[31791]: pgmap v124: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 898 op/s 2024-04-21T08:49:30.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:30 smithi136 ceph-mon[31791]: pgmap v125: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 2.3 MiB/s wr, 265 op/s 2024-04-21T08:49:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:30 smithi129 ceph-mon[31272]: pgmap v125: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 2.3 MiB/s wr, 265 op/s 2024-04-21T08:49:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:30 smithi057 ceph-mon[27968]: pgmap v125: 129 pgs: 129 active+clean; 272 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 2.3 MiB/s wr, 265 op/s 2024-04-21T08:49:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.885+0000 7f640effd640 1 -- 172.21.15.57:0/3844265985 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562cc62b83d0 con 0x562cc6433100 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.885+0000 7f640effd640 10 client.16341 handle_command_reply: tid=0 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.885+0000 7f6438b96740 1 client.16341 shutdown 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 msgr2=0x562cc63b6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63b6040 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6428009f30 tx=0x7f6428005f80 comp rx=0 tx=0).stop 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 shutdown_connections 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562cc6433100 0x562cc6435520 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562cc63bd0e0 0x562cc63b6040 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f640effd640 0 client.16341 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562cc63b48d0 0x562cc63c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 --2- 172.21.15.57:0/3844265985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562cc63b3b60 0x562cc63c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 >> 172.21.15.57:0/3844265985 conn(0x562cc637ac90 msgr2=0x562cc63afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 shutdown_connections 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 1 -- 172.21.15.57:0/3844265985 wait complete. 2024-04-21T08:49:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.886+0000 7f6438b96740 20 client.16341 trim_cache size 0 max 16384 2024-04-21T08:49:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:49:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:49:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:49:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f64180a3dc0 msgr2=0x7f64180a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f64180a3dc0 0x7f64180a6280 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x562cc63af9e0 tx=0x7f6424005f90 comp rx=0 tx=0).stop 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 msgr2=0x7f6430102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f6430102b30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6420005940 tx=0x7f64200045e0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 shutdown_connections 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f64180a3dc0 0x7f64180a6280 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64301035b0 0x7f64301a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6430106170 0x7f6430103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 --2- 172.21.15.57:0/3930351381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f64301010f0 0x7f6430102b30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 >> 172.21.15.57:0/3930351381 conn(0x7f6430077fd0 msgr2=0x7f6430069980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 shutdown_connections 2024-04-21T08:49:30.890 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:30.887+0000 7f6435fa4640 1 -- 172.21.15.57:0/3930351381 wait complete. 2024-04-21T08:49:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:32 smithi129 ceph-mon[31272]: pgmap v126: 129 pgs: 129 active+clean; 268 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 551 op/s 2024-04-21T08:49:32.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:32 smithi057 ceph-mon[27968]: pgmap v126: 129 pgs: 129 active+clean; 268 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 551 op/s 2024-04-21T08:49:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:32 smithi136 ceph-mon[31791]: pgmap v126: 129 pgs: 129 active+clean; 268 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 551 op/s 2024-04-21T08:49:33.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:33 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:33.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:33 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:33.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:33 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:49:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:34 smithi129 ceph-mon[31272]: pgmap v127: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 1.36k op/s 2024-04-21T08:49:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:34 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:34 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:34 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:49:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:34 smithi057 ceph-mon[27968]: pgmap v127: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 1.36k op/s 2024-04-21T08:49:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:34 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:34 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:34 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:49:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:34 smithi136 ceph-mon[31791]: pgmap v127: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.4 MiB/s wr, 1.36k op/s 2024-04-21T08:49:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:34 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:49:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:34 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:49:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:34 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:49:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:36 smithi129 ceph-mon[31272]: pgmap v128: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 1.36k op/s 2024-04-21T08:49:36.608 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:36 smithi057 ceph-mon[27968]: pgmap v128: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 1.36k op/s 2024-04-21T08:49:36.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:36 smithi136 ceph-mon[31791]: pgmap v128: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.4 MiB/s wr, 1.36k op/s 2024-04-21T08:49:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:38 smithi129 ceph-mon[31272]: pgmap v129: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 1.37k op/s 2024-04-21T08:49:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:38 smithi057 ceph-mon[27968]: pgmap v129: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 1.37k op/s 2024-04-21T08:49:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:38 smithi136 ceph-mon[31791]: pgmap v129: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.0 MiB/s wr, 1.37k op/s 2024-04-21T08:49:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:40 smithi129 ceph-mon[31272]: pgmap v130: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 1.11k op/s 2024-04-21T08:49:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:40 smithi057 ceph-mon[27968]: pgmap v130: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 1.11k op/s 2024-04-21T08:49:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:40 smithi136 ceph-mon[31791]: pgmap v130: 129 pgs: 129 active+clean; 272 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 1.11k op/s 2024-04-21T08:49:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:42 smithi129 ceph-mon[31272]: pgmap v131: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 1.13k op/s 2024-04-21T08:49:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:42 smithi057 ceph-mon[27968]: pgmap v131: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 1.13k op/s 2024-04-21T08:49:42.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:42 smithi136 ceph-mon[31791]: pgmap v131: 129 pgs: 129 active+clean; 275 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 1.13k op/s 2024-04-21T08:49:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1379683850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 msgr2=0x7f66c810c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1379683850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c810c000 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f66b8009900 tx=0x7f66b802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1379683850 shutdown_connections 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1379683850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66c810c5d0 0x7f66c810ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1379683850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c810c000 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1379683850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66c8103200 0x7f66c81035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1379683850 >> 172.21.15.57:0/1379683850 conn(0x7f66c80771d0 msgr2=0x7f66c80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1379683850 shutdown_connections 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.981+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1379683850 wait complete. 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 Processor -- start 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 -- start start 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66c8103200 0x7f66c81986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66c810c5d0 0x7f66c819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8112210 con 0x7f66c8103b20 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8112090 con 0x7f66c810c5d0 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66ccdb6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8112390 con 0x7f66c8103200 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c8198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57500/0 (socket says 172.21.15.57:57500) 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66c5d9b640 1 -- 172.21.15.57:0/1063698581 learned_addr learned my addr 172.21.15.57:0/1063698581 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66c810c5d0 0x7f66c819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.982+0000 7f66c659c640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66c8103200 0x7f66c81986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66c5d9b640 1 -- 172.21.15.57:0/1063698581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66c8103200 msgr2=0x7f66c81986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66c8103200 0x7f66c81986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66c5d9b640 1 -- 172.21.15.57:0/1063698581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66c810c5d0 msgr2=0x7f66c819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66c810c5d0 0x7f66c819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66c5d9b640 1 -- 172.21.15.57:0/1063698581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c819d690 con 0x7f66c8103b20 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c8198c30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f66b8005940 tx=0x7f66b802c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66bf7fe640 1 -- 172.21.15.57:0/1063698581 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8043020 con 0x7f66c8103b20 2024-04-21T08:49:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66bf7fe640 1 -- 172.21.15.57:0/1063698581 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b8004e00 con 0x7f66c8103b20 2024-04-21T08:49:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66bf7fe640 1 -- 172.21.15.57:0/1063698581 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8003970 con 0x7f66c8103b20 2024-04-21T08:49:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c819d920 con 0x7f66c8103b20 2024-04-21T08:49:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.983+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c81a5210 con 0x7f66c8103b20 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66bf7fe640 1 -- 172.21.15.57:0/1063698581 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f66b8046020 con 0x7f66c8103b20 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 Processor -- start 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 -- start start 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x557078405620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 0x557078405fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55707840f0e0 0x5570784114d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55707836a8d0 con 0x55707840f0e0 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.986+0000 7f66cf9af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557078307e80 con 0x557078405b60 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66cf9af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557078366890 con 0x5570784068d0 2024-04-21T08:49:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55707840f0e0 0x5570784114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 0x557078405fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 0x557078405fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44100/0 (socket says 172.21.15.57:44100) 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 -- 172.21.15.57:0/3400783583 learned_addr learned my addr 172.21.15.57:0/3400783583 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55707840f0e0 0x5570784114d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57508/0 (socket says 172.21.15.57:57508) 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x557078405620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66bf7fe640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66b40a3e40 0x7f66b40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 -- 172.21.15.57:0/3400783583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 msgr2=0x557078405620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x557078405620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 -- 172.21.15.57:0/3400783583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55707840f0e0 msgr2=0x5570784114d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55707840f0e0 0x5570784114d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 -- 172.21.15.57:0/3400783583 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557078329320 con 0x557078405b60 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66b40a3e40 0x7f66b40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66bf7fe640 1 -- 172.21.15.57:0/1063698581 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f66b80f2940 con 0x7f66c8103b20 2024-04-21T08:49:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c659c640 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 0x557078405fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009cf0 tx=0x7f66b0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce145585f9c1d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66bcff9640 1 -- 172.21.15.57:0/3400783583 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0004230 con 0x557078405b60 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66b40a3e40 0x7f66b40a6300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f66c806b860 tx=0x7f66c0014040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66bcff9640 1 -- 172.21.15.57:0/3400783583 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b00043d0 con 0x557078405b60 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.987+0000 7f66bcff9640 1 -- 172.21.15.57:0/3400783583 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0019880 con 0x557078405b60 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 -- 172.21.15.57:0/3400783583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 msgr2=0x557078405fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 0x557078405fe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009cf0 tx=0x7f66b0011810 comp rx=0 tx=0).stop 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 -- 172.21.15.57:0/3400783583 shutdown_connections 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55707840f0e0 0x5570784114d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557078405b60 0x557078405fe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 --2- 172.21.15.57:0/3400783583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x557078405620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 -- 172.21.15.57:0/3400783583 >> 172.21.15.57:0/3400783583 conn(0x5570783ccc90 msgr2=0x557078400850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 -- 172.21.15.57:0/3400783583 shutdown_connections 2024-04-21T08:49:43.994 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.988+0000 7f66cf9af740 1 -- 172.21.15.57:0/3400783583 wait complete. 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 -- start start 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 Processor -- start 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 0x55707840e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x55707840bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x55707840c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557078411e10 con 0x557078405b60 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570782f2eb0 con 0x55707840f0e0 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557078411f90 con 0x5570784068d0 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66cf9af740 10 client.0 _ll_register_callbacks cb 0x55707831fb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 0x55707840e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 0x55707840e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57512/0 (socket says 172.21.15.57:57512) 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 -- 172.21.15.57:0/2989815442 learned_addr learned my addr 172.21.15.57:0/2989815442 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:43.995 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x55707840c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x55707840bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 -- 172.21.15.57:0/2989815442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 msgr2=0x55707840bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x55707840bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 -- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 msgr2=0x55707840c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x55707840c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 -- 172.21.15.57:0/2989815442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707807d210 con 0x557078405b60 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.989+0000 7f66c659c640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 0x55707840e990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f66b0005e00 tx=0x7f66b00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d487a43f58456359 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66adffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66adffb640 1 -- 172.21.15.57:0/2989815442 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0027070 con 0x557078405b60 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66adffb640 1 -- 172.21.15.57:0/2989815442 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f66b0005240 con 0x557078405b60 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66b00043d0 con 0x557078405b60 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66adffb640 1 -- 172.21.15.57:0/2989815442 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f66b0033050 con 0x557078405b60 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 10 client.16353 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557078149620 con 0x557078405b60 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66adffb640 1 -- 172.21.15.57:0/2989815442 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b00043d0 con 0x557078405b60 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66adffb640 1 -- 172.21.15.57:0/2989815442 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f66b0004b70 con 0x557078405b60 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 10 client.16353 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 10 client.16353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 20 client.16353 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55707841efb0 0x5570784213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 4 client.16353 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.990+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55707837e880 con 0x55707841efb0 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.991+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55707841efb0 0x5570784213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.991+0000 7f66adffb640 10 client.16353 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:43.997 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.991+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55707841efb0 0x5570784213a0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b53a0c1d17132c4 server_cookie=f553dd2fd4714da8 in_seq=0 out_seq=0 2024-04-21T08:49:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66adffb640 1 -- 172.21.15.57:0/2989815442 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55707837e880 con 0x55707841efb0 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66adffb640 10 client.16353 handle_command_reply: tid=0 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 client.16353 shutdown 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 msgr2=0x55707840e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 0x55707840e990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f66b0005e00 tx=0x7f66b00038a0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 shutdown_connections 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55707841efb0 0x5570784213a0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x55707840c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66adffb640 0 client.16353 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5570784068d0 0x55707840bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 --2- 172.21.15.57:0/2989815442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557078405b60 0x55707840e990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 >> 172.21.15.57:0/2989815442 conn(0x5570783ccc90 msgr2=0x5570784025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 shutdown_connections 2024-04-21T08:49:43.999 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 1 -- 172.21.15.57:0/2989815442 wait complete. 2024-04-21T08:49:44.000 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:43.993+0000 7f66cf9af740 20 client.16353 trim_cache size 0 max 16384 2024-04-21T08:49:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 Processor -- start 2024-04-21T08:49:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 -- start start 2024-04-21T08:49:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x55707841ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x55707840e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078426b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81ad150 con 0x5570784068d0 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81af9b0 con 0x55707840f0e0 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66cf9af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81ad7a0 con 0x557078405b60 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x55707840e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x55707840e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57522/0 (socket says 172.21.15.57:57522) 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66c5d9b640 1 -- 172.21.15.57:0/47278682 learned_addr learned my addr 172.21.15.57:0/47278682 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078426b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.008+0000 7f66c659c640 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x55707841ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66c5d9b640 1 -- 172.21.15.57:0/47278682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 msgr2=0x55707841ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x55707841ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66c5d9b640 1 -- 172.21.15.57:0/47278682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 msgr2=0x557078426b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078426b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66c5d9b640 1 -- 172.21.15.57:0/47278682 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570783fbf30 con 0x5570784068d0 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x55707840e4c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f66b80f0330 tx=0x7f66b80baa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c548ba5f3e7b54a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66affff640 1 -- 172.21.15.57:0/47278682 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80bdff0 con 0x5570784068d0 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66affff640 1 -- 172.21.15.57:0/47278682 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b80bed90 con 0x5570784068d0 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66affff640 1 -- 172.21.15.57:0/47278682 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80bca60 con 0x5570784068d0 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66cf9af740 1 -- 172.21.15.57:0/47278682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 msgr2=0x55707840e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66cf9af740 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x55707840e4c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f66b80f0330 tx=0x7f66b80baa60 comp rx=0 tx=0).stop 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66cf9af740 1 -- 172.21.15.57:0/47278682 shutdown_connections 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66cf9af740 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078426b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66cf9af740 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x55707840e4c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.009+0000 7f66cf9af740 1 --2- 172.21.15.57:0/47278682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x55707841ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- 172.21.15.57:0/47278682 >> 172.21.15.57:0/47278682 conn(0x5570783ccc90 msgr2=0x557078407010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- 172.21.15.57:0/47278682 shutdown_connections 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- 172.21.15.57:0/47278682 wait complete. 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- start start 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 Processor -- start 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x557078417220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x557078417760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078409030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81ade70 con 0x5570784068d0 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81af2e0 con 0x55707840f0e0 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f66c81ae540 con 0x557078405b60 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66cf9af740 10 client.0 _ll_register_callbacks cb 0x7f66b80ba9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x557078417760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66c5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x557078417760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57532/0 (socket says 172.21.15.57:57532) 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66c5d9b640 1 -- 172.21.15.57:0/358827367 learned_addr learned my addr 172.21.15.57:0/358827367 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:44.015 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66c659c640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x557078417220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.010+0000 7f66c6d9d640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078409030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66c5d9b640 1 -- 172.21.15.57:0/358827367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 msgr2=0x557078417220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x557078417220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66c5d9b640 1 -- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 msgr2=0x557078409030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078409030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66c5d9b640 1 -- 172.21.15.57:0/358827367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557078409860 con 0x5570784068d0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66c5d9b640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x557078417760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f66b80be620 tx=0x7f66b80bdff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b219b8f0a538f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66adffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66adffb640 1 -- 172.21.15.57:0/358827367 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80c2a40 con 0x5570784068d0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66adffb640 1 -- 172.21.15.57:0/358827367 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f66b80bc990 con 0x5570784068d0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66b80bed90 con 0x5570784068d0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66adffb640 1 -- 172.21.15.57:0/358827367 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f66b80c7300 con 0x5570784068d0 2024-04-21T08:49:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 10 client.16359 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55707840ff90 con 0x5570784068d0 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66adffb640 1 -- 172.21.15.57:0/358827367 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b80bed90 con 0x5570784068d0 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66adffb640 1 -- 172.21.15.57:0/358827367 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f66b8003b10 con 0x5570784068d0 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 10 client.16359 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 20 client.16359 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5570784855f0 0x557078487a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 4 client.16359 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.011+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55707830a3d0 con 0x5570784855f0 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.012+0000 7f66c659c640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5570784855f0 0x557078487a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.012+0000 7f66c659c640 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5570784855f0 0x557078487a10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f6a637f05d0925 server_cookie=7753e36e6ba30c16 in_seq=0 out_seq=0 2024-04-21T08:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:44.012+0000 7f66adffb640 10 client.16359 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:44.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:44 smithi129 ceph-mon[31272]: pgmap v132: 129 pgs: 129 active+clean; 290 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.5 MiB/s wr, 997 op/s 2024-04-21T08:49:44.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:44 smithi057 ceph-mon[27968]: pgmap v132: 129 pgs: 129 active+clean; 290 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.5 MiB/s wr, 997 op/s 2024-04-21T08:49:44.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:44 smithi136 ceph-mon[31791]: pgmap v132: 129 pgs: 129 active+clean; 290 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.5 MiB/s wr, 997 op/s 2024-04-21T08:49:44.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:44.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:45.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 -- 172.21.15.57:0/576840725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800106150 msgr2=0x7f480010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 --2- 172.21.15.57:0/576840725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800106150 0x7f480010dd20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009ca0 tx=0x7f47f802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 -- 172.21.15.57:0/576840725 shutdown_connections 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 --2- 172.21.15.57:0/576840725 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800106150 0x7f480010dd20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 --2- 172.21.15.57:0/576840725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4800101a30 0x7f4800105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 --2- 172.21.15.57:0/576840725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4800101080 0x7f4800101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 -- 172.21.15.57:0/576840725 >> 172.21.15.57:0/576840725 conn(0x7f4800078000 msgr2=0x7f4800078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 -- 172.21.15.57:0/576840725 shutdown_connections 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 -- 172.21.15.57:0/576840725 wait complete. 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.080+0000 7f4808631640 1 Processor -- start 2024-04-21T08:49:45.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 -- start start 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 0x7f4800102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4800106150 0x7f4800103050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4800103590 0x7f48001a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f48001110c0 con 0x7f4800101080 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4800110f40 con 0x7f4800103590 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4808631640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4800111240 con 0x7f4800106150 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 0x7f4800102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 0x7f4800102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57552/0 (socket says 172.21.15.57:57552) 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 -- 172.21.15.57:0/719857202 learned_addr learned my addr 172.21.15.57:0/719857202 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4806bce640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4800103590 0x7f48001a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f4805bcc640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4800106150 0x7f4800103050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 -- 172.21.15.57:0/719857202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4800106150 msgr2=0x7f4800103050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4800106150 0x7f4800103050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 -- 172.21.15.57:0/719857202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4800103590 msgr2=0x7f48001a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4800103590 0x7f48001a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 -- 172.21.15.57:0/719857202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48001a6f80 con 0x7f4800101080 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.081+0000 7f48063cd640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 0x7f4800102b10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00e9f0 tx=0x7f47fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.082+0000 7f47f77fe640 1 -- 172.21.15.57:0/719857202 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00cd80 con 0x7f4800101080 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.082+0000 7f47f77fe640 1 -- 172.21.15.57:0/719857202 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47fc0108e0 con 0x7f4800101080 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.082+0000 7f47f77fe640 1 -- 172.21.15.57:0/719857202 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc018980 con 0x7f4800101080 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.082+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48001a7270 con 0x7f4800101080 2024-04-21T08:49:45.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.082+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48001a77c0 con 0x7f4800101080 2024-04-21T08:49:45.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.084+0000 7f47f77fe640 1 -- 172.21.15.57:0/719857202 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47fc010430 con 0x7f4800101080 2024-04-21T08:49:45.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.084+0000 7f480b228740 1 Processor -- start 2024-04-21T08:49:45.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.084+0000 7f480b228740 1 -- start start 2024-04-21T08:49:45.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f480b228740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6580 0x55e0c2cb6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f480b228740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f480b228740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cc1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f480b228740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e0c2c1b1e0 con 0x55e0c2cb6ea0 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f480b228740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e0c2bb8390 con 0x55e0c2cb6580 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f480b228740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e0c2c171a0 con 0x55e0c2cbfa80 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f4806bce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6580 0x55e0c2cb6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57568/0 (socket says 172.21.15.57:57568) 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f47f77fe640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47ec0a3d50 0x7f47ec0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 -- 172.21.15.57:0/1336216491 learned_addr learned my addr 172.21.15.57:0/1336216491 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f4805bcc640 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cc1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f4806bce640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47ec0a3d50 0x7f47ec0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f47f77fe640 1 -- 172.21.15.57:0/719857202 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f47fc0d0eb0 con 0x7f4800101080 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 -- 172.21.15.57:0/1336216491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cbfa80 msgr2=0x55e0c2cc1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cc1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 -- 172.21.15.57:0/1336216491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6580 msgr2=0x55e0c2cb6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6580 0x55e0c2cb6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f48063cd640 1 -- 172.21.15.57:0/1336216491 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c29f77d0 con 0x55e0c2cb6ea0 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.085+0000 7f4806bce640 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6580 0x55e0c2cb6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.086+0000 7f48063cd640 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbf540 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f47fc09c840 tx=0x7f47fc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71242363328f9fea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.086+0000 7f4806bce640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47ec0a3d50 0x7f47ec0a6210 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f47f8005e30 tx=0x7f47f802c0f0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.086+0000 7f47f4bfe640 1 -- 172.21.15.57:0/1336216491 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc09cb40 con 0x55e0c2cb6ea0 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.086+0000 7f47f4bfe640 1 -- 172.21.15.57:0/1336216491 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47fc0a0e40 con 0x55e0c2cb6ea0 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.086+0000 7f47f4bfe640 1 -- 172.21.15.57:0/1336216491 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0d4050 con 0x55e0c2cb6ea0 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 -- 172.21.15.57:0/1336216491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 msgr2=0x55e0c2cbf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbf540 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f47fc09c840 tx=0x7f47fc00cf20 comp rx=0 tx=0).stop 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 -- 172.21.15.57:0/1336216491 shutdown_connections 2024-04-21T08:49:45.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cc1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbf540 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- 172.21.15.57:0/1336216491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6580 0x55e0c2cb6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 -- 172.21.15.57:0/1336216491 >> 172.21.15.57:0/1336216491 conn(0x55e0c2cb0cb0 msgr2=0x55e0c2cb1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 -- 172.21.15.57:0/1336216491 shutdown_connections 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 -- 172.21.15.57:0/1336216491 wait complete. 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 -- start start 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 Processor -- start 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.087+0000 7f480b228740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e0c2cc2810 con 0x55e0c2cb6580 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e0c2bd42f0 con 0x55e0c2cbfa80 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e0c2cc2990 con 0x55e0c2cb6ea0 2024-04-21T08:49:45.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbcaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57572/0 (socket says 172.21.15.57:57572) 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 -- 172.21.15.57:0/3539737028 learned_addr learned my addr 172.21.15.57:0/3539737028 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 10 client.0 _ll_register_callbacks cb 0x55e0c2be1580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f4806bce640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f4805bcc640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 -- 172.21.15.57:0/3539737028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6ea0 msgr2=0x55e0c2cbcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 -- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cbfa80 msgr2=0x55e0c2cbd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 -- 172.21.15.57:0/3539737028 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c292d3f0 con 0x55e0c2cb6580 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f4806bce640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f48063cd640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbcaa0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f47fc09c840 tx=0x7f47fc098730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31870d1cecc0577c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:45.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f47e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f47e9ffb640 1 -- 172.21.15.57:0/3539737028 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0a5700 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f47e9ffb640 1 -- 172.21.15.57:0/3539737028 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f47fc09bf90 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47fc0a0e40 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f47e9ffb640 1 -- 172.21.15.57:0/3539737028 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f47fc09c190 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 10 client.16371 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e0c28ed0f0 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.088+0000 7f47e9ffb640 1 -- 172.21.15.57:0/3539737028 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47fc0a0e40 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f47e9ffb640 1 -- 172.21.15.57:0/3539737028 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f47fc09bb00 con 0x55e0c2cb6580 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f480b228740 10 client.16371 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f480b228740 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f480b228740 20 client.16371 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f480b228740 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2cd1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f480b228740 4 client.16371 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:45.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0c2c28c00 con 0x55e0c2ccf860 2024-04-21T08:49:45.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f4805bcc640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2cd1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:45.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f47e9ffb640 10 client.16371 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:45.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:45.089+0000 7f4805bcc640 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2cd1c50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75c26b36071160a5 server_cookie=52dcdc6df8a55d64 in_seq=0 out_seq=0 2024-04-21T08:49:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:46 smithi129 ceph-mon[31272]: pgmap v133: 129 pgs: 129 active+clean; 290 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.2 MiB/s wr, 186 op/s 2024-04-21T08:49:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:46 smithi057 ceph-mon[27968]: pgmap v133: 129 pgs: 129 active+clean; 290 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.2 MiB/s wr, 186 op/s 2024-04-21T08:49:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:46 smithi136 ceph-mon[31791]: pgmap v133: 129 pgs: 129 active+clean; 290 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.2 MiB/s wr, 186 op/s 2024-04-21T08:49:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:48 smithi129 ceph-mon[31272]: pgmap v134: 129 pgs: 129 active+clean; 355 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 512 op/s 2024-04-21T08:49:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:48 smithi057 ceph-mon[27968]: pgmap v134: 129 pgs: 129 active+clean; 355 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 512 op/s 2024-04-21T08:49:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:48 smithi136 ceph-mon[31791]: pgmap v134: 129 pgs: 129 active+clean; 355 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 512 op/s 2024-04-21T08:49:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66adffb640 1 -- 172.21.15.57:0/358827367 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55707830a3d0 con 0x5570784855f0 2024-04-21T08:49:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66adffb640 10 client.16359 handle_command_reply: tid=0 2024-04-21T08:49:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 client.16359 shutdown 2024-04-21T08:49:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 msgr2=0x557078417760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x557078417760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f66b80be620 tx=0x7f66b80bdff0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 shutdown_connections 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5570784855f0 0x557078487a10 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707840f0e0 0x557078409030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5570784068d0 0x557078417760 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 --2- 172.21.15.57:0/358827367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557078405b60 0x557078417220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 >> 172.21.15.57:0/358827367 conn(0x5570783ccc90 msgr2=0x55707840a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.069+0000 7f66adffb640 0 client.16359 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 shutdown_connections 2024-04-21T08:49:49.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f47e9ffb640 1 -- 172.21.15.57:0/3539737028 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0c2c28c00 con 0x55e0c2ccf860 2024-04-21T08:49:49.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f47e9ffb640 10 client.16371 handle_command_reply: tid=0 2024-04-21T08:49:49.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 client.16371 shutdown 2024-04-21T08:49:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f66cf9af740 1 -- 172.21.15.57:0/358827367 wait complete. 2024-04-21T08:49:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f66cf9af740 20 client.16359 trim_cache size 0 max 16384 2024-04-21T08:49:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:49:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:49:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:49:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:49:49.072 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 msgr2=0x55e0c2cbcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbcaa0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f47fc09c840 tx=0x7f47fc098730 comp rx=0 tx=0).stop 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 shutdown_connections 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2cd1c50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f47e9ffb640 0 client.16371 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cbcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 --2- 172.21.15.57:0/3539737028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbcaa0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 >> 172.21.15.57:0/3539737028 conn(0x55e0c2cb0cb0 msgr2=0x55e0c2cb2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 shutdown_connections 2024-04-21T08:49:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66b40a3e40 msgr2=0x7f66b40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66b40a3e40 0x7f66b40a6300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f66c806b860 tx=0x7f66c0014040 comp rx=0 tx=0).stop 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 msgr2=0x7f66c8198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c8198c30 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f66b8005940 tx=0x7f66b802c690 comp rx=0 tx=0).stop 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 shutdown_connections 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66b40a3e40 0x7f66b40a6300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66c810c5d0 0x7f66c819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66c8103b20 0x7f66c8198c30 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 --2- 172.21.15.57:0/1063698581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66c8103200 0x7f66c81986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 >> 172.21.15.57:0/1063698581 conn(0x7f66c80771d0 msgr2=0x7f66c80fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.070+0000 7f480b228740 1 -- 172.21.15.57:0/3539737028 wait complete. 2024-04-21T08:49:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 shutdown_connections 2024-04-21T08:49:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f66ccdb6640 1 -- 172.21.15.57:0/1063698581 wait complete. 2024-04-21T08:49:49.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.071+0000 7f480b228740 20 client.16371 trim_cache size 0 max 16384 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.095+0000 7f480b228740 1 Processor -- start 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.095+0000 7f480b228740 1 -- start start 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.095+0000 7f480b228740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.095+0000 7f480b228740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2ccb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f480b228740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f480b228740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f48001af490 con 0x55e0c2cbfa80 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f480b228740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f48001b1cf0 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f480b228740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f48001afae0 con 0x55e0c2cb6580 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57584/0 (socket says 172.21.15.57:57584) 2024-04-21T08:49:49.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 -- 172.21.15.57:0/3087961658 learned_addr learned my addr 172.21.15.57:0/3087961658 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4805bcc640 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2ccb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f48063cd640 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 -- 172.21.15.57:0/3087961658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 msgr2=0x55e0c2cbf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 -- 172.21.15.57:0/3087961658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 msgr2=0x55e0c2ccb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2ccb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 -- 172.21.15.57:0/3087961658 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0c2cc28b0 con 0x55e0c2cbfa80 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4805bcc640 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2ccb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f4806bce640 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbba80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f47f8036000 tx=0x7f47f80323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608bf6b807b72477 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f47ebfff640 1 -- 172.21.15.57:0/3087961658 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8039040 con 0x55e0c2cbfa80 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f47ebfff640 1 -- 172.21.15.57:0/3087961658 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f8053070 con 0x55e0c2cbfa80 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.096+0000 7f47ebfff640 1 -- 172.21.15.57:0/3087961658 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f804e440 con 0x55e0c2cbfa80 2024-04-21T08:49:49.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 -- 172.21.15.57:0/3087961658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 msgr2=0x55e0c2cbba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbba80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f47f8036000 tx=0x7f47f80323c0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 -- 172.21.15.57:0/3087961658 shutdown_connections 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cbba80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2ccb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 --2- 172.21.15.57:0/3087961658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cbf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 -- 172.21.15.57:0/3087961658 >> 172.21.15.57:0/3087961658 conn(0x55e0c2cb0cb0 msgr2=0x55e0c2cbb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 -- 172.21.15.57:0/3087961658 shutdown_connections 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.097+0000 7f480b228740 1 -- 172.21.15.57:0/3087961658 wait complete. 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 -- start start 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 Processor -- start 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cc7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cc8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cb9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f48001b01b0 con 0x55e0c2cbfa80 2024-04-21T08:49:49.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f48001b1620 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f48001b0880 con 0x55e0c2cb6580 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f480b228740 10 client.0 _ll_register_callbacks cb 0x7f47fc087070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4806bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cb9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f48063cd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cc7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4805bcc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cc8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4806bce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cb9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57590/0 (socket says 172.21.15.57:57590) 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f48063cd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cc7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:44928/0 (socket says 172.21.15.57:44928) 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4806bce640 1 -- 172.21.15.57:0/2067910326 learned_addr learned my addr 172.21.15.57:0/2067910326 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4805bcc640 1 -- 172.21.15.57:0/2067910326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 msgr2=0x55e0c2cc7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4805bcc640 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cc7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4805bcc640 1 -- 172.21.15.57:0/2067910326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 msgr2=0x55e0c2cb9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4805bcc640 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cb9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4805bcc640 1 -- 172.21.15.57:0/2067910326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0c2cb9a10 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.098+0000 7f4806bce640 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cb9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f4805bcc640 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cc8120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55e0c2cd2190 tx=0x7f47f00044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=953149e1d91065e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f47e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f47e9ffb640 1 -- 172.21.15.57:0/2067910326 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0024030 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f8053070 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f47e9ffb640 1 -- 172.21.15.57:0/2067910326 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f47f0003800 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f47e9ffb640 1 -- 172.21.15.57:0/2067910326 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f47f002e050 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 10 client.25871 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e0c2d356b0 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f47e9ffb640 1 -- 172.21.15.57:0/2067910326 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f8053070 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f47e9ffb640 1 -- 172.21.15.57:0/2067910326 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f47f0010a00 con 0x55e0c2cb6ea0 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 10 client.25871 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 20 client.25871 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2d37e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 4 client.25871 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.099+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e0c2c22a70 con 0x55e0c2ccf860 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.100+0000 7f48063cd640 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2d37e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.100+0000 7f48063cd640 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2d37e70 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1439fd4a05606bca server_cookie=b3fea7e7578ac53d in_seq=0 out_seq=0 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.100+0000 7f47e9ffb640 10 client.25871 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.100+0000 7f47e9ffb640 1 -- 172.21.15.57:0/2067910326 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e0c2c22a70 con 0x55e0c2ccf860 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.100+0000 7f47e9ffb640 10 client.25871 handle_command_reply: tid=0 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.100+0000 7f480b228740 1 client.25871 shutdown 2024-04-21T08:49:49.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 msgr2=0x55e0c2cc8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cc8120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55e0c2cd2190 tx=0x7f47f00044d0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 shutdown_connections 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e0c2ccf860 0x55e0c2d37e70 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e0c2cbfa80 0x55e0c2cb9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f47e9ffb640 0 client.25871 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e0c2cb6ea0 0x55e0c2cc8120 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 --2- 172.21.15.57:0/2067910326 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e0c2cb6580 0x55e0c2cc7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 >> 172.21.15.57:0/2067910326 conn(0x55e0c2cb0cb0 msgr2=0x55e0c2cb20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 shutdown_connections 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 1 -- 172.21.15.57:0/2067910326 wait complete. 2024-04-21T08:49:49.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f480b228740 20 client.25871 trim_cache size 0 max 16384 2024-04-21T08:49:49.109 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:49:49.109 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:49:49.109 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:49:49.109 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:49:49.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47ec0a3d50 msgr2=0x7f47ec0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47ec0a3d50 0x7f47ec0a6210 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f47f8005e30 tx=0x7f47f802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 msgr2=0x7f4800102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 0x7f4800102b10 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00e9f0 tx=0x7f47fc00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 shutdown_connections 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f47ec0a3d50 0x7f47ec0a6210 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4800103590 0x7f48001a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4800106150 0x7f4800103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 --2- 172.21.15.57:0/719857202 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4800101080 0x7f4800102b10 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 >> 172.21.15.57:0/719857202 conn(0x7f4800078000 msgr2=0x7f4800069490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 shutdown_connections 2024-04-21T08:49:49.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.101+0000 7f4808631640 1 -- 172.21.15.57:0/719857202 wait complete. 2024-04-21T08:49:49.112 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:49:49.112 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:49:49.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 -- 172.21.15.57:0/3109540608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102bb0 msgr2=0x7f195c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 --2- 172.21.15.57:0/3109540608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102bb0 0x7f195c106bf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:49.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 -- 172.21.15.57:0/3109540608 shutdown_connections 2024-04-21T08:49:49.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 --2- 172.21.15.57:0/3109540608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 0x7f195c10eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 --2- 172.21.15.57:0/3109540608 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102bb0 0x7f195c106bf0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 --2- 172.21.15.57:0/3109540608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f195c102200 0x7f195c1025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 -- 172.21.15.57:0/3109540608 >> 172.21.15.57:0/3109540608 conn(0x7f195c0fda60 msgr2=0x7f195c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 -- 172.21.15.57:0/3109540608 shutdown_connections 2024-04-21T08:49:49.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.219+0000 7f19614ba640 1 -- 172.21.15.57:0/3109540608 wait complete. 2024-04-21T08:49:49.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 Processor -- start 2024-04-21T08:49:49.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 -- start start 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 0x7f195c19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 0x7f195c19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f195c198590 0x7f195c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112210 con 0x7f195c102200 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112090 con 0x7f195c1072d0 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f19614ba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f195c112390 con 0x7f195c198590 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 0x7f195c19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 0x7f195c19e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57608/0 (socket says 172.21.15.57:57608) 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195a7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 0x7f195c19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 -- 172.21.15.57:0/1925636479 learned_addr learned my addr 172.21.15.57:0/1925636479 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195b7fe640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f195c198590 0x7f195c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 -- 172.21.15.57:0/1925636479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f195c198590 msgr2=0x7f195c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f195c198590 0x7f195c198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 -- 172.21.15.57:0/1925636479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 msgr2=0x7f195c19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.220+0000 7f195affd640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 0x7f195c19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f195affd640 1 -- 172.21.15.57:0/1925636479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c1990b0 con 0x7f195c102200 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f195a7fc640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 0x7f195c19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f195affd640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 0x7f195c19e3e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f194c00e9f0 tx=0x7f194c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c00cd80 con 0x7f195c102200 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c1993a0 con 0x7f195c102200 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f194c0054d0 con 0x7f195c102200 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c010690 con 0x7f195c102200 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.221+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c1a55a0 con 0x7f195c102200 2024-04-21T08:49:49.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.222+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195c069230 con 0x7f195c102200 2024-04-21T08:49:49.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.223+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f194c004e90 con 0x7f195c102200 2024-04-21T08:49:49.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.224+0000 7f1943fff640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19440a3d20 0x7f19440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.224+0000 7f195a7fc640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19440a3d20 0x7f19440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.224+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f194c0d0fc0 con 0x7f195c102200 2024-04-21T08:49:49.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.225+0000 7f195a7fc640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19440a3d20 0x7f19440a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1950009230 tx=0x7f1950005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.229+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f194c09d2a0 con 0x7f195c102200 2024-04-21T08:49:49.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.460+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f195c10a500 con 0x7f195c102200 2024-04-21T08:49:49.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.460+0000 7f1943fff640 1 -- 172.21.15.57:0/1925636479 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f194c09cba0 con 0x7f195c102200 2024-04-21T08:49:49.460 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:49:49.460 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:49:49.460 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:49:49.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19440a3d20 msgr2=0x7f19440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19440a3d20 0x7f19440a61e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1950009230 tx=0x7f1950005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 msgr2=0x7f195c19e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 0x7f195c19e3e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f194c00e9f0 tx=0x7f194c00edf0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 shutdown_connections 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f19440a3d20 0x7f19440a61e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f195c198590 0x7f195c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f195c1072d0 0x7f195c19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 --2- 172.21.15.57:0/1925636479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195c102200 0x7f195c19e3e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 >> 172.21.15.57:0/1925636479 conn(0x7f195c0fda60 msgr2=0x7f195c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.462+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 shutdown_connections 2024-04-21T08:49:49.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.463+0000 7f19614ba640 1 -- 172.21.15.57:0/1925636479 wait complete. 2024-04-21T08:49:49.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 -- 172.21.15.57:0/3610321143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 msgr2=0x7fd4fc10bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 --2- 172.21.15.57:0/3610321143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc10bfd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009900 tx=0x7fd4e402bd60 comp rx=0 tx=0).stop 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 -- 172.21.15.57:0/3610321143 shutdown_connections 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 --2- 172.21.15.57:0/3610321143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4fc10c510 0x7fd4fc10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 --2- 172.21.15.57:0/3610321143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc10bfd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 --2- 172.21.15.57:0/3610321143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 0x7fd4fc1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 -- 172.21.15.57:0/3610321143 >> 172.21.15.57:0/3610321143 conn(0x7fd4fc077f50 msgr2=0x7fd4fc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 -- 172.21.15.57:0/3610321143 shutdown_connections 2024-04-21T08:49:49.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.581+0000 7fd501e8c640 1 -- 172.21.15.57:0/3610321143 wait complete. 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 Processor -- start 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 -- start start 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 0x7fd4fc1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4fc10c510 0x7fd4fc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc112180 con 0x7fd4fc103af0 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc112000 con 0x7fd4fc10c510 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd501e8c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc112300 con 0x7fd4fc1031d0 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd4f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd4f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc1a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57630/0 (socket says 172.21.15.57:57630) 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd4f3fff640 1 -- 172.21.15.57:0/2289194177 learned_addr learned my addr 172.21.15.57:0/2289194177 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4fc10c510 0x7fd4fc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.582+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 0x7fd4fc1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f3fff640 1 -- 172.21.15.57:0/2289194177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 msgr2=0x7fd4fc1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 0x7fd4fc1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f3fff640 1 -- 172.21.15.57:0/2289194177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4fc10c510 msgr2=0x7fd4fc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4fc10c510 0x7fd4fc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f3fff640 1 -- 172.21.15.57:0/2289194177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc19af90 con 0x7fd4fc103af0 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 0x7fd4fc1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc1a0b90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005940 tx=0x7fd4e40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f97fa640 1 -- 172.21.15.57:0/2289194177 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4043020 con 0x7fd4fc103af0 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4fc19b220 con 0x7fd4fc103af0 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f97fa640 1 -- 172.21.15.57:0/2289194177 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e402dc30 con 0x7fd4fc103af0 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd4f97fa640 1 -- 172.21.15.57:0/2289194177 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4003a10 con 0x7fd4fc103af0 2024-04-21T08:49:49.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.583+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4fc1a74a0 con 0x7fd4fc103af0 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd4f97fa640 1 -- 172.21.15.57:0/2289194177 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd4e4046020 con 0x7fd4fc103af0 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 Processor -- start 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 -- start start 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e4664d0 0x562a3e4668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 0x562a3e471e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.588 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562a3e3cb1f0 con 0x562a3e46fa00 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562a3e3684b0 con 0x562a3e466e80 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd504a84740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562a3e3c71b0 con 0x562a3e4664d0 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd4f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 0x562a3e471e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.586+0000 7fd4f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 0x562a3e471e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57634/0 (socket says 172.21.15.57:57634) 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44206/0 (socket says 172.21.15.57:44206) 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 -- 172.21.15.57:0/3947764769 learned_addr learned my addr 172.21.15.57:0/3947764769 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e4664d0 0x562a3e4668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 -- 172.21.15.57:0/3947764769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e4664d0 msgr2=0x562a3e4668b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e4664d0 0x562a3e4668b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 -- 172.21.15.57:0/3947764769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 msgr2=0x562a3e46f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 -- 172.21.15.57:0/3947764769 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3e35a1b0 con 0x562a3e46fa00 2024-04-21T08:49:49.589 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f97fa640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4e80a3d20 0x7fd4e80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4e80a3d20 0x7fd4e80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 0x562a3e471e50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40f0010 tx=0x7fd4e40f6ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27103e2c1a832816 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f97fa640 1 -- 172.21.15.57:0/2289194177 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd4e40f2ba0 con 0x7fd4fc103af0 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f1ffb640 1 -- 172.21.15.57:0/3947764769 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40bab90 con 0x562a3e46fa00 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f1ffb640 1 -- 172.21.15.57:0/3947764769 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e40beff0 con 0x562a3e46fa00 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.587+0000 7fd4f1ffb640 1 -- 172.21.15.57:0/3947764769 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40be250 con 0x562a3e46fa00 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 -- 172.21.15.57:0/3947764769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 msgr2=0x562a3e471e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4e80a3d20 0x7fd4e80a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4005fd0 tx=0x7fd4f4009500 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 0x562a3e471e50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40f0010 tx=0x7fd4e40f6ce0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 -- 172.21.15.57:0/3947764769 shutdown_connections 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e46fa00 0x562a3e471e50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 --2- 172.21.15.57:0/3947764769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e4664d0 0x562a3e4668b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 -- 172.21.15.57:0/3947764769 >> 172.21.15.57:0/3947764769 conn(0x562a3e460ba0 msgr2=0x562a3e460f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 -- 172.21.15.57:0/3947764769 shutdown_connections 2024-04-21T08:49:49.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 -- 172.21.15.57:0/3947764769 wait complete. 2024-04-21T08:49:49.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.588+0000 7fd504a84740 1 -- start start 2024-04-21T08:49:49.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 Processor -- start 2024-04-21T08:49:49.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e46f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562a3e472730 con 0x562a3e4664d0 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562a3e384500 con 0x562a3e466e80 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562a3e4728b0 con 0x562a3e46fa00 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd504a84740 10 client.0 _ll_register_callbacks cb 0x562a3e352b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e46f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e46f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57646/0 (socket says 172.21.15.57:57646) 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/1544546006 learned_addr learned my addr 172.21.15.57:0/1544546006 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/1544546006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 msgr2=0x562a3e46cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 msgr2=0x562a3e46c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.589+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/1544546006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3e0dd3f0 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e46f2c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00be30 tx=0x7fd4ec012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884ab895a210330f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4daffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4daffd640 1 -- 172.21.15.57:0/1544546006 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec0108a0 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4daffd640 1 -- 172.21.15.57:0/1544546006 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd4ec010a40 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4e40beff0 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4daffd640 1 -- 172.21.15.57:0/1544546006 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd4ec026050 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 10 client.16395 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562a3e1a77d0 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4daffd640 1 -- 172.21.15.57:0/1544546006 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4e40beff0 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd4daffd640 1 -- 172.21.15.57:0/1544546006 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd4ec00d350 con 0x562a3e4664d0 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 10 client.16395 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 10 client.16395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:49.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 20 client.16395 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e47f9b0 0x562a3e481e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 4 client.16395 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.590+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a3e36a930 con 0x562a3e47f9b0 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.591+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e47f9b0 0x562a3e481e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.591+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e47f9b0 0x562a3e481e20 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833e8a4570d7991c server_cookie=4d0857b2094ac511 in_seq=0 out_seq=0 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.591+0000 7fd4daffd640 10 client.16395 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd4daffd640 1 -- 172.21.15.57:0/1544546006 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a3e36a930 con 0x562a3e47f9b0 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd4daffd640 10 client.16395 handle_command_reply: tid=0 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 client.16395 shutdown 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 msgr2=0x562a3e46f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e46f2c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00be30 tx=0x7fd4ec012820 comp rx=0 tx=0).stop 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 shutdown_connections 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e47f9b0 0x562a3e481e20 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd4daffd640 0 client.16395 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e46c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 --2- 172.21.15.57:0/1544546006 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e46f2c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 >> 172.21.15.57:0/1544546006 conn(0x562a3e460ba0 msgr2=0x562a3e4628f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 shutdown_connections 2024-04-21T08:49:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 1 -- 172.21.15.57:0/1544546006 wait complete. 2024-04-21T08:49:49.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.592+0000 7fd504a84740 20 client.16395 trim_cache size 0 max 16384 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 Processor -- start 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 -- start start 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e47b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e45fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc1af4a0 con 0x562a3e4664d0 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc1b1d00 con 0x562a3e466e80 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd504a84740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc1afaf0 con 0x562a3e46fa00 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e47b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e47b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57662/0 (socket says 172.21.15.57:57662) 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2570587715 learned_addr learned my addr 172.21.15.57:0/2570587715 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e45fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2570587715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 msgr2=0x562a3e46ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.606+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2570587715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 msgr2=0x562a3e45fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.607+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e45fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.607+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2570587715 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3e466090 con 0x562a3e4664d0 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.607+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e47b5f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c010 tx=0x7fd4ec005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8900265a2dad7c16 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.607+0000 7fd4f17fa640 1 -- 172.21.15.57:0/2570587715 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec00d990 con 0x562a3e4664d0 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.607+0000 7fd4f17fa640 1 -- 172.21.15.57:0/2570587715 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4ec005040 con 0x562a3e4664d0 2024-04-21T08:49:49.608 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.607+0000 7fd4f17fa640 1 -- 172.21.15.57:0/2570587715 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec010a20 con 0x562a3e4664d0 2024-04-21T08:49:49.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- 172.21.15.57:0/2570587715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 msgr2=0x562a3e47b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e47b5f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c010 tx=0x7fd4ec005970 comp rx=0 tx=0).stop 2024-04-21T08:49:49.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- 172.21.15.57:0/2570587715 shutdown_connections 2024-04-21T08:49:49.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e46fa00 0x562a3e46ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e466e80 0x562a3e45fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- 172.21.15.57:0/2570587715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e47b5f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- 172.21.15.57:0/2570587715 >> 172.21.15.57:0/2570587715 conn(0x562a3e460ba0 msgr2=0x562a3e45f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- 172.21.15.57:0/2570587715 shutdown_connections 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- 172.21.15.57:0/2570587715 wait complete. 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- start start 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 Processor -- start 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e478050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e466e80 0x562a3e469140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e46fa00 0x562a3e469860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc1b01c0 con 0x562a3e4664d0 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc1b1630 con 0x562a3e46fa00 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.608+0000 7fd504a84740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc1b0890 con 0x562a3e466e80 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd504a84740 10 client.0 _ll_register_callbacks cb 0x7fd4ec00ba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e478050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e478050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57668/0 (socket says 172.21.15.57:57668) 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2816447803 learned_addr learned my addr 172.21.15.57:0/2816447803 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fbfff640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e46fa00 0x562a3e469860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e466e80 0x562a3e469140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2816447803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e466e80 msgr2=0x562a3e469140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e466e80 0x562a3e469140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e46fa00 msgr2=0x562a3e469860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e46fa00 0x562a3e469860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 -- 172.21.15.57:0/2816447803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3e4e4f10 con 0x562a3e4664d0 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e466e80 0x562a3e469140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4fb7fe640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e478050 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00bfe0 tx=0x7fd4ec005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872418fe893bcb3b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4daffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4daffd640 1 -- 172.21.15.57:0/2816447803 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec037020 con 0x562a3e4664d0 2024-04-21T08:49:49.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4daffd640 1 -- 172.21.15.57:0/2816447803 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd4ec00dda0 con 0x562a3e4664d0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd4ec005040 con 0x562a3e4664d0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd4daffd640 1 -- 172.21.15.57:0/2816447803 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd4ec042050 con 0x562a3e4664d0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd504a84740 10 client.16401 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.609+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562a3e4e6030 con 0x562a3e4664d0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd4daffd640 1 -- 172.21.15.57:0/2816447803 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4ec005040 con 0x562a3e4664d0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd4daffd640 1 -- 172.21.15.57:0/2816447803 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd4ec029070 con 0x562a3e4664d0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd504a84740 10 client.16401 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd504a84740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd504a84740 20 client.16401 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd504a84740 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e4e66e0 0x562a3e4e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd504a84740 4 client.16401 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a3e3d8c10 con 0x562a3e4e66e0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e4e66e0 0x562a3e4e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd4f3fff640 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e4e66e0 0x562a3e4e8b00 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=930d7b2167ea1e91 server_cookie=12e0134e1ea6d38f in_seq=0 out_seq=0 2024-04-21T08:49:49.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.610+0000 7fd4daffd640 10 client.16401 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd4daffd640 1 -- 172.21.15.57:0/2816447803 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a3e3d8c10 con 0x562a3e4e66e0 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd4daffd640 10 client.16401 handle_command_reply: tid=0 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 client.16401 shutdown 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 msgr2=0x562a3e478050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e478050 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00bfe0 tx=0x7fd4ec005490 comp rx=0 tx=0).stop 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 shutdown_connections 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562a3e4e66e0 0x562a3e4e8b00 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562a3e46fa00 0x562a3e469860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562a3e466e80 0x562a3e469140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 --2- 172.21.15.57:0/2816447803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562a3e4664d0 0x562a3e478050 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 >> 172.21.15.57:0/2816447803 conn(0x562a3e460ba0 msgr2=0x562a3e467af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd4daffd640 0 client.16401 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 shutdown_connections 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 1 -- 172.21.15.57:0/2816447803 wait complete. 2024-04-21T08:49:49.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.611+0000 7fd504a84740 20 client.16401 trim_cache size 0 max 16384 2024-04-21T08:49:49.617 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4e80a3d20 msgr2=0x7fd4e80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4e80a3d20 0x7fd4e80a61e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4005fd0 tx=0x7fd4f4009500 comp rx=0 tx=0).stop 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 msgr2=0x7fd4fc1a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc1a0b90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4005940 tx=0x7fd4e40045e0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 shutdown_connections 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd4e80a3d20 0x7fd4e80a61e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4fc10c510 0x7fd4fc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4fc103af0 0x7fd4fc1a0b90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 --2- 172.21.15.57:0/2289194177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4fc1031d0 0x7fd4fc1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 >> 172.21.15.57:0/2289194177 conn(0x7fd4fc077f50 msgr2=0x7fd4fc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 shutdown_connections 2024-04-21T08:49:49.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:49.612+0000 7fd501e8c640 1 -- 172.21.15.57:0/2289194177 wait complete. 2024-04-21T08:49:50.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:50 smithi129 ceph-mon[31272]: pgmap v135: 129 pgs: 129 active+clean; 355 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.2 MiB/s wr, 501 op/s 2024-04-21T08:49:50.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:50 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1925636479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:50 smithi057 ceph-mon[27968]: pgmap v135: 129 pgs: 129 active+clean; 355 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.2 MiB/s wr, 501 op/s 2024-04-21T08:49:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:50 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1925636479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:50.623 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:49:50.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:49:50.624 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:49:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:50 smithi136 ceph-mon[31791]: pgmap v135: 129 pgs: 129 active+clean; 355 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.2 MiB/s wr, 501 op/s 2024-04-21T08:49:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:50 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1925636479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.743+0000 7f8e10481640 1 -- 172.21.15.57:0/2417215633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 msgr2=0x7f8e0806ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.743+0000 7f8e10481640 1 --2- 172.21.15.57:0/2417215633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0806ac20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8e04009900 tx=0x7f8e0402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.743+0000 7f8e10481640 1 -- 172.21.15.57:0/2417215633 shutdown_connections 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.743+0000 7f8e10481640 1 --2- 172.21.15.57:0/2417215633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 0x7f8e0810ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.743+0000 7f8e10481640 1 --2- 172.21.15.57:0/2417215633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e0806b1f0 0x7f8e08106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.743+0000 7f8e10481640 1 --2- 172.21.15.57:0/2417215633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0806ac20 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- 172.21.15.57:0/2417215633 >> 172.21.15.57:0/2417215633 conn(0x7f8e08077eb0 msgr2=0x7f8e080782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- 172.21.15.57:0/2417215633 shutdown_connections 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- 172.21.15.57:0/2417215633 wait complete. 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 Processor -- start 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- start start 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 0x7f8e0819ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e08198650 0x7f8e08198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8e08112250 con 0x7f8e0806a840 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081120d0 con 0x7f8e08107270 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e10481640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081123d0 con 0x7f8e08198650 2024-04-21T08:49:50.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.744+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0819e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57686/0 (socket says 172.21.15.57:57686) 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 -- 172.21.15.57:0/1977875882 learned_addr learned my addr 172.21.15.57:0/1977875882 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e08198650 0x7f8e08198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 0x7f8e0819ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 -- 172.21.15.57:0/1977875882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e08198650 msgr2=0x7f8e08198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e08198650 0x7f8e08198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 -- 172.21.15.57:0/1977875882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 msgr2=0x7f8e0819ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 0x7f8e0819ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 -- 172.21.15.57:0/1977875882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e08199230 con 0x7f8e0806a840 2024-04-21T08:49:50.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 0x7f8e0819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:50.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0819e500 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8e04005940 tx=0x7f8e0402c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8dff7fe640 1 -- 172.21.15.57:0/1977875882 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04043020 con 0x7f8e0806a840 2024-04-21T08:49:50.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e081a5310 con 0x7f8e0806a840 2024-04-21T08:49:50.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8dff7fe640 1 -- 172.21.15.57:0/1977875882 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e0402cc60 con 0x7f8e0806a840 2024-04-21T08:49:50.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8dff7fe640 1 -- 172.21.15.57:0/1977875882 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e04003c30 con 0x7f8e0806a840 2024-04-21T08:49:50.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.745+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e081a56c0 con 0x7f8e0806a840 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.748+0000 7f8dff7fe640 1 -- 172.21.15.57:0/1977875882 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8e04046020 con 0x7f8e0806a840 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 Processor -- start 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 -- start start 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a18e0 0x5571743a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x5571743aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x5571743ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557174305f00 con 0x5571743a2290 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5571742a3590 con 0x5571743a18e0 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e1307b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557174301ec0 con 0x5571743aae10 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x5571743aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.754 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x5571743aa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57700/0 (socket says 172.21.15.57:57700) 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e0e21d640 1 -- 172.21.15.57:0/122030720 learned_addr learned my addr 172.21.15.57:0/122030720 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a18e0 0x5571743a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x5571743ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8dff7fe640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8df40a3d20 0x7f8df40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.749+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8df40a3d20 0x7f8df40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8dff7fe640 1 -- 172.21.15.57:0/1977875882 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8e040f2db0 con 0x7f8e0806a840 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0e21d640 1 -- 172.21.15.57:0/122030720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 msgr2=0x5571743ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x5571743ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0e21d640 1 -- 172.21.15.57:0/122030720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a18e0 msgr2=0x5571743a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a18e0 0x5571743a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0e21d640 1 -- 172.21.15.57:0/122030720 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571740e07e0 con 0x5571743a2290 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a18e0 0x5571743a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:50.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x5571743ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x5571743aa840 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8e040be800 tx=0x7f8e0402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e421b2193125d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8df40a3d20 0x7f8df40a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009fd0 tx=0x7f8e00004a40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8dfcbfe640 1 -- 172.21.15.57:0/122030720 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e040be640 con 0x5571743a2290 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8dfcbfe640 1 -- 172.21.15.57:0/122030720 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e04005560 con 0x5571743a2290 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.750+0000 7f8dfcbfe640 1 -- 172.21.15.57:0/122030720 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e040be460 con 0x5571743a2290 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 -- 172.21.15.57:0/122030720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 msgr2=0x5571743aa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x5571743aa840 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f8e040be800 tx=0x7f8e0402bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 -- 172.21.15.57:0/122030720 shutdown_connections 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x5571743ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x5571743aa840 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 --2- 172.21.15.57:0/122030720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a18e0 0x5571743a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 -- 172.21.15.57:0/122030720 >> 172.21.15.57:0/122030720 conn(0x55717439bfb0 msgr2=0x55717439c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 -- 172.21.15.57:0/122030720 shutdown_connections 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 -- 172.21.15.57:0/122030720 wait complete. 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 -- start start 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.751+0000 7f8e1307b740 1 Processor -- start 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a2290 0x5571743a82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x5571743a8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5571743adb40 con 0x5571743a18e0 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5571742bf580 con 0x5571743aae10 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5571743adcc0 con 0x5571743a2290 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 10 client.0 _ll_register_callbacks cb 0x5571742c5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743a7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57706/0 (socket says 172.21.15.57:57706) 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 -- 172.21.15.57:0/3608680276 learned_addr learned my addr 172.21.15.57:0/3608680276 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a2290 0x5571743a82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x5571743a8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 -- 172.21.15.57:0/3608680276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a2290 msgr2=0x5571743a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a2290 0x5571743a82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 -- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 msgr2=0x5571743a8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x5571743a8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 -- 172.21.15.57:0/3608680276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557174140f70 con 0x5571743a18e0 2024-04-21T08:49:50.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x5571743a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743a7db0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8e040f0360 tx=0x7f8e040c2f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b643252b0c01ad server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8df1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8df1ffb640 1 -- 172.21.15.57:0/3608680276 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e040beb00 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8df1ffb640 1 -- 172.21.15.57:0/3608680276 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8e040c72e0 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.752+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557174313920 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8df1ffb640 1 -- 172.21.15.57:0/3608680276 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8e040bf200 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 10 client.16413 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557173fd8170 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8df1ffb640 1 -- 172.21.15.57:0/3608680276 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557174313920 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8df1ffb640 1 -- 172.21.15.57:0/3608680276 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8e040ba930 con 0x5571743a18e0 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 10 client.16413 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 20 client.16413 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571743bae90 0x5571743bd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 4 client.16413 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:50.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55717430d790 con 0x5571743bae90 2024-04-21T08:49:50.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.753+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571743bae90 0x5571743bd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.754+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571743bae90 0x5571743bd280 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ea124ae105aa97c server_cookie=5849bfdb4280291c in_seq=0 out_seq=0 2024-04-21T08:49:50.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.754+0000 7f8df1ffb640 10 client.16413 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8df1ffb640 1 -- 172.21.15.57:0/3608680276 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55717430d790 con 0x5571743bae90 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8df1ffb640 10 client.16413 handle_command_reply: tid=0 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 client.16413 shutdown 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 msgr2=0x5571743a7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743a7db0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f8e040f0360 tx=0x7f8e040c2f30 comp rx=0 tx=0).stop 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 shutdown_connections 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5571743bae90 0x5571743bd280 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x5571743a8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a2290 0x5571743a82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 --2- 172.21.15.57:0/3608680276 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743a7db0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 >> 172.21.15.57:0/3608680276 conn(0x55717439bfb0 msgr2=0x55717439dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8df1ffb640 0 client.16413 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 shutdown_connections 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 1 -- 172.21.15.57:0/3608680276 wait complete. 2024-04-21T08:49:50.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.755+0000 7f8e1307b740 20 client.16413 trim_cache size 0 max 16384 2024-04-21T08:49:50.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 Processor -- start 2024-04-21T08:49:50.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 -- start start 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743b6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a2290 0x5571743a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x55717439db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081ad240 con 0x5571743a18e0 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081afaa0 con 0x5571743a2290 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.773+0000 7f8e1307b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081ad890 con 0x5571743aae10 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743b6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57708/0 (socket says 172.21.15.57:57708) 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a2290 0x5571743a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 -- 172.21.15.57:0/2816986597 learned_addr learned my addr 172.21.15.57:0/2816986597 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x55717439db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 -- 172.21.15.57:0/2816986597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 msgr2=0x55717439db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x55717439db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 -- 172.21.15.57:0/2816986597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a2290 msgr2=0x5571743a8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a2290 0x5571743a8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 -- 172.21.15.57:0/2816986597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571743a6300 con 0x5571743a18e0 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743b6760 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8e040bea90 tx=0x7f8e040f0360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cae5d35843b8ab server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8df3fff640 1 -- 172.21.15.57:0/2816986597 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e040bada0 con 0x5571743a18e0 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8df3fff640 1 -- 172.21.15.57:0/2816986597 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e040beb00 con 0x5571743a18e0 2024-04-21T08:49:50.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.774+0000 7f8df3fff640 1 -- 172.21.15.57:0/2816986597 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e040c7540 con 0x5571743a18e0 2024-04-21T08:49:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 -- 172.21.15.57:0/2816986597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 msgr2=0x5571743b6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743b6760 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8e040bea90 tx=0x7f8e040f0360 comp rx=0 tx=0).stop 2024-04-21T08:49:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 -- 172.21.15.57:0/2816986597 shutdown_connections 2024-04-21T08:49:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743aae10 0x55717439db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743a2290 0x5571743a8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 --2- 172.21.15.57:0/2816986597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a18e0 0x5571743b6760 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 -- 172.21.15.57:0/2816986597 >> 172.21.15.57:0/2816986597 conn(0x55717439bfb0 msgr2=0x55717439d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 -- 172.21.15.57:0/2816986597 shutdown_connections 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 -- 172.21.15.57:0/2816986597 wait complete. 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 -- start start 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.775+0000 7f8e1307b740 1 Processor -- start 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a18e0 0x5571743b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x55717441c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x55717441c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081adf60 con 0x5571743a2290 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081af3d0 con 0x5571743aae10 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8e081ae630 con 0x5571743a18e0 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e1307b740 10 client.0 _ll_register_callbacks cb 0x7f8e000099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x55717441c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x55717441c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57720/0 (socket says 172.21.15.57:57720) 2024-04-21T08:49:50.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 -- 172.21.15.57:0/1431943624 learned_addr learned my addr 172.21.15.57:0/1431943624 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a18e0 0x5571743b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x55717441c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 -- 172.21.15.57:0/1431943624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a18e0 msgr2=0x5571743b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a18e0 0x5571743b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 -- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 msgr2=0x55717441c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x55717441c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 -- 172.21.15.57:0/1431943624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55717441cd40 con 0x5571743a2290 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0ea1e640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x55717441c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8e0da1c640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x55717441c0c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8e08199ac0 tx=0x7f8df8011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40fba9419097743 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8df1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.776+0000 7f8df1ffb640 1 -- 172.21.15.57:0/1431943624 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df801c420 con 0x5571743a2290 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8df1ffb640 1 -- 172.21.15.57:0/1431943624 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8df801c5c0 con 0x5571743a2290 2024-04-21T08:49:50.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e040beb00 con 0x5571743a2290 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8df1ffb640 1 -- 172.21.15.57:0/1431943624 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8df802b050 con 0x5571743a2290 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 10 client.16419 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557174425560 con 0x5571743a2290 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8df1ffb640 1 -- 172.21.15.57:0/1431943624 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e040beb00 con 0x5571743a2290 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8df1ffb640 1 -- 172.21.15.57:0/1431943624 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8df800f490 con 0x5571743a2290 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 10 client.16419 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 20 client.16419 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557174425d60 0x557174428180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 4 client.16419 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557174289c50 con 0x557174425d60 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.777+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557174425d60 0x557174428180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.778+0000 7f8e0e21d640 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557174425d60 0x557174428180 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3ae8e2f233ae244 server_cookie=9d78f54c6a7f7aaf in_seq=0 out_seq=0 2024-04-21T08:49:50.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.778+0000 7f8df1ffb640 10 client.16419 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8df1ffb640 1 -- 172.21.15.57:0/1431943624 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557174289c50 con 0x557174425d60 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8df1ffb640 10 client.16419 handle_command_reply: tid=0 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 client.16419 shutdown 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 msgr2=0x55717441c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x55717441c0c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8e08199ac0 tx=0x7f8df8011b50 comp rx=0 tx=0).stop 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 shutdown_connections 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557174425d60 0x557174428180 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8df1ffb640 0 client.16419 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5571743aae10 0x55717441c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5571743a2290 0x55717441c0c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 --2- 172.21.15.57:0/1431943624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5571743a18e0 0x5571743b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 >> 172.21.15.57:0/1431943624 conn(0x55717439bfb0 msgr2=0x55717439dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 shutdown_connections 2024-04-21T08:49:50.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 1 -- 172.21.15.57:0/1431943624 wait complete. 2024-04-21T08:49:50.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.779+0000 7f8e1307b740 20 client.16419 trim_cache size 0 max 16384 2024-04-21T08:49:50.785 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:49:50.785 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:49:50.785 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "07c89291-ef4f-4376-be44-a7079e3bbd90", 2024-04-21T08:49:50.785 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:49:50.785 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8df40a3d20 msgr2=0x7f8df40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8df40a3d20 0x7f8df40a61e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009fd0 tx=0x7f8e00004a40 comp rx=0 tx=0).stop 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 msgr2=0x7f8e0819e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0819e500 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8e04005940 tx=0x7f8e0402c860 comp rx=0 tx=0).stop 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 shutdown_connections 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8df40a3d20 0x7f8df40a61e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e08198650 0x7f8e08198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e08107270 0x7f8e0819ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 --2- 172.21.15.57:0/1977875882 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e0806a840 0x7f8e0819e500 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 >> 172.21.15.57:0/1977875882 conn(0x7f8e08077eb0 msgr2=0x7f8e08108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 shutdown_connections 2024-04-21T08:49:50.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.780+0000 7f8e10481640 1 -- 172.21.15.57:0/1977875882 wait complete. 2024-04-21T08:49:50.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:49:50.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:49:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 -- 172.21.15.57:0/4260957154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b24101a30 msgr2=0x7f6b24105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/4260957154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b24101a30 0x7f6b24105a70 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009900 tx=0x7f6b0c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 -- 172.21.15.57:0/4260957154 shutdown_connections 2024-04-21T08:49:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/4260957154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 0x7f6b2410dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/4260957154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b24101a30 0x7f6b24105a70 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/4260957154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 0x7f6b24101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 -- 172.21.15.57:0/4260957154 >> 172.21.15.57:0/4260957154 conn(0x7f6b24078000 msgr2=0x7f6b24078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 -- 172.21.15.57:0/4260957154 shutdown_connections 2024-04-21T08:49:50.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.900+0000 7f6b28ceb640 1 -- 172.21.15.57:0/4260957154 wait complete. 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 Processor -- start 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 -- start start 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 0x7f6b241986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 0x7f6b24198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 0x7f6b2419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241110e0 con 0x7f6b2419cfa0 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b24110f60 con 0x7f6b24101080 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.901+0000 7f6b28ceb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b24111260 con 0x7f6b24106150 2024-04-21T08:49:50.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 0x7f6b2419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 0x7f6b2419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57744/0 (socket says 172.21.15.57:57744) 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 -- 172.21.15.57:0/2400278250 learned_addr learned my addr 172.21.15.57:0/2400278250 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b2259c640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 0x7f6b241986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 0x7f6b24198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 -- 172.21.15.57:0/2400278250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 msgr2=0x7f6b24198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 0x7f6b24198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 -- 172.21.15.57:0/2400278250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 msgr2=0x7f6b241986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 0x7f6b241986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 -- 172.21.15.57:0/2400278250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2419da90 con 0x7f6b2419cfa0 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 0x7f6b24198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b2259c640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 0x7f6b241986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b22d9d640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 0x7f6b2419d450 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00eaa0 tx=0x7f6b1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b1b7fe640 1 -- 172.21.15.57:0/2400278250 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c00cdb0 con 0x7f6b2419cfa0 2024-04-21T08:49:50.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b1b7fe640 1 -- 172.21.15.57:0/2400278250 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b1c005020 con 0x7f6b2419cfa0 2024-04-21T08:49:50.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2419dd80 con 0x7f6b2419cfa0 2024-04-21T08:49:50.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b1b7fe640 1 -- 172.21.15.57:0/2400278250 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c0116d0 con 0x7f6b2419cfa0 2024-04-21T08:49:50.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.902+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b241a5570 con 0x7f6b2419cfa0 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b1b7fe640 1 -- 172.21.15.57:0/2400278250 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6b1c0051d0 con 0x7f6b2419cfa0 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 Processor -- start 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 -- start start 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb8f60 0x560ecedb9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 0x560ecedc6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x560eced1e1e0 con 0x560ecedb9910 2024-04-21T08:49:50.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560ececbb390 con 0x560ecedb8f60 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560eced1a1a0 con 0x560ecedc6e90 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 0x560ecedc6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 0x560ecedc6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57754/0 (socket says 172.21.15.57:57754) 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb8f60 0x560ecedb9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b2259c640 1 -- 172.21.15.57:0/4202999399 learned_addr learned my addr 172.21.15.57:0/4202999399 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.905+0000 7f6b1b7fe640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b100a3d20 0x7f6b100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b2259c640 1 -- 172.21.15.57:0/4202999399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 msgr2=0x560ecedc9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b2259c640 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b2259c640 1 -- 172.21.15.57:0/4202999399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb8f60 msgr2=0x560ecedb9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b2259c640 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb8f60 0x560ecedb9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b2259c640 1 -- 172.21.15.57:0/4202999399 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eceafa7d0 con 0x560ecedb9910 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b22d9d640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b100a3d20 0x7f6b100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b2259c640 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 0x560ecedc6950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6b140115f0 tx=0x7f6b14011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229098d1fc004975 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b1b7fe640 1 -- 172.21.15.57:0/2400278250 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6b1c0d0ca0 con 0x7f6b2419cfa0 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b18ff9640 1 -- 172.21.15.57:0/4202999399 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1401e070 con 0x560ecedb9910 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b18ff9640 1 -- 172.21.15.57:0/4202999399 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b1400faf0 con 0x560ecedb9910 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.906+0000 7f6b18ff9640 1 -- 172.21.15.57:0/4202999399 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14019b10 con 0x560ecedb9910 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4202999399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 msgr2=0x560ecedc6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b22d9d640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b100a3d20 0x7f6b100a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c098d40 tx=0x7f6b1c06e320 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 0x560ecedc6950 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6b140115f0 tx=0x7f6b14011ab0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4202999399 shutdown_connections 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb9910 0x560ecedc6950 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4202999399 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb8f60 0x560ecedb9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4202999399 >> 172.21.15.57:0/4202999399 conn(0x560ecedb3cb0 msgr2=0x560ecedb4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4202999399 shutdown_connections 2024-04-21T08:49:50.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4202999399 wait complete. 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 -- start start 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 Processor -- start 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 0x560ecedc3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.907+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x560ecedc9c50 con 0x560ecedb8f60 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560ececd72f0 con 0x560ecedb9910 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560ecedc9dd0 con 0x560ecedc6e90 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 10 client.0 _ll_register_callbacks cb 0x560ecece4580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 0x560ecedc3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 0x560ecedc3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57756/0 (socket says 172.21.15.57:57756) 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 -- 172.21.15.57:0/4246092428 learned_addr learned my addr 172.21.15.57:0/4246092428 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2259c640 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 -- 172.21.15.57:0/4246092428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 msgr2=0x560ecedc4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 -- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 msgr2=0x560ecedc43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 -- 172.21.15.57:0/4246092428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ecea303f0 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 0x560ecedc3ea0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c036040 tx=0x7f6b0c009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4884d8d0f15af066 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b09ffb640 1 -- 172.21.15.57:0/4246092428 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c046070 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b09ffb640 1 -- 172.21.15.57:0/4246092428 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6b0c02c9f0 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b1400faf0 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b09ffb640 1 -- 172.21.15.57:0/4246092428 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b0c051050 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 10 client.16431 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.908+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560ece9f00f0 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b09ffb640 1 -- 172.21.15.57:0/4246092428 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b1400faf0 con 0x560ecedb8f60 2024-04-21T08:49:50.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b09ffb640 1 -- 172.21.15.57:0/4246092428 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6b0c034440 con 0x560ecedb8f60 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2b9ba740 10 client.16431 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2b9ba740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2b9ba740 20 client.16431 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecedd6dc0 0x560ecedd91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2b9ba740 4 client.16431 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eced2bc00 con 0x560ecedd6dc0 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2259c640 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecedd6dc0 0x560ecedd91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b09ffb640 10 client.16431 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.909+0000 7f6b2259c640 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecedd6dc0 0x560ecedd91f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=139e8e20ce904e51 server_cookie=1386606068299313 in_seq=0 out_seq=0 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b09ffb640 1 -- 172.21.15.57:0/4246092428 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560eced2bc00 con 0x560ecedd6dc0 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b09ffb640 10 client.16431 handle_command_reply: tid=0 2024-04-21T08:49:50.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 client.16431 shutdown 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 msgr2=0x560ecedc3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 0x560ecedc3ea0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c036040 tx=0x7f6b0c009630 comp rx=0 tx=0).stop 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 shutdown_connections 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecedd6dc0 0x560ecedd91f0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc6e90 0x560ecedc4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/4246092428 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedb8f60 0x560ecedc3ea0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b09ffb640 0 client.16431 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 >> 172.21.15.57:0/4246092428 conn(0x560ecedb3cb0 msgr2=0x560ecedb5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 shutdown_connections 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/4246092428 wait complete. 2024-04-21T08:49:50.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.911+0000 7f6b2b9ba740 20 client.16431 trim_cache size 0 max 16384 2024-04-21T08:49:50.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 Processor -- start 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 -- start start 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecedd2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedb2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241ab050 con 0x560ecedc6e90 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241ad8b0 con 0x560ecedb9910 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241ab6a0 con 0x560ecedc3000 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecedd2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecedd2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57770/0 (socket says 172.21.15.57:57770) 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b2259c640 1 -- 172.21.15.57:0/84064910 learned_addr learned my addr 172.21.15.57:0/84064910 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.929+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44354/0 (socket says 172.21.15.57:44354) 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedb2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 -- 172.21.15.57:0/84064910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 msgr2=0x560ecedb2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedb2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 -- 172.21.15.57:0/84064910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 msgr2=0x560ecedd2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecedd2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 -- 172.21.15.57:0/84064910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ecedb96b0 con 0x560ecedb9910 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b2259c640 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecedd2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc65d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c034840 tx=0x7f6b0c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67689159e69cc455 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b0bfff640 1 -- 172.21.15.57:0/84064910 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c05c070 con 0x560ecedb9910 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b0bfff640 1 -- 172.21.15.57:0/84064910 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b0c034440 con 0x560ecedb9910 2024-04-21T08:49:50.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.930+0000 7f6b0bfff640 1 -- 172.21.15.57:0/84064910 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c046040 con 0x560ecedb9910 2024-04-21T08:49:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/84064910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 msgr2=0x560ecedc65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc65d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c034840 tx=0x7f6b0c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/84064910 shutdown_connections 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedb2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecedd2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/84064910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc65d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/84064910 >> 172.21.15.57:0/84064910 conn(0x560ecedb3cb0 msgr2=0x560ecedb24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/84064910 shutdown_connections 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/84064910 wait complete. 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- start start 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 Processor -- start 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedc1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecee3c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241abd70 con 0x560ecedc6e90 2024-04-21T08:49:50.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241ad1e0 con 0x560ecedb9910 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b241ac440 con 0x560ecedc3000 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b2b9ba740 10 client.0 _ll_register_callbacks cb 0x7f6b0c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.931+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecee3c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecee3c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57786/0 (socket says 172.21.15.57:57786) 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 -- 172.21.15.57:0/1300083533 learned_addr learned my addr 172.21.15.57:0/1300083533 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b2259c640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedc1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 -- 172.21.15.57:0/1300083533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 msgr2=0x560ecedc1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedc1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 -- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 msgr2=0x560ecedc0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 -- 172.21.15.57:0/1300083533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ecee3c870 con 0x560ecedc6e90 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b2259c640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedc1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:49:50.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b1bfff640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecee3c330 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c034930 tx=0x7f6b0c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71a3f0c1bdcca99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b09ffb640 1 -- 172.21.15.57:0/1300083533 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c05c030 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b09ffb640 1 -- 172.21.15.57:0/1300083533 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6b0c0035f0 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b0c034440 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b09ffb640 1 -- 172.21.15.57:0/1300083533 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b0c06a050 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b2b9ba740 10 client.16437 fetch_fsmap learned FSMap version 21 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560ecee3d9c0 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.932+0000 7f6b09ffb640 1 -- 172.21.15.57:0/1300083533 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b0c034440 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b09ffb640 1 -- 172.21.15.57:0/1300083533 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6b0c0424c0 con 0x560ecedc6e90 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2b9ba740 10 client.16437 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2b9ba740 10 client.16437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2b9ba740 20 client.16437 populate_metadata read hostname 'smithi057' 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecee3dfc0 0x560ecee403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:49:50.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2b9ba740 4 client.16437 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560eced25a70 con 0x560ecee3dfc0 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2259c640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecee3dfc0 0x560ecee403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b09ffb640 10 client.16437 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.933+0000 7f6b2259c640 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecee3dfc0 0x560ecee403b0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbfc4d33b2735f2c server_cookie=fa1a40dcbacba990 in_seq=0 out_seq=0 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b09ffb640 1 -- 172.21.15.57:0/1300083533 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x560eced25a70 con 0x560ecee3dfc0 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b09ffb640 10 client.16437 handle_command_reply: tid=0 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 client.16437 shutdown 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 msgr2=0x560ecee3c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecee3c330 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c034930 tx=0x7f6b0c02bc30 comp rx=0 tx=0).stop 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 shutdown_connections 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560ecee3dfc0 0x560ecee403b0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560ecedc6e90 0x560ecee3c330 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560ecedc3000 0x560ecedc1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b09ffb640 0 client.16437 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 --2- 172.21.15.57:0/1300083533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560ecedb9910 0x560ecedc0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 >> 172.21.15.57:0/1300083533 conn(0x560ecedb3cb0 msgr2=0x560ecedc46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.935+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 shutdown_connections 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b2b9ba740 1 -- 172.21.15.57:0/1300083533 wait complete. 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b2b9ba740 20 client.16437 trim_cache size 0 max 16384 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (382 inodes in the stack)", 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stdout: "07c89291-ef4f-4376-be44-a7079e3bbd90": { 2024-04-21T08:49:50.941 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:49:50.942 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "07c89291-ef4f-4376-be44-a7079e3bbd90", 2024-04-21T08:49:50.942 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:49:50.942 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:50.942 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:49:50.942 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b100a3d20 msgr2=0x7f6b100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b100a3d20 0x7f6b100a61e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c098d40 tx=0x7f6b1c06e320 comp rx=0 tx=0).stop 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 msgr2=0x7f6b2419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 0x7f6b2419d450 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00eaa0 tx=0x7f6b1c00ef60 comp rx=0 tx=0).stop 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 shutdown_connections 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b100a3d20 0x7f6b100a61e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2419cfa0 0x7f6b2419d450 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b24106150 0x7f6b24198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 --2- 172.21.15.57:0/2400278250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b24101080 0x7f6b241986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 >> 172.21.15.57:0/2400278250 conn(0x7f6b24078000 msgr2=0x7f6b240693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 shutdown_connections 2024-04-21T08:49:50.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:49:50.936+0000 7f6b28ceb640 1 -- 172.21.15.57:0/2400278250 wait complete. 2024-04-21T08:49:50.947 INFO:tasks.cephfs.filesystem:scrub status for tag:07c89291-ef4f-4376-be44-a7079e3bbd90 - {'path': '/', 'tag': '07c89291-ef4f-4376-be44-a7079e3bbd90', 'options': 'recursive,force'} 2024-04-21T08:49:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:52 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:49:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:52 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:52 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:49:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:52 smithi129 ceph-mon[31272]: pgmap v136: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 652 op/s 2024-04-21T08:49:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:52 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:49:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:52 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:52 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:49:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:52 smithi057 ceph-mon[27968]: pgmap v136: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 652 op/s 2024-04-21T08:49:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:52 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:49:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:52 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:49:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:52 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:49:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:52 smithi136 ceph-mon[31791]: pgmap v136: 129 pgs: 129 active+clean; 339 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.3 MiB/s wr, 652 op/s 2024-04-21T08:49:54.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:54 smithi129 ceph-mon[31272]: pgmap v137: 129 pgs: 129 active+clean; 334 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.5 MiB/s wr, 1.96k op/s 2024-04-21T08:49:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:54 smithi057 ceph-mon[27968]: pgmap v137: 129 pgs: 129 active+clean; 334 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.5 MiB/s wr, 1.96k op/s 2024-04-21T08:49:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:54 smithi136 ceph-mon[31791]: pgmap v137: 129 pgs: 129 active+clean; 334 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 7.5 MiB/s wr, 1.96k op/s 2024-04-21T08:49:56.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:56 smithi129 ceph-mon[31272]: pgmap v138: 129 pgs: 129 active+clean; 334 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 1.80k op/s 2024-04-21T08:49:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:56 smithi057 ceph-mon[27968]: pgmap v138: 129 pgs: 129 active+clean; 334 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 1.80k op/s 2024-04-21T08:49:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:56 smithi136 ceph-mon[31791]: pgmap v138: 129 pgs: 129 active+clean; 334 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 1.80k op/s 2024-04-21T08:49:58.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:49:58 smithi129 ceph-mon[31272]: pgmap v139: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.5 MiB/s wr, 3.59k op/s 2024-04-21T08:49:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:49:58 smithi057 ceph-mon[27968]: pgmap v139: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.5 MiB/s wr, 3.59k op/s 2024-04-21T08:49:58.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:49:58 smithi136 ceph-mon[31791]: pgmap v139: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 6.5 MiB/s wr, 3.59k op/s 2024-04-21T08:50:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:00 smithi129 ceph-mon[31272]: pgmap v140: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 3.26k op/s 2024-04-21T08:50:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:00 smithi129 ceph-mon[31272]: overall HEALTH_OK 2024-04-21T08:50:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:00 smithi057 ceph-mon[27968]: pgmap v140: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 3.26k op/s 2024-04-21T08:50:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:00 smithi057 ceph-mon[27968]: overall HEALTH_OK 2024-04-21T08:50:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:00 smithi136 ceph-mon[31791]: pgmap v140: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 900 KiB/s wr, 3.26k op/s 2024-04-21T08:50:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:00 smithi136 ceph-mon[31791]: overall HEALTH_OK 2024-04-21T08:50:01.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:01 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:01.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:01 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:01.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:01 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.234+0000 7f66123ce640 1 -- 172.21.15.57:0/1262842266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 msgr2=0x7f660c11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.238 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.234+0000 7f66123ce640 1 --2- 172.21.15.57:0/1262842266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c11db20 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6600009930 tx=0x7f660002bd10 comp rx=0 tx=0).stop 2024-04-21T08:50:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.241+0000 7f66123ce640 1 -- 172.21.15.57:0/1262842266 shutdown_connections 2024-04-21T08:50:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.241+0000 7f66123ce640 1 --2- 172.21.15.57:0/1262842266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c11db20 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.241+0000 7f66123ce640 1 --2- 172.21.15.57:0/1262842266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f660c110b20 0x7f660c109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.241+0000 7f66123ce640 1 --2- 172.21.15.57:0/1262842266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660c110740 0x7f660c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.241+0000 7f66123ce640 1 -- 172.21.15.57:0/1262842266 >> 172.21.15.57:0/1262842266 conn(0x7f660c06d020 msgr2=0x7f660c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.246+0000 7f66123ce640 1 -- 172.21.15.57:0/1262842266 shutdown_connections 2024-04-21T08:50:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.249+0000 7f66123ce640 1 -- 172.21.15.57:0/1262842266 wait complete. 2024-04-21T08:50:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.250+0000 7f66123ce640 1 Processor -- start 2024-04-21T08:50:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 -- start start 2024-04-21T08:50:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660c110740 0x7f660c118d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c111df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f660c112330 0x7f660c1127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f660c121100 con 0x7f660c11b730 2024-04-21T08:50:02.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f660c120f80 con 0x7f660c110740 2024-04-21T08:50:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.255+0000 7f66123ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f660c121280 con 0x7f660c112330 2024-04-21T08:50:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.257+0000 7f660b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c111df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.257+0000 7f660b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c111df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55592/0 (socket says 172.21.15.57:55592) 2024-04-21T08:50:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.257+0000 7f660b7fe640 1 -- 172.21.15.57:0/1656193542 learned_addr learned my addr 172.21.15.57:0/1656193542 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.257+0000 7f660bfff640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660c110740 0x7f660c118d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.260+0000 7f660b7fe640 1 -- 172.21.15.57:0/1656193542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f660c112330 msgr2=0x7f660c1127e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:50:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.260+0000 7f660b7fe640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f660c112330 0x7f660c1127e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.260+0000 7f660b7fe640 1 -- 172.21.15.57:0/1656193542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660c110740 msgr2=0x7f660c118d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.260+0000 7f660b7fe640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660c110740 0x7f660c118d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.260+0000 7f660b7fe640 1 -- 172.21.15.57:0/1656193542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c116890 con 0x7f660c11b730 2024-04-21T08:50:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.260+0000 7f660b7fe640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c111df0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00b470 tx=0x7f65fc00b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.261+0000 7f66097fa640 1 -- 172.21.15.57:0/1656193542 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc015020 con 0x7f660c11b730 2024-04-21T08:50:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.261+0000 7f66097fa640 1 -- 172.21.15.57:0/1656193542 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65fc011d40 con 0x7f660c11b730 2024-04-21T08:50:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.261+0000 7f66097fa640 1 -- 172.21.15.57:0/1656193542 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc00f340 con 0x7f660c11b730 2024-04-21T08:50:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.262+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c116b80 con 0x7f660c11b730 2024-04-21T08:50:02.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.263+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c117080 con 0x7f660c11b730 2024-04-21T08:50:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.267+0000 7f66097fa640 1 -- 172.21.15.57:0/1656193542 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f65fc018020 con 0x7f660c11b730 2024-04-21T08:50:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.268+0000 7f66097fa640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f65f40a3ff0 0x7f65f40a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.269 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.268+0000 7f66097fa640 1 -- 172.21.15.57:0/1656193542 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f65fc0d04c0 con 0x7f660c11b730 2024-04-21T08:50:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.269+0000 7f660bfff640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f65f40a3ff0 0x7f65f40a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.270+0000 7f660bfff640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f65f40a3ff0 0x7f65f40a64b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6604009f40 tx=0x7f66040094d0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.272+0000 7f6614fc4740 1 Processor -- start 2024-04-21T08:50:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 -- start start 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801e9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55848014e8d0 con 0x5584801e9b60 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584800ebe80 con 0x5584801f30e0 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.275+0000 7f6614fc4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55848014a890 con 0x5584801ea8d0 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.276+0000 7f660b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.276+0000 7f660b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801e9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55594/0 (socket says 172.21.15.57:55594) 2024-04-21T08:50:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.276+0000 7f660b7fe640 1 -- 172.21.15.57:0/819815138 learned_addr learned my addr 172.21.15.57:0/819815138 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.277+0000 7f660bfff640 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.277+0000 7f661096b640 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.278+0000 7f660bfff640 1 -- 172.21.15.57:0/819815138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 msgr2=0x5584801f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.278+0000 7f660bfff640 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.278+0000 7f660bfff640 1 -- 172.21.15.57:0/819815138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 msgr2=0x5584801e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.278+0000 7f660bfff640 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.278+0000 7f660bfff640 1 -- 172.21.15.57:0/819815138 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848010d320 con 0x5584801ea8d0 2024-04-21T08:50:02.280 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.280+0000 7f660bfff640 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801e9620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6604017730 tx=0x7f6604011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7cc817ba7625982 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.285+0000 7f65f27fc640 1 -- 172.21.15.57:0/819815138 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604027070 con 0x5584801ea8d0 2024-04-21T08:50:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.285+0000 7f65f27fc640 1 -- 172.21.15.57:0/819815138 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6604017de0 con 0x5584801ea8d0 2024-04-21T08:50:02.287 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.285+0000 7f65f27fc640 1 -- 172.21.15.57:0/819815138 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604016560 con 0x5584801ea8d0 2024-04-21T08:50:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.288+0000 7f6614fc4740 1 -- 172.21.15.57:0/819815138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 msgr2=0x5584801e9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.288 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.288+0000 7f6614fc4740 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801e9620 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6604017730 tx=0x7f6604011000 comp rx=0 tx=0).stop 2024-04-21T08:50:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.289+0000 7f6614fc4740 1 -- 172.21.15.57:0/819815138 shutdown_connections 2024-04-21T08:50:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.289+0000 7f6614fc4740 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.289+0000 7f6614fc4740 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.289+0000 7f6614fc4740 1 --2- 172.21.15.57:0/819815138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801e9620 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.290 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.289+0000 7f6614fc4740 1 -- 172.21.15.57:0/819815138 >> 172.21.15.57:0/819815138 conn(0x5584801b0c90 msgr2=0x5584801e4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.293+0000 7f6614fc4740 1 -- 172.21.15.57:0/819815138 shutdown_connections 2024-04-21T08:50:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.299+0000 7f6614fc4740 1 -- 172.21.15.57:0/819815138 wait complete. 2024-04-21T08:50:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.300+0000 7f6614fc4740 1 -- start start 2024-04-21T08:50:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.300+0000 7f6614fc4740 1 Processor -- start 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801e9b60 0x5584801f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 0x5584801efee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5584801f5e10 con 0x5584801ea8d0 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584800d6eb0 con 0x5584801f30e0 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5584801f5f90 con 0x5584801e9b60 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f6614fc4740 10 client.0 _ll_register_callbacks cb 0x558480103b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.303+0000 7f660b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 0x5584801efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.304 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.304+0000 7f660b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 0x5584801efee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55598/0 (socket says 172.21.15.57:55598) 2024-04-21T08:50:02.305 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.304+0000 7f660b7fe640 1 -- 172.21.15.57:0/1526493692 learned_addr learned my addr 172.21.15.57:0/1526493692 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.305+0000 7f660b7fe640 1 -- 172.21.15.57:0/1526493692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801e9b60 msgr2=0x5584801f2b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:50:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.305+0000 7f660b7fe640 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801e9b60 0x5584801f2b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.305+0000 7f660b7fe640 1 -- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 msgr2=0x5584801f0620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:50:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.305+0000 7f660b7fe640 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f0620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.305+0000 7f660b7fe640 1 -- 172.21.15.57:0/1526493692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55847fe61210 con 0x5584801ea8d0 2024-04-21T08:50:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.306+0000 7f660b7fe640 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 0x5584801efee0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0cd8d0 tx=0x7f65fc09be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b395c38d0c5e7328 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.308+0000 7f65eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.308+0000 7f65eb7fe640 1 -- 172.21.15.57:0/1526493692 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc0b4020 con 0x5584801ea8d0 2024-04-21T08:50:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.308+0000 7f65eb7fe640 1 -- 172.21.15.57:0/1526493692 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f65fc0cdad0 con 0x5584801ea8d0 2024-04-21T08:50:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.309+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6604017de0 con 0x5584801ea8d0 2024-04-21T08:50:02.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.312+0000 7f65eb7fe640 1 -- 172.21.15.57:0/1526493692 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f65fc0cdcd0 con 0x5584801ea8d0 2024-04-21T08:50:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.313+0000 7f6614fc4740 10 client.16449 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:02.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.313+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55847ff2d620 con 0x5584801ea8d0 2024-04-21T08:50:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.320+0000 7f65eb7fe640 1 -- 172.21.15.57:0/1526493692 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6604017de0 con 0x5584801ea8d0 2024-04-21T08:50:02.320 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.320+0000 7f65eb7fe640 1 -- 172.21.15.57:0/1526493692 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f65fc017b50 con 0x5584801ea8d0 2024-04-21T08:50:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.322+0000 7f6614fc4740 10 client.16449 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.322+0000 7f6614fc4740 10 client.16449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:02.322 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.322+0000 7f6614fc4740 20 client.16449 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.322+0000 7f6614fc4740 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480203220 0x558480205630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.322+0000 7f6614fc4740 4 client.16449 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:02.323 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.322+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558480162880 con 0x558480203220 2024-04-21T08:50:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.326+0000 7f660bfff640 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480203220 0x558480205630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.346 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.346+0000 7f660bfff640 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480203220 0x558480205630 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=844a1928284d10cd server_cookie=f2bd41e68371cb41 in_seq=0 out_seq=0 2024-04-21T08:50:02.347 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.347+0000 7f65eb7fe640 10 client.16449 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.358+0000 7f65eb7fe640 1 -- 172.21.15.57:0/1526493692 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558480162880 con 0x558480203220 2024-04-21T08:50:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.358+0000 7f65eb7fe640 10 client.16449 handle_command_reply: tid=0 2024-04-21T08:50:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.359+0000 7f6614fc4740 1 client.16449 shutdown 2024-04-21T08:50:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.359+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 msgr2=0x5584801efee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.359+0000 7f6614fc4740 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 0x5584801efee0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0cd8d0 tx=0x7f65fc09be80 comp rx=0 tx=0).stop 2024-04-21T08:50:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.361+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 shutdown_connections 2024-04-21T08:50:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.361+0000 7f6614fc4740 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480203220 0x558480205630 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.361+0000 7f6614fc4740 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.361+0000 7f6614fc4740 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ea8d0 0x5584801efee0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.361+0000 7f6614fc4740 1 --2- 172.21.15.57:0/1526493692 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801e9b60 0x5584801f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.362 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.361+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 >> 172.21.15.57:0/1526493692 conn(0x5584801b0c90 msgr2=0x5584801e65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.362+0000 7f65eb7fe640 0 client.16449 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:02.363 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 shutdown_connections 2024-04-21T08:50:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/1526493692 wait complete. 2024-04-21T08:50:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.364+0000 7f6614fc4740 20 client.16449 trim_cache size 0 max 16384 2024-04-21T08:50:02.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.397+0000 7f6614fc4740 1 Processor -- start 2024-04-21T08:50:02.399 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.398+0000 7f6614fc4740 1 -- start start 2024-04-21T08:50:02.400 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.398+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801fed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801ea8d0 0x5584801f2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f660bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f660bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55612/0 (socket says 172.21.15.57:55612) 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f660bfff640 1 -- 172.21.15.57:0/3137763891 learned_addr learned my addr 172.21.15.57:0/3137763891 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f6614fc4740 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801f30e0 0x5584801f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1c4a40 con 0x5584801e9b60 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f660b7fe640 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801ea8d0 0x5584801f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1c72a0 con 0x5584801ea8d0 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1c5090 con 0x5584801f30e0 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.399+0000 7f661096b640 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801f30e0 0x5584801f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.401 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.400+0000 7f660bfff640 1 -- 172.21.15.57:0/3137763891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801f30e0 msgr2=0x5584801f2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:02.402 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.400+0000 7f660bfff640 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801f30e0 0x5584801f2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.402 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.400+0000 7f660bfff640 1 -- 172.21.15.57:0/3137763891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801ea8d0 msgr2=0x5584801f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:02.402 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.400+0000 7f660bfff640 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801ea8d0 0x5584801f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.402 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.400+0000 7f660bfff640 1 -- 172.21.15.57:0/3137763891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584801e8e30 con 0x5584801e9b60 2024-04-21T08:50:02.402 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.400+0000 7f660bfff640 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801fed10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6604022fa0 tx=0x7f6604017f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294f00125de02a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.404+0000 7f65f1ffb640 1 -- 172.21.15.57:0/3137763891 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604027040 con 0x5584801e9b60 2024-04-21T08:50:02.404 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.404+0000 7f65f1ffb640 1 -- 172.21.15.57:0/3137763891 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6604035070 con 0x5584801e9b60 2024-04-21T08:50:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.405+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 msgr2=0x5584801fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:02.405 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.405+0000 7f6614fc4740 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801fed10 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6604022fa0 tx=0x7f6604017f90 comp rx=0 tx=0).stop 2024-04-21T08:50:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.405+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 shutdown_connections 2024-04-21T08:50:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.405+0000 7f6614fc4740 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801f30e0 0x5584801f2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.406+0000 7f6614fc4740 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801ea8d0 0x5584801f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.406+0000 7f6614fc4740 1 --2- 172.21.15.57:0/3137763891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801e9b60 0x5584801fed10 secure :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6604022fa0 tx=0x7f6604017f90 comp rx=0 tx=0).stop 2024-04-21T08:50:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.406+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 >> 172.21.15.57:0/3137763891 conn(0x5584801b0c90 msgr2=0x5584801eaf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:02.409 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.409+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 shutdown_connections 2024-04-21T08:50:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.412+0000 7f6614fc4740 1 -- 172.21.15.57:0/3137763891 wait complete. 2024-04-21T08:50:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.412+0000 7f6614fc4740 1 -- start start 2024-04-21T08:50:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 Processor -- start 2024-04-21T08:50:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801fb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801ed160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ed6a0 0x5584801edb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1c5760 con 0x5584801ed6a0 2024-04-21T08:50:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1c6bd0 con 0x5584801f30e0 2024-04-21T08:50:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f660c1c5e30 con 0x5584801ea8d0 2024-04-21T08:50:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.413+0000 7f6614fc4740 10 client.0 _ll_register_callbacks cb 0x7f65fc09c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.415+0000 7f661096b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ed6a0 0x5584801edb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.415+0000 7f661096b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ed6a0 0x5584801edb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:55616/0 (socket says 172.21.15.57:55616) 2024-04-21T08:50:02.418 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.415+0000 7f661096b640 1 -- 172.21.15.57:0/638584734 learned_addr learned my addr 172.21.15.57:0/638584734 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:02.418 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.416+0000 7f660bfff640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801fb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.416+0000 7f660b7fe640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801ed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.419+0000 7f660bfff640 1 -- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 msgr2=0x5584801ed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.419+0000 7f660bfff640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801ed160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.419+0000 7f660bfff640 1 -- 172.21.15.57:0/638584734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ed6a0 msgr2=0x5584801edb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.419+0000 7f660bfff640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ed6a0 0x5584801edb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.419+0000 7f660bfff640 1 -- 172.21.15.57:0/638584734 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584801fbab0 con 0x5584801ea8d0 2024-04-21T08:50:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.423+0000 7f660bfff640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801fb410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6604030000 tx=0x7f6604009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bddf417021084bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.423+0000 7f65eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:50:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.429+0000 7f65eb7fe640 1 -- 172.21.15.57:0/638584734 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660403d020 con 0x5584801ea8d0 2024-04-21T08:50:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.429+0000 7f65eb7fe640 1 -- 172.21.15.57:0/638584734 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6604017d20 con 0x5584801ea8d0 2024-04-21T08:50:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.429+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5584800ee3d0 con 0x5584801ea8d0 2024-04-21T08:50:02.438 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.437+0000 7f65eb7fe640 1 -- 172.21.15.57:0/638584734 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6604027040 con 0x5584801ea8d0 2024-04-21T08:50:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.444+0000 7f6614fc4740 10 client.25837 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.444+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x558480269900 con 0x5584801ea8d0 2024-04-21T08:50:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.447+0000 7f65eb7fe640 1 -- 172.21.15.57:0/638584734 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5584800ee3d0 con 0x5584801ea8d0 2024-04-21T08:50:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.447+0000 7f65eb7fe640 1 -- 172.21.15.57:0/638584734 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f66040040c0 con 0x5584801ea8d0 2024-04-21T08:50:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.449+0000 7f6614fc4740 10 client.25837 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.449+0000 7f6614fc4740 10 client.25837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.449+0000 7f6614fc4740 20 client.25837 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.449+0000 7f6614fc4740 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480269c30 0x55848026c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.449+0000 7f6614fc4740 4 client.25837 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.449+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55848015c2f0 con 0x558480269c30 2024-04-21T08:50:02.452 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.451+0000 7f660b7fe640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480269c30 0x55848026c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.478+0000 7f660b7fe640 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480269c30 0x55848026c020 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ba9df71ec868d4 server_cookie=81d5b4d863531f28 in_seq=0 out_seq=0 2024-04-21T08:50:02.479 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:02.478+0000 7f65eb7fe640 10 client.25837 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:02 smithi129 ceph-mon[31272]: pgmap v141: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 902 KiB/s wr, 3.33k op/s 2024-04-21T08:50:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:02 smithi057 ceph-mon[27968]: pgmap v141: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 902 KiB/s wr, 3.33k op/s 2024-04-21T08:50:02.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:02 smithi136 ceph-mon[31791]: pgmap v141: 129 pgs: 129 active+clean; 337 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 902 KiB/s wr, 3.33k op/s 2024-04-21T08:50:03.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:03 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:50:03.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:03 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:50:03.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:03 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:50:04.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:04 smithi129 ceph-mon[31272]: pgmap v142: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 3.92k op/s 2024-04-21T08:50:04.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:04 smithi057 ceph-mon[27968]: pgmap v142: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 3.92k op/s 2024-04-21T08:50:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:04 smithi136 ceph-mon[31791]: pgmap v142: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.3 MiB/s wr, 3.92k op/s 2024-04-21T08:50:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:06 smithi129 ceph-mon[31272]: pgmap v143: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 818 KiB/s wr, 2.59k op/s 2024-04-21T08:50:06.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:06 smithi057 ceph-mon[27968]: pgmap v143: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 818 KiB/s wr, 2.59k op/s 2024-04-21T08:50:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:06 smithi136 ceph-mon[31791]: pgmap v143: 129 pgs: 129 active+clean; 342 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 818 KiB/s wr, 2.59k op/s 2024-04-21T08:50:08.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:08 smithi129 ceph-mon[31272]: pgmap v144: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 17 MiB/s wr, 3.26k op/s 2024-04-21T08:50:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:08 smithi057 ceph-mon[27968]: pgmap v144: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 17 MiB/s wr, 3.26k op/s 2024-04-21T08:50:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:08 smithi136 ceph-mon[31791]: pgmap v144: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 17 MiB/s wr, 3.26k op/s 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.362+0000 7f65eb7fe640 1 -- 172.21.15.57:0/638584734 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55848015c2f0 con 0x558480269c30 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.362+0000 7f65eb7fe640 10 client.25837 handle_command_reply: tid=0 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.362+0000 7f6614fc4740 1 client.25837 shutdown 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 msgr2=0x5584801fb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801fb410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6604030000 tx=0x7f6604009b80 comp rx=0 tx=0).stop 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 shutdown_connections 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x558480269c30 0x55848026c020 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5584801ed6a0 0x5584801edb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584801f30e0 0x5584801ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.364 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 --2- 172.21.15.57:0/638584734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5584801ea8d0 0x5584801fb410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f65eb7fe640 0 client.25837 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 >> 172.21.15.57:0/638584734 conn(0x5584801b0c90 msgr2=0x5584801ee7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 shutdown_connections 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 1 -- 172.21.15.57:0/638584734 wait complete. 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.363+0000 7f6614fc4740 20 client.25837 trim_cache size 0 max 16384 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:50:10.365 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:50:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f65f40a3ff0 msgr2=0x7f65f40a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f65f40a3ff0 0x7f65f40a64b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6604009f40 tx=0x7f66040094d0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 msgr2=0x7f660c111df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c111df0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00b470 tx=0x7f65fc00b930 comp rx=0 tx=0).stop 2024-04-21T08:50:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 shutdown_connections 2024-04-21T08:50:10.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f65f40a3ff0 0x7f65f40a64b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f660c112330 0x7f660c1127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f660c11b730 0x7f660c111df0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 --2- 172.21.15.57:0/1656193542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f660c110740 0x7f660c118d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:10.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 >> 172.21.15.57:0/1656193542 conn(0x7f660c06d020 msgr2=0x7f660c10e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:10.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 shutdown_connections 2024-04-21T08:50:10.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:10.364+0000 7f66123ce640 1 -- 172.21.15.57:0/1656193542 wait complete. 2024-04-21T08:50:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:10 smithi129 ceph-mon[31272]: pgmap v145: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 1.48k op/s 2024-04-21T08:50:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:10 smithi057 ceph-mon[27968]: pgmap v145: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 1.48k op/s 2024-04-21T08:50:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:10 smithi136 ceph-mon[31791]: pgmap v145: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 1.48k op/s 2024-04-21T08:50:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:12 smithi129 ceph-mon[31272]: pgmap v146: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 1.48k op/s 2024-04-21T08:50:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:12 smithi057 ceph-mon[27968]: pgmap v146: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 1.48k op/s 2024-04-21T08:50:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:12 smithi136 ceph-mon[31791]: pgmap v146: 129 pgs: 129 active+clean; 531 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 17 MiB/s wr, 1.48k op/s 2024-04-21T08:50:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:14 smithi129 ceph-mon[31272]: pgmap v147: 129 pgs: 129 active+clean; 523 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 18 MiB/s wr, 2.45k op/s 2024-04-21T08:50:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:14 smithi057 ceph-mon[27968]: pgmap v147: 129 pgs: 129 active+clean; 523 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 18 MiB/s wr, 2.45k op/s 2024-04-21T08:50:14.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:14 smithi136 ceph-mon[31791]: pgmap v147: 129 pgs: 129 active+clean; 523 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 18 MiB/s wr, 2.45k op/s 2024-04-21T08:50:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:16 smithi129 ceph-mon[31272]: pgmap v148: 129 pgs: 129 active+clean; 523 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 1.71k op/s 2024-04-21T08:50:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:16 smithi057 ceph-mon[27968]: pgmap v148: 129 pgs: 129 active+clean; 523 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 1.71k op/s 2024-04-21T08:50:16.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:16 smithi136 ceph-mon[31791]: pgmap v148: 129 pgs: 129 active+clean; 523 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 1.71k op/s 2024-04-21T08:50:18.578 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:18 smithi129 ceph-mon[31272]: pgmap v149: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 33 MiB/s wr, 1.74k op/s 2024-04-21T08:50:18.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:18 smithi057 ceph-mon[27968]: pgmap v149: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 33 MiB/s wr, 1.74k op/s 2024-04-21T08:50:18.638 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:18 smithi136 ceph-mon[31791]: pgmap v149: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 33 MiB/s wr, 1.74k op/s 2024-04-21T08:50:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:20 smithi129 ceph-mon[31272]: pgmap v150: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 17 MiB/s wr, 1.07k op/s 2024-04-21T08:50:20.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:20 smithi057 ceph-mon[27968]: pgmap v150: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 17 MiB/s wr, 1.07k op/s 2024-04-21T08:50:20.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:20 smithi136 ceph-mon[31791]: pgmap v150: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 17 MiB/s wr, 1.07k op/s 2024-04-21T08:50:20.947 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:20.948 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- 172.21.15.57:0/352964098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 msgr2=0x7fa68410dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- 172.21.15.57:0/352964098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410dca0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009ca0 tx=0x7fa67c02c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- 172.21.15.57:0/352964098 shutdown_connections 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- 172.21.15.57:0/352964098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410dca0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- 172.21.15.57:0/352964098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6841010a0 0x7fa684101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- 172.21.15.57:0/352964098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa684105dd0 0x7fa6841061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- 172.21.15.57:0/352964098 >> 172.21.15.57:0/352964098 conn(0x7fa684077c40 msgr2=0x7fa684078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- 172.21.15.57:0/352964098 shutdown_connections 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- 172.21.15.57:0/352964098 wait complete. 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 Processor -- start 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- start start 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6841010a0 0x7fa68410e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa684105dd0 0x7fa68410f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841122d0 con 0x7fa684101a40 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa684112150 con 0x7fa6841010a0 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.066+0000 7fa68afb2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa684112450 con 0x7fa684105dd0 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37960/0 (socket says 172.21.15.57:37960) 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 -- 172.21.15.57:0/1033709820 learned_addr learned my addr 172.21.15.57:0/1033709820 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa68954f640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa684105dd0 0x7fa68410f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa688d4e640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6841010a0 0x7fa68410e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 -- 172.21.15.57:0/1033709820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa684105dd0 msgr2=0x7fa68410f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa684105dd0 0x7fa68410f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 -- 172.21.15.57:0/1033709820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6841010a0 msgr2=0x7fa68410e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6841010a0 0x7fa68410e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 -- 172.21.15.57:0/1033709820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6841a6bd0 con 0x7fa684101a40 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa683fff640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410ecf0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa67000d950 tx=0x7fa67000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa681ffb640 1 -- 172.21.15.57:0/1033709820 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670017070 con 0x7fa684101a40 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa681ffb640 1 -- 172.21.15.57:0/1033709820 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa670009c30 con 0x7fa684101a40 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6841a6ec0 con 0x7fa684101a40 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.067+0000 7fa681ffb640 1 -- 172.21.15.57:0/1033709820 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67000f620 con 0x7fa684101a40 2024-04-21T08:50:21.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.068+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6841a7380 con 0x7fa684101a40 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa681ffb640 1 -- 172.21.15.57:0/1033709820 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa670010460 con 0x7fa684101a40 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 Processor -- start 2024-04-21T08:50:21.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 -- start start 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa681ffb640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa6740a3d50 0x7fa6740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e118104960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104ea0 0x55e11810d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 0x55e11810fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e1180691e0 con 0x55e11810da80 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e118006390 con 0x55e118104ea0 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68dba1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e1180651a0 con 0x55e118104580 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa681ffb640 1 -- 172.21.15.57:0/1033709820 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa6700d06b0 con 0x7fa684101a40 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68954f640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa6740a3d50 0x7fa6740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.070+0000 7fa68954f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 0x55e11810fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 0x55e11810fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37970/0 (socket says 172.21.15.57:37970) 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 -- 172.21.15.57:0/633442027 learned_addr learned my addr 172.21.15.57:0/633442027 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa688d4e640 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e118104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa683fff640 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104ea0 0x55e11810d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 -- 172.21.15.57:0/633442027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 msgr2=0x55e118104960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e118104960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 -- 172.21.15.57:0/633442027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104ea0 msgr2=0x55e11810d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104ea0 0x55e11810d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 -- 172.21.15.57:0/633442027 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e117e457d0 con 0x55e11810da80 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa6740a3d50 0x7fa6740a6210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa67c037040 tx=0x7fa67c009960 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa68954f640 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 0x55e11810fe70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa67c036000 tx=0x7fa67c047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e32d6cf5e9fffc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa66effd640 1 -- 172.21.15.57:0/633442027 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67c039040 con 0x55e11810da80 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa66effd640 1 -- 172.21.15.57:0/633442027 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa67c003e20 con 0x55e11810da80 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.071+0000 7fa66effd640 1 -- 172.21.15.57:0/633442027 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67c0046f0 con 0x55e11810da80 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 -- 172.21.15.57:0/633442027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 msgr2=0x55e11810fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 0x55e11810fe70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa67c036000 tx=0x7fa67c047000 comp rx=0 tx=0).stop 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 -- 172.21.15.57:0/633442027 shutdown_connections 2024-04-21T08:50:21.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e11810da80 0x55e11810fe70 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104ea0 0x55e11810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 --2- 172.21.15.57:0/633442027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e118104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 -- 172.21.15.57:0/633442027 >> 172.21.15.57:0/633442027 conn(0x55e1180fecb0 msgr2=0x55e1180ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 -- 172.21.15.57:0/633442027 shutdown_connections 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 -- 172.21.15.57:0/633442027 wait complete. 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.072+0000 7fa68dba1740 1 -- start start 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 Processor -- start 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e11810b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e11810b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e118110810 con 0x55e118104ea0 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e1180222f0 con 0x55e11810da80 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e118110990 con 0x55e118104580 2024-04-21T08:50:21.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68dba1740 10 client.0 _ll_register_callbacks cb 0x55e11802f580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e11810b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e11810b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37972/0 (socket says 172.21.15.57:37972) 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 -- 172.21.15.57:0/1845935333 learned_addr learned my addr 172.21.15.57:0/1845935333 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa688d4e640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68954f640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e11810b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 -- 172.21.15.57:0/1845935333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 msgr2=0x55e11810aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 -- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 msgr2=0x55e11810b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e11810b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa683fff640 1 -- 172.21.15.57:0/1845935333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e117d7b3f0 con 0x55e118104ea0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.073+0000 7fa68954f640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e11810b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa683fff640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e11810b020 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa670017040 tx=0x7fa67009bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc8c1504cf4d46b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa663fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa663fff640 1 -- 172.21.15.57:0/1845935333 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670013050 con 0x55e118104ea0 2024-04-21T08:50:21.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa663fff640 1 -- 172.21.15.57:0/1845935333 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa6700139e0 con 0x55e118104ea0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa67c003e20 con 0x55e118104ea0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa663fff640 1 -- 172.21.15.57:0/1845935333 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa6700ba050 con 0x55e118104ea0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 10 client.16467 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e117d3b0f0 con 0x55e118104ea0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa663fff640 1 -- 172.21.15.57:0/1845935333 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa67c003e20 con 0x55e118104ea0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa663fff640 1 -- 172.21.15.57:0/1845935333 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa6700cdd20 con 0x55e118104ea0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 10 client.16467 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 20 client.16467 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e11811da50 0x55e11811fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 4 client.16467 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.074+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e118076c00 con 0x55e11811da50 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.075+0000 7fa688d4e640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e11811da50 0x55e11811fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.075+0000 7fa663fff640 10 client.16467 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.075+0000 7fa688d4e640 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e11811da50 0x55e11811fed0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8546308971557ef server_cookie=9eacde8bd3dc66c6 in_seq=0 out_seq=0 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa663fff640 1 -- 172.21.15.57:0/1845935333 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e118076c00 con 0x55e11811da50 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa663fff640 10 client.16467 handle_command_reply: tid=0 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 client.16467 shutdown 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 msgr2=0x55e11810b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e11810b020 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fa670017040 tx=0x7fa67009bdc0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 shutdown_connections 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e11811da50 0x55e11811fed0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e11810b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa663fff640 0 client.16467 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e11810b020 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1845935333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 >> 172.21.15.57:0/1845935333 conn(0x55e1180fecb0 msgr2=0x55e118100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 shutdown_connections 2024-04-21T08:50:21.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 1 -- 172.21.15.57:0/1845935333 wait complete. 2024-04-21T08:50:21.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.076+0000 7fa68dba1740 20 client.16467 trim_cache size 0 max 16384 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 Processor -- start 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 -- start start 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118119720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e118109c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841af470 con 0x55e118104ea0 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b1cd0 con 0x55e11810da80 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68dba1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841afac0 con 0x55e118104580 2024-04-21T08:50:21.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118119720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118119720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37984/0 (socket says 172.21.15.57:37984) 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa683fff640 1 -- 172.21.15.57:0/1044982143 learned_addr learned my addr 172.21.15.57:0/1044982143 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa68954f640 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e118109c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.094+0000 7fa688d4e640 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa683fff640 1 -- 172.21.15.57:0/1044982143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 msgr2=0x55e11810d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa683fff640 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa683fff640 1 -- 172.21.15.57:0/1044982143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 msgr2=0x55e118109c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa683fff640 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e118109c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa683fff640 1 -- 172.21.15.57:0/1044982143 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e11810fe70 con 0x55e118104ea0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa683fff640 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118119720 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa6700cdb50 tx=0x7fa67009a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ae9a3c6b695401 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa66e7fc640 1 -- 172.21.15.57:0/1044982143 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670098bb0 con 0x55e118104ea0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa66e7fc640 1 -- 172.21.15.57:0/1044982143 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa670013050 con 0x55e118104ea0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa66e7fc640 1 -- 172.21.15.57:0/1044982143 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa67009bee0 con 0x55e118104ea0 2024-04-21T08:50:21.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa68dba1740 1 -- 172.21.15.57:0/1044982143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 msgr2=0x55e118119720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.095+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118119720 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fa6700cdb50 tx=0x7fa67009a8b0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- 172.21.15.57:0/1044982143 shutdown_connections 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e11810da80 0x55e118109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118119720 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 --2- 172.21.15.57:0/1044982143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e118104580 0x55e11810d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- 172.21.15.57:0/1044982143 >> 172.21.15.57:0/1044982143 conn(0x55e1180fecb0 msgr2=0x55e1181000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- 172.21.15.57:0/1044982143 shutdown_connections 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- 172.21.15.57:0/1044982143 wait complete. 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- start start 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 Processor -- start 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104580 0x55e118111680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118111bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e11810da80 0x55e118112100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b0190 con 0x55e118104ea0 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b1600 con 0x55e118104580 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa6841b0860 con 0x55e11810da80 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa68dba1740 10 client.0 _ll_register_callbacks cb 0x7fa670098640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118111bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa683fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118111bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37998/0 (socket says 172.21.15.57:37998) 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.096+0000 7fa683fff640 1 -- 172.21.15.57:0/726623949 learned_addr learned my addr 172.21.15.57:0/726623949 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa688d4e640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104580 0x55e118111680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa68954f640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e11810da80 0x55e118112100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa683fff640 1 -- 172.21.15.57:0/726623949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e11810da80 msgr2=0x55e118112100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa683fff640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e11810da80 0x55e118112100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa683fff640 1 -- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104580 msgr2=0x55e118111680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa683fff640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104580 0x55e118111680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa683fff640 1 -- 172.21.15.57:0/726623949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e118183040 con 0x55e118104ea0 2024-04-21T08:50:21.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa683fff640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118111bc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa670012040 tx=0x7fa6700a06a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4203ccdd06ef9932 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa663fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa663fff640 1 -- 172.21.15.57:0/726623949 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670098bb0 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa663fff640 1 -- 172.21.15.57:0/726623949 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa67009b7f0 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa670013050 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa663fff640 1 -- 172.21.15.57:0/726623949 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa67009c3a0 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa68dba1740 10 client.16473 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e118184330 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.097+0000 7fa663fff640 1 -- 172.21.15.57:0/726623949 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa670013050 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa663fff640 1 -- 172.21.15.57:0/726623949 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa670010750 con 0x55e118104ea0 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa68dba1740 10 client.16473 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa68dba1740 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa68dba1740 20 client.16473 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa68dba1740 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e118184680 0x55e118186aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa68dba1740 4 client.16473 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:21.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e118070a70 con 0x55e118184680 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa688d4e640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e118184680 0x55e118186aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa688d4e640 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e118184680 0x55e118186aa0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7486a0a6934bf51 server_cookie=868ae9f6f9e35bf3 in_seq=0 out_seq=0 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.098+0000 7fa663fff640 10 client.16473 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa663fff640 1 -- 172.21.15.57:0/726623949 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e118070a70 con 0x55e118184680 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa663fff640 10 client.16473 handle_command_reply: tid=0 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 client.16473 shutdown 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 msgr2=0x55e118111bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118111bc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa670012040 tx=0x7fa6700a06a0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 shutdown_connections 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e118184680 0x55e118186aa0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e11810da80 0x55e118112100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e118104ea0 0x55e118111bc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa663fff640 0 client.16473 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 --2- 172.21.15.57:0/726623949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e118104580 0x55e118111680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 >> 172.21.15.57:0/726623949 conn(0x55e1180fecb0 msgr2=0x55e1181000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 shutdown_connections 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 1 -- 172.21.15.57:0/726623949 wait complete. 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68dba1740 20 client.16473 trim_cache size 0 max 16384 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:50:21.106 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa6740a3d50 msgr2=0x7fa6740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68afb2640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa6740a3d50 0x7fa6740a6210 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa67c037040 tx=0x7fa67c009960 comp rx=0 tx=0).stop 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 msgr2=0x7fa68410ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.099+0000 7fa68afb2640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410ecf0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa67000d950 tx=0x7fa67000de10 comp rx=0 tx=0).stop 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 shutdown_connections 2024-04-21T08:50:21.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa6740a3d50 0x7fa6740a6210 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa684105dd0 0x7fa68410f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa684101a40 0x7fa68410ecf0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 --2- 172.21.15.57:0/1033709820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6841010a0 0x7fa68410e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 >> 172.21.15.57:0/1033709820 conn(0x7fa684077c40 msgr2=0x7fa6840fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 shutdown_connections 2024-04-21T08:50:21.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.100+0000 7fa68afb2640 1 -- 172.21.15.57:0/1033709820 wait complete. 2024-04-21T08:50:21.110 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:50:21.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 -- 172.21.15.57:0/4100977036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54106f10 msgr2=0x7f4a541072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 --2- 172.21.15.57:0/4100977036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54106f10 0x7f4a541072f0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009900 tx=0x7f4a4402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 -- 172.21.15.57:0/4100977036 shutdown_connections 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 --2- 172.21.15.57:0/4100977036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 0x7f4a5410dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 --2- 172.21.15.57:0/4100977036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54101130 0x7f4a54101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 --2- 172.21.15.57:0/4100977036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54106f10 0x7f4a541072f0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 -- 172.21.15.57:0/4100977036 >> 172.21.15.57:0/4100977036 conn(0x7f4a54077e30 msgr2=0x7f4a54078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 -- 172.21.15.57:0/4100977036 shutdown_connections 2024-04-21T08:50:21.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 -- 172.21.15.57:0/4100977036 wait complete. 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.214+0000 7f4a58f67640 1 Processor -- start 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 -- start start 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 0x7f4a54197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 0x7f4a54197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54106f10 0x7f4a5419bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a541110c0 con 0x7f4a54101130 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54110f40 con 0x7f4a54101ad0 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a58f67640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54111240 con 0x7f4a54106f10 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 0x7f4a54197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 0x7f4a54197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38010/0 (socket says 172.21.15.57:38010) 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 -- 172.21.15.57:0/718110836 learned_addr learned my addr 172.21.15.57:0/718110836 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a52d9d640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54106f10 0x7f4a5419bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a49d9b640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 0x7f4a54197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 -- 172.21.15.57:0/718110836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54106f10 msgr2=0x7f4a5419bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.217 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54106f10 0x7f4a5419bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 -- 172.21.15.57:0/718110836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 msgr2=0x7f4a54197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 0x7f4a54197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a5259c640 1 -- 172.21.15.57:0/718110836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5419c610 con 0x7f4a54101130 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a52d9d640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54106f10 0x7f4a5419bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.215+0000 7f4a49d9b640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 0x7f4a54197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.216+0000 7f4a5259c640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 0x7f4a54197620 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005940 tx=0x7f4a4402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.216+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44043020 con 0x7f4a54101130 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.216+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a4402cbe0 con 0x7f4a54101130 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.216+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5419c8a0 con 0x7f4a54101130 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.216+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44003bf0 con 0x7f4a54101130 2024-04-21T08:50:21.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.216+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a541a4190 con 0x7f4a54101130 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.217+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a54069230 con 0x7f4a54101130 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.218+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a44046020 con 0x7f4a54101130 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.219+0000 7f4a4bfff640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a400a3d20 0x7f4a400a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.219+0000 7f4a49d9b640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a400a3d20 0x7f4a400a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.219+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f4a440f2d50 con 0x7f4a54101130 2024-04-21T08:50:21.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.219+0000 7f4a49d9b640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a400a3d20 0x7f4a400a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0058c0 tx=0x7f4a3c005850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.224+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a440f6030 con 0x7f4a54101130 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.504+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a54109370 con 0x7f4a54101130 2024-04-21T08:50:21.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.505+0000 7f4a4bfff640 1 -- 172.21.15.57:0/718110836 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f4a4404d020 con 0x7f4a54101130 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:50:21.506 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a400a3d20 msgr2=0x7f4a400a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a400a3d20 0x7f4a400a61e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0058c0 tx=0x7f4a3c005850 comp rx=0 tx=0).stop 2024-04-21T08:50:21.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 msgr2=0x7f4a54197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 0x7f4a54197620 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4a44005940 tx=0x7f4a4402c7e0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 shutdown_connections 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a400a3d20 0x7f4a400a61e0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0058c0 tx=0x7f4a3c005850 comp rx=0 tx=0).stop 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a54106f10 0x7f4a5419bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a54101ad0 0x7f4a54197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 --2- 172.21.15.57:0/718110836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a54101130 0x7f4a54197620 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 >> 172.21.15.57:0/718110836 conn(0x7f4a54077e30 msgr2=0x7f4a540ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.510+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 shutdown_connections 2024-04-21T08:50:21.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.511+0000 7f4a58f67640 1 -- 172.21.15.57:0/718110836 wait complete. 2024-04-21T08:50:21.535 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:50:21.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.717+0000 7f5e58b8f640 1 -- 172.21.15.57:0/2446236073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e54106160 msgr2=0x7f5e5410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.717+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/2446236073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e54106160 0x7f5e5410dd30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009ca0 tx=0x7f5e4c02c100 comp rx=0 tx=0).stop 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.718+0000 7f5e58b8f640 1 -- 172.21.15.57:0/2446236073 shutdown_connections 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.718+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/2446236073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e54106160 0x7f5e5410dd30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.718+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/2446236073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101a40 0x7f5e54105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.718+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/2446236073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e54101090 0x7f5e54101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.718+0000 7f5e58b8f640 1 -- 172.21.15.57:0/2446236073 >> 172.21.15.57:0/2446236073 conn(0x7f5e54077f50 msgr2=0x7f5e54078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.719+0000 7f5e58b8f640 1 -- 172.21.15.57:0/2446236073 shutdown_connections 2024-04-21T08:50:21.719 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.719+0000 7f5e58b8f640 1 -- 172.21.15.57:0/2446236073 wait complete. 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.721+0000 7f5e58b8f640 1 Processor -- start 2024-04-21T08:50:21.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.721+0000 7f5e58b8f640 1 -- start start 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.722+0000 7f5e58b8f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 0x7f5e541a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.722+0000 7f5e58b8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e54106160 0x7f5e541a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.722+0000 7f5e58b8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e5419a820 0x7f5e5419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.722+0000 7f5e58b8f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54110010 con 0x7f5e54106160 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.722+0000 7f5e58b8f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e5410fe90 con 0x7f5e5419a820 2024-04-21T08:50:21.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.722+0000 7f5e58b8f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54110190 con 0x7f5e54101090 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 0x7f5e541a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 0x7f5e541a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47416/0 (socket says 172.21.15.57:47416) 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 -- 172.21.15.57:0/154407951 learned_addr learned my addr 172.21.15.57:0/154407951 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 -- 172.21.15.57:0/154407951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e5419a820 msgr2=0x7f5e5419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e5419a820 0x7f5e5419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 -- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e54106160 msgr2=0x7f5e541a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e54106160 0x7f5e541a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.723+0000 7f5e5259c640 1 -- 172.21.15.57:0/154407951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5419b400 con 0x7f5e54101090 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.724+0000 7f5e5259c640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 0x7f5e541a0690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400e9f0 tx=0x7f5e4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.724+0000 7f5e437fe640 1 -- 172.21.15.57:0/154407951 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400cd80 con 0x7f5e54101090 2024-04-21T08:50:21.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.724+0000 7f5e437fe640 1 -- 172.21.15.57:0/154407951 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e440108e0 con 0x7f5e54101090 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.724+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e541a74e0 con 0x7f5e54101090 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.724+0000 7f5e437fe640 1 -- 172.21.15.57:0/154407951 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44018860 con 0x7f5e54101090 2024-04-21T08:50:21.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.724+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e541a7890 con 0x7f5e54101090 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.727+0000 7f5e5b77f740 1 Processor -- start 2024-04-21T08:50:21.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.727+0000 7f5e5b77f740 1 -- start start 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f7c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f85430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f87e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e335ee11f0 con 0x55e335f7c4d0 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e335e7e4b0 con 0x55e335f7ce80 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e335edd1b0 con 0x55e335f85a00 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e437fe640 1 -- 172.21.15.57:0/154407951 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5e44010430 con 0x7f5e54101090 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e52d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f7c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f85430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e52d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f7c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38054/0 (socket says 172.21.15.57:38054) 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e52d9d640 1 -- 172.21.15.57:0/3563452609 learned_addr learned my addr 172.21.15.57:0/3563452609 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.729 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f87e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5259c640 1 -- 172.21.15.57:0/3563452609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 msgr2=0x55e335f87e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5259c640 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f87e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5259c640 1 -- 172.21.15.57:0/3563452609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 msgr2=0x55e335f7c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5259c640 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f7c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.728+0000 7f5e5259c640 1 -- 172.21.15.57:0/3563452609 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e335e701b0 con 0x55e335f7ce80 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e437fe640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e5259c640 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f85430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e440ce2d0 tx=0x7f5e440d9570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ffa2d46238261e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e52d9d640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e437fe640 1 -- 172.21.15.57:0/154407951 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5e440d0f40 con 0x7f5e54101090 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e40ff9640 1 -- 172.21.15.57:0/3563452609 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44098dc0 con 0x55e335f7ce80 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e40ff9640 1 -- 172.21.15.57:0/3563452609 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4409de50 con 0x55e335f7ce80 2024-04-21T08:50:21.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.729+0000 7f5e40ff9640 1 -- 172.21.15.57:0/3563452609 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e440d4a70 con 0x55e335f7ce80 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 -- 172.21.15.57:0/3563452609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 msgr2=0x55e335f85430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f85430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5e440ce2d0 tx=0x7f5e440d9570 comp rx=0 tx=0).stop 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e52d9d640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c032460 tx=0x7f5e4c005dc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 -- 172.21.15.57:0/3563452609 shutdown_connections 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f87e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f85430 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/3563452609 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f7c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 -- 172.21.15.57:0/3563452609 >> 172.21.15.57:0/3563452609 conn(0x55e335f76ba0 msgr2=0x55e335f76f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 -- 172.21.15.57:0/3563452609 shutdown_connections 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.730+0000 7f5e5b77f740 1 -- 172.21.15.57:0/3563452609 wait complete. 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.731+0000 7f5e5b77f740 1 -- start start 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.731+0000 7f5e5b77f740 1 Processor -- start 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.731+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f85250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.731+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f825e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.731+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f82b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.731+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f85250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.735 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f85250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38068/0 (socket says 172.21.15.57:38068) 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 -- 172.21.15.57:0/1562842988 learned_addr learned my addr 172.21.15.57:0/1562842988 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e335f91180 con 0x55e335f7c4d0 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e335e9a500 con 0x55e335f7ce80 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e335f91300 con 0x55e335f85a00 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5b77f740 10 client.0 _ll_register_callbacks cb 0x55e335e68b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:21.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e52d9d640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f82b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 -- 172.21.15.57:0/1562842988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 msgr2=0x55e335f82b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f82b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 -- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 msgr2=0x55e335f825e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f825e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e335bf33f0 con 0x55e335f7c4d0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e5259c640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f85250 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5e440994f0 tx=0x7f5e44009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f7a3defa744e00 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e39ffb640 1 -- 172.21.15.57:0/1562842988 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4409c480 con 0x55e335f7c4d0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.732+0000 7f5e39ffb640 1 -- 172.21.15.57:0/1562842988 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5e44010a90 con 0x55e335f7c4d0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4409de50 con 0x55e335f7c4d0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e39ffb640 1 -- 172.21.15.57:0/1562842988 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5e4409c660 con 0x55e335f7c4d0 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 10 client.16488 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:21.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e335cbd7d0 con 0x55e335f7c4d0 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e39ffb640 1 -- 172.21.15.57:0/1562842988 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4409de50 con 0x55e335f7c4d0 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e39ffb640 1 -- 172.21.15.57:0/1562842988 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5e440bbce0 con 0x55e335f7c4d0 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 10 client.16488 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 20 client.16488 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f9e1a0 0x55e335fa0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 4 client.16488 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.733+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e335e80930 con 0x55e335f9e1a0 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.734+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f9e1a0 0x55e335fa0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.734+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f9e1a0 0x55e335fa0590 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8975d5210b1fd3 server_cookie=f8af6314a26a344c in_seq=0 out_seq=0 2024-04-21T08:50:21.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.734+0000 7f5e39ffb640 10 client.16488 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.736+0000 7f5e39ffb640 1 -- 172.21.15.57:0/1562842988 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e335e80930 con 0x55e335f9e1a0 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.736+0000 7f5e39ffb640 10 client.16488 handle_command_reply: tid=0 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.736+0000 7f5e5b77f740 1 client.16488 shutdown 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 msgr2=0x55e335f85250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f85250 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f5e440994f0 tx=0x7f5e44009700 comp rx=0 tx=0).stop 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 shutdown_connections 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f9e1a0 0x55e335fa0590 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f85a00 0x55e335f82b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f7ce80 0x55e335f825e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/1562842988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f85250 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 >> 172.21.15.57:0/1562842988 conn(0x55e335f76ba0 msgr2=0x55e335f788f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e39ffb640 0 client.16488 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.740 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 shutdown_connections 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 1 -- 172.21.15.57:0/1562842988 wait complete. 2024-04-21T08:50:21.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.737+0000 7f5e5b77f740 20 client.16488 trim_cache size 0 max 16384 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 Processor -- start 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 -- start start 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335f99c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f84d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7ce80 0x55e335f85250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541ab230 con 0x55e335f7c4d0 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541ada50 con 0x55e335f76ba0 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541ab880 con 0x55e335f7ce80 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f84d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f84d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38080/0 (socket says 172.21.15.57:38080) 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 -- 172.21.15.57:0/4100649968 learned_addr learned my addr 172.21.15.57:0/4100649968 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e5259c640 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335f99c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 -- 172.21.15.57:0/4100649968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7ce80 msgr2=0x55e335f85250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7ce80 0x55e335f85250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 -- 172.21.15.57:0/4100649968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 msgr2=0x55e335f99c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335f99c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.757 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.752+0000 7f5e51d9b640 1 -- 172.21.15.57:0/4100649968 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e335f7c0b0 con 0x55e335f7c4d0 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f84d10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5e48000c00 tx=0x7f5e4800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c2952f8a992052 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e3bfff640 1 -- 172.21.15.57:0/4100649968 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e480192d0 con 0x55e335f7c4d0 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e3bfff640 1 -- 172.21.15.57:0/4100649968 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4801d030 con 0x55e335f7c4d0 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e3bfff640 1 -- 172.21.15.57:0/4100649968 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48016dc0 con 0x55e335f7c4d0 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4100649968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 msgr2=0x55e335f84d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f84d10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5e48000c00 tx=0x7f5e4800b870 comp rx=0 tx=0).stop 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4100649968 shutdown_connections 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7ce80 0x55e335f85250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7c4d0 0x55e335f84d10 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4100649968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335f99c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.753+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4100649968 >> 172.21.15.57:0/4100649968 conn(0x55e335f82c80 msgr2=0x55e335f99550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4100649968 shutdown_connections 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4100649968 wait complete. 2024-04-21T08:50:21.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 -- start start 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 Processor -- start 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335fa3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7c4d0 0x55e335fa09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7ce80 0x55e335fa0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541abf10 con 0x55e335f7ce80 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541ad380 con 0x55e335f76ba0 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541ac5e0 con 0x55e335f7c4d0 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335fa3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335fa3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55916/0 (socket says 172.21.15.57:55916) 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5259c640 1 -- 172.21.15.57:0/4178171812 learned_addr learned my addr 172.21.15.57:0/4178171812 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e5b77f740 10 client.0 _ll_register_callbacks cb 0x7f5e4400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e52d9d640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7ce80 0x55e335fa0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.754+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7c4d0 0x55e335fa09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5259c640 1 -- 172.21.15.57:0/4178171812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7c4d0 msgr2=0x55e335fa09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5259c640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7c4d0 0x55e335fa09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5259c640 1 -- 172.21.15.57:0/4178171812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7ce80 msgr2=0x55e335fa0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5259c640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7ce80 0x55e335fa0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5259c640 1 -- 172.21.15.57:0/4178171812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e335fa17c0 con 0x55e335f76ba0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e52d9d640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7ce80 0x55e335fa0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5259c640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335fa3660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e44098f80 tx=0x7f5e440a1050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2199a2d854be4033 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e39ffb640 1 -- 172.21.15.57:0/4178171812 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4409c8f0 con 0x55e335f76ba0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4801d030 con 0x55e335f76ba0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e39ffb640 1 -- 172.21.15.57:0/4178171812 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f5e440a5300 con 0x55e335f76ba0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e39ffb640 1 -- 172.21.15.57:0/4178171812 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5e4409c480 con 0x55e335f76ba0 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5b77f740 10 client.25967 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:21.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e336004430 con 0x55e335f76ba0 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.755+0000 7f5e39ffb640 1 -- 172.21.15.57:0/4178171812 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e4801d030 con 0x55e335f76ba0 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e39ffb640 1 -- 172.21.15.57:0/4178171812 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f5e44079020 con 0x55e335f76ba0 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e5b77f740 10 client.25967 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e5b77f740 10 client.25967 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e5b77f740 20 client.25967 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f85a00 0x55e336006910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e5b77f740 4 client.25967 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e335eeec10 con 0x55e335f85a00 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f85a00 0x55e336006910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e39ffb640 10 client.25967 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.756+0000 7f5e51d9b640 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f85a00 0x55e336006910 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f7a97c2b519bdd8 server_cookie=2f788945e717343a in_seq=0 out_seq=0 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.762+0000 7f5e39ffb640 1 -- 172.21.15.57:0/4178171812 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e335eeec10 con 0x55e335f85a00 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.762+0000 7f5e39ffb640 10 client.25967 handle_command_reply: tid=0 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.762+0000 7f5e5b77f740 1 client.25967 shutdown 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 msgr2=0x55e335fa3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335fa3660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e44098f80 tx=0x7f5e440a1050 comp rx=0 tx=0).stop 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 shutdown_connections 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e335f85a00 0x55e336006910 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e335f7ce80 0x55e335fa0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e39ffb640 0 client.25967 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e335f7c4d0 0x55e335fa09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 --2- 172.21.15.57:0/4178171812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e335f76ba0 0x55e335fa3660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 >> 172.21.15.57:0/4178171812 conn(0x55e335f82c80 msgr2=0x55e335f99de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 shutdown_connections 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 1 -- 172.21.15.57:0/4178171812 wait complete. 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e5b77f740 20 client.25967 trim_cache size 0 max 16384 2024-04-21T08:50:21.764 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5e3c0a3e20 msgr2=0x7f5e3c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c032460 tx=0x7f5e4c005dc0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 msgr2=0x7f5e541a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 0x7f5e541a0690 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400e9f0 tx=0x7f5e4400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 shutdown_connections 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e5419a820 0x7f5e5419acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5e54106160 0x7f5e541a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 --2- 172.21.15.57:0/154407951 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e54101090 0x7f5e541a0690 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 >> 172.21.15.57:0/154407951 conn(0x7f5e54077f50 msgr2=0x7f5e540ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 shutdown_connections 2024-04-21T08:50:21.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:21.763+0000 7f5e58b8f640 1 -- 172.21.15.57:0/154407951 wait complete. 2024-04-21T08:50:22.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:22 smithi129 ceph-mon[31272]: pgmap v151: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 17 MiB/s wr, 1.07k op/s 2024-04-21T08:50:22.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:22 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/718110836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:22.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:22 smithi057 ceph-mon[27968]: pgmap v151: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 17 MiB/s wr, 1.07k op/s 2024-04-21T08:50:22.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:22 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/718110836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:22.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:22 smithi136 ceph-mon[31791]: pgmap v151: 129 pgs: 129 active+clean; 715 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 17 MiB/s wr, 1.07k op/s 2024-04-21T08:50:22.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:22 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/718110836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:50:22.774 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:50:22.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:50:22.775 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:50:22.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.891+0000 7f00909f4640 1 -- 172.21.15.57:0/101947027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c107270 msgr2=0x7f008c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.891+0000 7f00909f4640 1 --2- 172.21.15.57:0/101947027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c107270 0x7f008c10ee40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0084009ca0 tx=0x7f008402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 -- 172.21.15.57:0/101947027 shutdown_connections 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 --2- 172.21.15.57:0/101947027 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c107270 0x7f008c10ee40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 --2- 172.21.15.57:0/101947027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f008c06b1f0 0x7f008c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 --2- 172.21.15.57:0/101947027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c06a840 0x7f008c06ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 -- 172.21.15.57:0/101947027 >> 172.21.15.57:0/101947027 conn(0x7f008c077eb0 msgr2=0x7f008c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 -- 172.21.15.57:0/101947027 shutdown_connections 2024-04-21T08:50:22.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 -- 172.21.15.57:0/101947027 wait complete. 2024-04-21T08:50:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.892+0000 7f00909f4640 1 Processor -- start 2024-04-21T08:50:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 -- start start 2024-04-21T08:50:22.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 0x7f008c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c107270 0x7f008c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f008c19cfd0 0x7f008c19d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1121b0 con 0x7f008c06a840 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f008c112030 con 0x7f008c107270 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f00909f4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f008c112330 con 0x7f008c19cfd0 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 0x7f008c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 0x7f008c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45926/0 (socket says 172.21.15.57:45926) 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 -- 172.21.15.57:0/870018952 learned_addr learned my addr 172.21.15.57:0/870018952 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f0089d9b640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c107270 0x7f008c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008ad9d640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f008c19cfd0 0x7f008c19d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 -- 172.21.15.57:0/870018952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f008c19cfd0 msgr2=0x7f008c19d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f008c19cfd0 0x7f008c19d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 -- 172.21.15.57:0/870018952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c107270 msgr2=0x7f008c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c107270 0x7f008c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 -- 172.21.15.57:0/870018952 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c19dac0 con 0x7f008c06a840 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f0089d9b640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c107270 0x7f008c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.893+0000 7f008a59c640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 0x7f008c198710 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f007c00d950 tx=0x7f007c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.894+0000 7f007b7fe640 1 -- 172.21.15.57:0/870018952 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c017070 con 0x7f008c06a840 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.894+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008c19ddb0 con 0x7f008c06a840 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.894+0000 7f007b7fe640 1 -- 172.21.15.57:0/870018952 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f007c009c30 con 0x7f008c06a840 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.894+0000 7f007b7fe640 1 -- 172.21.15.57:0/870018952 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c00f620 con 0x7f008c06a840 2024-04-21T08:50:22.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.894+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008c1a55a0 con 0x7f008c06a840 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.896+0000 7f007b7fe640 1 -- 172.21.15.57:0/870018952 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f007c010460 con 0x7f008c06a840 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 Processor -- start 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 -- start start 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce7335acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce73363840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce73366260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ce732bef00 con 0x55ce7335a8e0 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce7325c590 con 0x55ce73363e10 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f00935e9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ce732baec0 con 0x55ce7335b290 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce7335acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce7335acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45930/0 (socket says 172.21.15.57:45930) 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f008ad9d640 1 -- 172.21.15.57:0/1946344418 learned_addr learned my addr 172.21.15.57:0/1946344418 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f0089d9b640 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce73366260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f008a59c640 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce73363840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.897+0000 7f008ad9d640 1 -- 172.21.15.57:0/1946344418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 msgr2=0x55ce73363840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f008ad9d640 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce73363840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f008ad9d640 1 -- 172.21.15.57:0/1946344418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 msgr2=0x55ce73366260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f008ad9d640 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce73366260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f008ad9d640 1 -- 172.21.15.57:0/1946344418 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce730997e0 con 0x55ce7335a8e0 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f007b7fe640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f00740a3d20 0x7f00740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f0089d9b640 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce73366260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f008a59c640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f00740a3d20 0x7f00740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f008ad9d640 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce7335acc0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0084005e30 tx=0x7f0084032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498c5cb996070c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f007b7fe640 1 -- 172.21.15.57:0/870018952 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f007c0d0850 con 0x7f008c06a840 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f0078ff9640 1 -- 172.21.15.57:0/1946344418 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084039050 con 0x55ce7335a8e0 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f0078ff9640 1 -- 172.21.15.57:0/1946344418 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0084045070 con 0x55ce7335a8e0 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.898+0000 7f0078ff9640 1 -- 172.21.15.57:0/1946344418 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084004040 con 0x55ce7335a8e0 2024-04-21T08:50:22.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 -- 172.21.15.57:0/1946344418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 msgr2=0x55ce7335acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f008a59c640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f00740a3d20 0x7f00740a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f007c0cdcf0 tx=0x7f007c010cb0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce7335acc0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0084005e30 tx=0x7f0084032420 comp rx=0 tx=0).stop 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 -- 172.21.15.57:0/1946344418 shutdown_connections 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce73366260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce73363840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 --2- 172.21.15.57:0/1946344418 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce7335acc0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 -- 172.21.15.57:0/1946344418 >> 172.21.15.57:0/1946344418 conn(0x55ce73354fb0 msgr2=0x55ce73355390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 -- 172.21.15.57:0/1946344418 shutdown_connections 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.899+0000 7f00935e9740 1 -- 172.21.15.57:0/1946344418 wait complete. 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 -- start start 2024-04-21T08:50:22.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 Processor -- start 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce73360e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce733613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce733618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ce73366d20 con 0x55ce7335a8e0 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ce73278580 con 0x55ce73363e10 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ce73366ea0 con 0x55ce7335b290 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f00935e9740 10 client.0 _ll_register_callbacks cb 0x55ce7327e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:22.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce73360e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce73360e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45936/0 (socket says 172.21.15.57:45936) 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 -- 172.21.15.57:0/3766119137 learned_addr learned my addr 172.21.15.57:0/3766119137 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f0089d9b640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce733613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce733618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 -- 172.21.15.57:0/3766119137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 msgr2=0x55ce733613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce733613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 -- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 msgr2=0x55ce733618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce733618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.900+0000 7f008ad9d640 1 -- 172.21.15.57:0/3766119137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce730f9f70 con 0x55ce7335a8e0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f008ad9d640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce73360e60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f00840096a0 tx=0x7f0084003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1fc209d77c0206 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f0071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f0071ffb640 1 -- 172.21.15.57:0/3766119137 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084032980 con 0x55ce7335a8e0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f0071ffb640 1 -- 172.21.15.57:0/3766119137 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f0084004170 con 0x55ce7335a8e0 2024-04-21T08:50:22.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0084045070 con 0x55ce7335a8e0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f0071ffb640 1 -- 172.21.15.57:0/3766119137 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f008405d050 con 0x55ce7335a8e0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f00935e9740 10 client.16506 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ce72f91170 con 0x55ce7335a8e0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f0071ffb640 1 -- 172.21.15.57:0/3766119137 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0084045070 con 0x55ce7335a8e0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f0071ffb640 1 -- 172.21.15.57:0/3766119137 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0084040ac0 con 0x55ce7335a8e0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f00935e9740 10 client.16506 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.901+0000 7f00935e9740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f00935e9740 20 client.16506 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f00935e9740 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce73374200 0x55ce733765f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f00935e9740 4 client.16506 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce732cc920 con 0x55ce73374200 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f0089d9b640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce73374200 0x55ce733765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f0089d9b640 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce73374200 0x55ce733765f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13acfa3bb40d5f2e server_cookie=c138f0287d5d7be5 in_seq=0 out_seq=0 2024-04-21T08:50:22.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.902+0000 7f0071ffb640 10 client.16506 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f0071ffb640 1 -- 172.21.15.57:0/3766119137 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce732cc920 con 0x55ce73374200 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f0071ffb640 10 client.16506 handle_command_reply: tid=0 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 client.16506 shutdown 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 msgr2=0x55ce73360e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce73360e60 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f00840096a0 tx=0x7f0084003c40 comp rx=0 tx=0).stop 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 shutdown_connections 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce73374200 0x55ce733765f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce733618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335b290 0x55ce733613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f0071ffb640 0 client.16506 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 --2- 172.21.15.57:0/3766119137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335a8e0 0x55ce73360e60 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 >> 172.21.15.57:0/3766119137 conn(0x55ce73354fb0 msgr2=0x55ce73356d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 shutdown_connections 2024-04-21T08:50:22.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 1 -- 172.21.15.57:0/3766119137 wait complete. 2024-04-21T08:50:22.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.903+0000 7f00935e9740 20 client.16506 trim_cache size 0 max 16384 2024-04-21T08:50:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 Processor -- start 2024-04-21T08:50:22.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 -- start start 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73363590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 0x55ce7335e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce7335e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1ad0b0 con 0x55ce7335b290 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1af910 con 0x55ce73363e10 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f00935e9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1ad700 con 0x55ce7335a8e0 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f0089d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 0x55ce7335e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f0089d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 0x55ce7335e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45952/0 (socket says 172.21.15.57:45952) 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.921+0000 7f0089d9b640 1 -- 172.21.15.57:0/1440853116 learned_addr learned my addr 172.21.15.57:0/1440853116 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f008ad9d640 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73363590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f008ad9d640 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce7335e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0089d9b640 1 -- 172.21.15.57:0/1440853116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 msgr2=0x55ce73363590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0089d9b640 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73363590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0089d9b640 1 -- 172.21.15.57:0/1440853116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 msgr2=0x55ce7335e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0089d9b640 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce7335e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0089d9b640 1 -- 172.21.15.57:0/1440853116 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce7335a3c0 con 0x55ce7335b290 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f008ad9d640 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce7335e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0089d9b640 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 0x55ce7335e0c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f008000c8a0 tx=0x7f0080005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf4e8b707e4ae11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0073fff640 1 -- 172.21.15.57:0/1440853116 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080004220 con 0x55ce7335b290 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0073fff640 1 -- 172.21.15.57:0/1440853116 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0080021030 con 0x55ce7335b290 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.922+0000 7f0073fff640 1 -- 172.21.15.57:0/1440853116 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080004c30 con 0x55ce7335b290 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- 172.21.15.57:0/1440853116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 msgr2=0x55ce7335e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 0x55ce7335e0c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f008000c8a0 tx=0x7f0080005f50 comp rx=0 tx=0).stop 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- 172.21.15.57:0/1440853116 shutdown_connections 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce73363e10 0x55ce7335e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce7335b290 0x55ce7335e0c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- 172.21.15.57:0/1440853116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73363590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.929 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- 172.21.15.57:0/1440853116 >> 172.21.15.57:0/1440853116 conn(0x55ce73354fb0 msgr2=0x55ce73356c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- 172.21.15.57:0/1440853116 shutdown_connections 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- 172.21.15.57:0/1440853116 wait complete. 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- start start 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 Processor -- start 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73377380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce7335b290 0x55ce733778c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 0x55ce733d93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1addd0 con 0x55ce73363e10 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1af240 con 0x55ce7335b290 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1ae4a0 con 0x55ce7335a8e0 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f00935e9740 10 client.0 _ll_register_callbacks cb 0x7f0084009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 0x55ce733d93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f008ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 0x55ce733d93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45962/0 (socket says 172.21.15.57:45962) 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f008ad9d640 1 -- 172.21.15.57:0/2566870595 learned_addr learned my addr 172.21.15.57:0/2566870595 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.923+0000 7f008ad9d640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73377380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0089d9b640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce7335b290 0x55ce733778c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f008ad9d640 1 -- 172.21.15.57:0/2566870595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 msgr2=0x55ce73377380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f008ad9d640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73377380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f008ad9d640 1 -- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce7335b290 msgr2=0x55ce733778c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f008ad9d640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce7335b290 0x55ce733778c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f008ad9d640 1 -- 172.21.15.57:0/2566870595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce73378190 con 0x55ce73363e10 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f008ad9d640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 0x55ce733d93e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f0084009bf0 tx=0x7f008402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190ab92239d686d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0071ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0071ffb640 1 -- 172.21.15.57:0/2566870595 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084039040 con 0x55ce73363e10 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0071ffb640 1 -- 172.21.15.57:0/2566870595 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f00840348b0 con 0x55ce73363e10 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0080021030 con 0x55ce73363e10 2024-04-21T08:50:22.931 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0071ffb640 1 -- 172.21.15.57:0/2566870595 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0084034ab0 con 0x55ce73363e10 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f00935e9740 10 client.16512 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ce733da9d0 con 0x55ce73363e10 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0071ffb640 1 -- 172.21.15.57:0/2566870595 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0080021030 con 0x55ce73363e10 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.924+0000 7f0071ffb640 1 -- 172.21.15.57:0/2566870595 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0084040990 con 0x55ce73363e10 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f00935e9740 10 client.16512 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f00935e9740 10 client.16512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f00935e9740 20 client.16512 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f00935e9740 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce733db180 0x55ce733dd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f00935e9740 4 client.16512 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ce732c6790 con 0x55ce733db180 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f0089d9b640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce733db180 0x55ce733dd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f0089d9b640 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce733db180 0x55ce733dd5a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5001b31a7f25add3 server_cookie=1d28a1073825c23b in_seq=0 out_seq=0 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.925+0000 7f0071ffb640 10 client.16512 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:22.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.926+0000 7f0071ffb640 1 -- 172.21.15.57:0/2566870595 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ce732c6790 con 0x55ce733db180 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.926+0000 7f0071ffb640 10 client.16512 handle_command_reply: tid=0 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.926+0000 7f00935e9740 1 client.16512 shutdown 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 msgr2=0x55ce733d93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 0x55ce733d93e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f0084009bf0 tx=0x7f008402c0f0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 shutdown_connections 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ce733db180 0x55ce733dd5a0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ce73363e10 0x55ce733d93e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ce7335b290 0x55ce733778c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f0071ffb640 0 client.16512 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 --2- 172.21.15.57:0/2566870595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ce7335a8e0 0x55ce73377380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 >> 172.21.15.57:0/2566870595 conn(0x55ce73354fb0 msgr2=0x55ce733563e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 shutdown_connections 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 1 -- 172.21.15.57:0/2566870595 wait complete. 2024-04-21T08:50:22.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00935e9740 20 client.16512 trim_cache size 0 max 16384 2024-04-21T08:50:22.934 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:50:22.934 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:50:22.934 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "76c668b2-07a6-444b-87fc-10f228b0bd24", 2024-04-21T08:50:22.934 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:50:22.934 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f00740a3d20 msgr2=0x7f00740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f00740a3d20 0x7f00740a61e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f007c0cdcf0 tx=0x7f007c010cb0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 msgr2=0x7f008c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 0x7f008c198710 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f007c00d950 tx=0x7f007c00de10 comp rx=0 tx=0).stop 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 shutdown_connections 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f00740a3d20 0x7f00740a61e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f008c19cfd0 0x7f008c19d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f008c107270 0x7f008c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 --2- 172.21.15.57:0/870018952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c06a840 0x7f008c198710 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:22.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.927+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 >> 172.21.15.57:0/870018952 conn(0x7f008c077eb0 msgr2=0x7f008c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:22.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.928+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 shutdown_connections 2024-04-21T08:50:22.936 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:22.928+0000 7f00909f4640 1 -- 172.21.15.57:0/870018952 wait complete. 2024-04-21T08:50:22.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:22.938 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:23.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 -- 172.21.15.57:0/3870970835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110100d40 msgr2=0x7fe1101011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 --2- 172.21.15.57:0/3870970835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110100d40 0x7fe1101011c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fe104009900 tx=0x7fe10402bc80 comp rx=0 tx=0).stop 2024-04-21T08:50:23.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 -- 172.21.15.57:0/3870970835 shutdown_connections 2024-04-21T08:50:23.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 --2- 172.21.15.57:0/3870970835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe110101700 0x7fe11010da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 --2- 172.21.15.57:0/3870970835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110100d40 0x7fe1101011c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 --2- 172.21.15.57:0/3870970835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110106d60 0x7fe110107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 -- 172.21.15.57:0/3870970835 >> 172.21.15.57:0/3870970835 conn(0x7fe110077df0 msgr2=0x7fe110078200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 -- 172.21.15.57:0/3870970835 shutdown_connections 2024-04-21T08:50:23.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.048+0000 7fe115001640 1 -- 172.21.15.57:0/3870970835 wait complete. 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.049+0000 7fe115001640 1 Processor -- start 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.049+0000 7fe115001640 1 -- start start 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe115001640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 0x7fe110198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe115001640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe110101700 0x7fe110198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe115001640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110106d60 0x7fe11019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe115001640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe110110f60 con 0x7fe110106d60 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe115001640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe110110de0 con 0x7fe110101700 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe115001640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101110e0 con 0x7fe110100d40 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 0x7fe110198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe110101700 0x7fe110198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 0x7fe110198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:53938/0 (socket says 172.21.15.57:53938) 2024-04-21T08:50:23.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 -- 172.21.15.57:0/2073031511 learned_addr learned my addr 172.21.15.57:0/2073031511 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe114800640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110106d60 0x7fe11019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 -- 172.21.15.57:0/2073031511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe110101700 msgr2=0x7fe110198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe110101700 0x7fe110198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 -- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110106d60 msgr2=0x7fe11019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110106d60 0x7fe11019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 -- 172.21.15.57:0/2073031511 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11019d520 con 0x7fe110100d40 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.050+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 0x7fe110198530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe10000e9f0 tx=0x7fe10000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.051+0000 7fe0f7fff640 1 -- 172.21.15.57:0/2073031511 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10000cd80 con 0x7fe110100d40 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.051+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11019d810 con 0x7fe110100d40 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.051+0000 7fe0f7fff640 1 -- 172.21.15.57:0/2073031511 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1000054d0 con 0x7fe110100d40 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.051+0000 7fe0f7fff640 1 -- 172.21.15.57:0/2073031511 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100010690 con 0x7fe110100d40 2024-04-21T08:50:23.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.051+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1101a50a0 con 0x7fe110100d40 2024-04-21T08:50:23.054 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.053+0000 7fe0f7fff640 1 -- 172.21.15.57:0/2073031511 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe100004e90 con 0x7fe110100d40 2024-04-21T08:50:23.054 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.053+0000 7fe117bea740 1 Processor -- start 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.053+0000 7fe117bea740 1 -- start start 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe117bea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81210ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe117bea740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81211020 0x55fe812196c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe117bea740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 0x55fe8121bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe117bea740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fe811751e0 con 0x55fe81219c00 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe117bea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fe81112390 con 0x55fe81210700 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe117bea740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fe811711a0 con 0x55fe81211020 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 0x55fe8121bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe114800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81210ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe114800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81210ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:41308/0 (socket says 172.21.15.57:41308) 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 0x55fe8121bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45980/0 (socket says 172.21.15.57:45980) 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe114800640 1 -- 172.21.15.57:0/534744284 learned_addr learned my addr 172.21.15.57:0/534744284 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81211020 0x55fe812196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.055 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe0f7fff640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0f80a3d20 0x7fe0f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 -- 172.21.15.57:0/534744284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81211020 msgr2=0x55fe812196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81211020 0x55fe812196c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 -- 172.21.15.57:0/534744284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 msgr2=0x55fe81210ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81210ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 -- 172.21.15.57:0/534744284 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe80f517d0 con 0x55fe81219c00 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe114800640 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81210ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe0f7fff640 1 -- 172.21.15.57:0/2073031511 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe1000d0f50 con 0x7fe110100d40 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe114800640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0f80a3d20 0x7fe0f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.054+0000 7fe10e59c640 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 0x55fe8121bff0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe10402c1c0 tx=0x7fe1040095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0b4b50d6cf8f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.055+0000 7fe0f57fa640 1 -- 172.21.15.57:0/534744284 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104045020 con 0x55fe81219c00 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.055+0000 7fe0f57fa640 1 -- 172.21.15.57:0/534744284 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10402c7d0 con 0x55fe81219c00 2024-04-21T08:50:23.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.055+0000 7fe0f57fa640 1 -- 172.21.15.57:0/534744284 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040054b0 con 0x55fe81219c00 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.055+0000 7fe117bea740 1 -- 172.21.15.57:0/534744284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 msgr2=0x55fe8121bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.055+0000 7fe114800640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0f80a3d20 0x7fe0f80a61e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe108003e10 tx=0x7fe108004a40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.055+0000 7fe117bea740 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 0x55fe8121bff0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe10402c1c0 tx=0x7fe1040095e0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- 172.21.15.57:0/534744284 shutdown_connections 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81219c00 0x55fe8121bff0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81211020 0x55fe812196c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 --2- 172.21.15.57:0/534744284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81210ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- 172.21.15.57:0/534744284 >> 172.21.15.57:0/534744284 conn(0x55fe8120ae30 msgr2=0x55fe8120b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- 172.21.15.57:0/534744284 shutdown_connections 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- 172.21.15.57:0/534744284 wait complete. 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- start start 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 Processor -- start 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81216c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 0x55fe812171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81219c00 0x55fe812176e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fe8121c990 con 0x55fe81211020 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fe8112e2f0 con 0x55fe81210700 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fe8121cb10 con 0x55fe81219c00 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe117bea740 10 client.0 _ll_register_callbacks cb 0x55fe8113b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.056+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 0x55fe812171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 0x55fe812171a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:45992/0 (socket says 172.21.15.57:45992) 2024-04-21T08:50:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81216c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 -- 172.21.15.57:0/3205251873 learned_addr learned my addr 172.21.15.57:0/3205251873 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe114800640 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81219c00 0x55fe812176e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 -- 172.21.15.57:0/3205251873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81219c00 msgr2=0x55fe812176e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81219c00 0x55fe812176e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 -- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 msgr2=0x55fe81216c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81216c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 -- 172.21.15.57:0/3205251873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe80e873f0 con 0x55fe81211020 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe10e59c640 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 0x55fe812171a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe104036040 tx=0x7fe104005460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14e441f8d81026 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe0f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/3205251873 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104045070 con 0x55fe81211020 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.057+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/3205251873 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe104004450 con 0x55fe81211020 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe10402c7d0 con 0x55fe81211020 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/3205251873 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe104005650 con 0x55fe81211020 2024-04-21T08:50:23.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 10 client.16524 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fe80e470f0 con 0x55fe81211020 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/3205251873 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10402c7d0 con 0x55fe81211020 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/3205251873 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe104004880 con 0x55fe81211020 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 10 client.16524 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 20 client.16524 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe81229a50 0x55fe8122be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 4 client.16524 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe81182c00 con 0x55fe81229a50 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.058+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe81229a50 0x55fe8122be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.059+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe81229a50 0x55fe8122be40 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=246ab41c21030cbf server_cookie=36623002f6f424f3 in_seq=0 out_seq=0 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.059+0000 7fe0f2ff5640 10 client.16524 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/3205251873 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fe81182c00 con 0x55fe81229a50 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe0f2ff5640 10 client.16524 handle_command_reply: tid=0 2024-04-21T08:50:23.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 client.16524 shutdown 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 msgr2=0x55fe812171a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 0x55fe812171a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe104036040 tx=0x7fe104005460 comp rx=0 tx=0).stop 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 shutdown_connections 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe81229a50 0x55fe8122be40 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe0f2ff5640 0 client.16524 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81219c00 0x55fe812176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81211020 0x55fe812171a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 --2- 172.21.15.57:0/3205251873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81210700 0x55fe81216c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 >> 172.21.15.57:0/3205251873 conn(0x55fe8120ae30 msgr2=0x55fe8120cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 shutdown_connections 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 1 -- 172.21.15.57:0/3205251873 wait complete. 2024-04-21T08:50:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.060+0000 7fe117bea740 20 client.16524 trim_cache size 0 max 16384 2024-04-21T08:50:23.080 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.079+0000 7fe117bea740 1 Processor -- start 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.079+0000 7fe117bea740 1 -- start start 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe117bea740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe81219390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe117bea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe81215bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe117bea740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe812314a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe117bea740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101acfa0 con 0x55fe81225850 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe117bea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101af800 con 0x55fe81219c00 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe117bea740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ad5f0 con 0x55fe81210700 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe114800640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe812314a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe81219390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe114800640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe812314a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46002/0 (socket says 172.21.15.57:46002) 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe81219390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:53966/0 (socket says 172.21.15.57:53966) 2024-04-21T08:50:23.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe81215bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe114800640 1 -- 172.21.15.57:0/1890470426 learned_addr learned my addr 172.21.15.57:0/1890470426 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 -- 172.21.15.57:0/1890470426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 msgr2=0x55fe81215bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe81215bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 -- 172.21.15.57:0/1890470426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 msgr2=0x55fe812314a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe812314a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 -- 172.21.15.57:0/1890470426 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe81225cd0 con 0x55fe81210700 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe114800640 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe812314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.080+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe81219390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe1000ce340 tx=0x7fe1000d3460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d64b34bc6dc7a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe0f4ff9640 1 -- 172.21.15.57:0/1890470426 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10009c6d0 con 0x55fe81210700 2024-04-21T08:50:23.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe0f4ff9640 1 -- 172.21.15.57:0/1890470426 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe100005680 con 0x55fe81210700 2024-04-21T08:50:23.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe0f4ff9640 1 -- 172.21.15.57:0/1890470426 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100005180 con 0x55fe81210700 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 -- 172.21.15.57:0/1890470426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 msgr2=0x55fe81219390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe81219390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe1000ce340 tx=0x7fe1000d3460 comp rx=0 tx=0).stop 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 -- 172.21.15.57:0/1890470426 shutdown_connections 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe812314a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe81215bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 --2- 172.21.15.57:0/1890470426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe81219390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 -- 172.21.15.57:0/1890470426 >> 172.21.15.57:0/1890470426 conn(0x55fe8120ae30 msgr2=0x55fe812174b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 -- 172.21.15.57:0/1890470426 shutdown_connections 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.081+0000 7fe117bea740 1 -- 172.21.15.57:0/1890470426 wait complete. 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 -- start start 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 Processor -- start 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe8121ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe8121f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe8128fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101adcc0 con 0x55fe81225850 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101af130 con 0x55fe81219c00 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe1101ae390 con 0x55fe81210700 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe117bea740 10 client.0 _ll_register_callbacks cb 0x7fe108009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe114800640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe8128fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe8121ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe10e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe8121f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe114800640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe8128fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46014/0 (socket says 172.21.15.57:46014) 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe114800640 1 -- 172.21.15.57:0/1727177693 learned_addr learned my addr 172.21.15.57:0/1727177693 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.082+0000 7fe10ed9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe8121ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:53976/0 (socket says 172.21.15.57:53976) 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10e59c640 1 -- 172.21.15.57:0/1727177693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 msgr2=0x55fe8121ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10e59c640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe8121ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10e59c640 1 -- 172.21.15.57:0/1727177693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 msgr2=0x55fe8128fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10e59c640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe8128fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10e59c640 1 -- 172.21.15.57:0/1727177693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe81290260 con 0x55fe81219c00 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe8121ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe114800640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe8128fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe10e59c640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe8121f270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55fe81217cf0 tx=0x7fe1040050f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c92a5206ce386836 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe0f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:50:23.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/1727177693 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104045070 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe100005680 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/1727177693 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fe1040048d0 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/1727177693 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe10402c7d0 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe117bea740 10 client.26000 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fe81291370 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/1727177693 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe100005680 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.083+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/1727177693 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fe104041480 con 0x55fe81219c00 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe117bea740 10 client.26000 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe117bea740 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe117bea740 20 client.26000 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:23.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe117bea740 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe812919d0 0x55fe81293dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe117bea740 4 client.26000 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe8117ca70 con 0x55fe812919d0 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe812919d0 0x55fe81293dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe0f2ff5640 10 client.26000 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.084+0000 7fe10ed9d640 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe812919d0 0x55fe81293dc0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=649af705e134e787 server_cookie=8d711133be7eb003 in_seq=0 out_seq=0 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe0f2ff5640 1 -- 172.21.15.57:0/1727177693 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55fe8117ca70 con 0x55fe812919d0 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe0f2ff5640 10 client.26000 handle_command_reply: tid=0 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 client.26000 shutdown 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 msgr2=0x55fe8121f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe8121f270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x55fe81217cf0 tx=0x7fe1040050f0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 shutdown_connections 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fe812919d0 0x55fe81293dc0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fe81225850 0x55fe8128fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe0f2ff5640 0 client.26000 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fe81219c00 0x55fe8121f270 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 --2- 172.21.15.57:0/1727177693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fe81210700 0x55fe8121ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 >> 172.21.15.57:0/1727177693 conn(0x55fe8120ae30 msgr2=0x55fe81209640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 shutdown_connections 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 1 -- 172.21.15.57:0/1727177693 wait complete. 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe117bea740 20 client.26000 trim_cache size 0 max 16384 2024-04-21T08:50:23.093 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (3811 inodes in the stack)", 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: "76c668b2-07a6-444b-87fc-10f228b0bd24": { 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "76c668b2-07a6-444b-87fc-10f228b0bd24", 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:50:23.094 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:50:23.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0f80a3d20 msgr2=0x7fe0f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe115001640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0f80a3d20 0x7fe0f80a61e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe108003e10 tx=0x7fe108004a40 comp rx=0 tx=0).stop 2024-04-21T08:50:23.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 msgr2=0x7fe110198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.085+0000 7fe115001640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 0x7fe110198530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe10000e9f0 tx=0x7fe10000edf0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 shutdown_connections 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe0f80a3d20 0x7fe0f80a61e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe110106d60 0x7fe11019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe110101700 0x7fe110198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 --2- 172.21.15.57:0/2073031511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe110100d40 0x7fe110198530 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 >> 172.21.15.57:0/2073031511 conn(0x7fe110077df0 msgr2=0x7fe110069240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 shutdown_connections 2024-04-21T08:50:23.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:23.086+0000 7fe115001640 1 -- 172.21.15.57:0/2073031511 wait complete. 2024-04-21T08:50:23.097 INFO:tasks.cephfs.filesystem:scrub status for tag:76c668b2-07a6-444b-87fc-10f228b0bd24 - {'path': '/', 'tag': '76c668b2-07a6-444b-87fc-10f228b0bd24', 'options': 'recursive,force'} 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 -- 172.21.15.57:0/1791396931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 msgr2=0x7f6b4410e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 --2- 172.21.15.57:0/1791396931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4410e9c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009ca0 tx=0x7f6b3c02c230 comp rx=0 tx=0).stop 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 -- 172.21.15.57:0/1791396931 shutdown_connections 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 --2- 172.21.15.57:0/1791396931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4410e9c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 --2- 172.21.15.57:0/1791396931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 0x7f6b4410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 --2- 172.21.15.57:0/1791396931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 0x7f6b441035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.458 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 -- 172.21.15.57:0/1791396931 >> 172.21.15.57:0/1791396931 conn(0x7f6b44077fd0 msgr2=0x7f6b440783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 -- 172.21.15.57:0/1791396931 shutdown_connections 2024-04-21T08:50:23.459 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.458+0000 7f6b488a5640 1 -- 172.21.15.57:0/1791396931 wait complete. 2024-04-21T08:50:23.460 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 Processor -- start 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 -- start start 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 0x7f6b44198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 0x7f6b44198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b44112190 con 0x7f6b4410c520 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b44112010 con 0x7f6b441031e0 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b488a5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b44112310 con 0x7f6b44103b00 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b42d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 0x7f6b44198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b42d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4419cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46038/0 (socket says 172.21.15.57:46038) 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b42d9d640 1 -- 172.21.15.57:0/3953313175 learned_addr learned my addr 172.21.15.57:0/3953313175 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.459+0000 7f6b4259c640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 0x7f6b44198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b42d9d640 1 -- 172.21.15.57:0/3953313175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 msgr2=0x7f6b44198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b42d9d640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 0x7f6b44198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b42d9d640 1 -- 172.21.15.57:0/3953313175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 msgr2=0x7f6b44198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b42d9d640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 0x7f6b44198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b42d9d640 1 -- 172.21.15.57:0/3953313175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4419d640 con 0x7f6b4410c520 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 0x7f6b44198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b4259c640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 0x7f6b44198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b42d9d640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4419cf10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c005e30 tx=0x7f6b3c035dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b3b7fe640 1 -- 172.21.15.57:0/3953313175 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c039040 con 0x7f6b4410c520 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b3b7fe640 1 -- 172.21.15.57:0/3953313175 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b3c044070 con 0x7f6b4410c520 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4419d8d0 con 0x7f6b4410c520 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b3b7fe640 1 -- 172.21.15.57:0/3953313175 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0328a0 con 0x7f6b4410c520 2024-04-21T08:50:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.460+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b44100d60 con 0x7f6b4410c520 2024-04-21T08:50:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 Processor -- start 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b3b7fe640 1 -- 172.21.15.57:0/3953313175 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6b3c032a40 con 0x7f6b4410c520 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 -- start start 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 0x55636d3f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55636d3578d0 con 0x55636d3f2b60 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55636d2f4e80 con 0x55636d3f38d0 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4b575740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55636d353890 con 0x55636d3fc0e0 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 0x55636d3f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 0x55636d3f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46050/0 (socket says 172.21.15.57:46050) 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b4259c640 1 -- 172.21.15.57:0/2605465432 learned_addr learned my addr 172.21.15.57:0/2605465432 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.464+0000 7f6b3b7fe640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b300a3dc0 0x7f6b300a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3bfff640 1 -- 172.21.15.57:0/2605465432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 msgr2=0x55636d3f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3bfff640 1 -- 172.21.15.57:0/2605465432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 msgr2=0x55636d3f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 0x55636d3f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3bfff640 1 -- 172.21.15.57:0/2605465432 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636d316320 con 0x55636d3fc0e0 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b4259c640 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 0x55636d3f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b42d9d640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b300a3dc0 0x7f6b300a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fe4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009730 tx=0x7f6b2c0096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c5a52c8aa798a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b3b7fe640 1 -- 172.21.15.57:0/3953313175 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6b3c0f2dd0 con 0x7f6b4410c520 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b38ff9640 1 -- 172.21.15.57:0/2605465432 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c004030 con 0x55636d3fc0e0 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b38ff9640 1 -- 172.21.15.57:0/2605465432 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b2c00f910 con 0x55636d3fc0e0 2024-04-21T08:50:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.465+0000 7f6b38ff9640 1 -- 172.21.15.57:0/2605465432 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c0054f0 con 0x55636d3fc0e0 2024-04-21T08:50:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 -- 172.21.15.57:0/2605465432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 msgr2=0x55636d3fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fe4d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009730 tx=0x7f6b2c0096c0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.470 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b42d9d640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b300a3dc0 0x7f6b300a6280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0bb4f0 tx=0x7f6b3c090560 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 -- 172.21.15.57:0/2605465432 shutdown_connections 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fe4d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f2b60 0x55636d3f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 --2- 172.21.15.57:0/2605465432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 -- 172.21.15.57:0/2605465432 >> 172.21.15.57:0/2605465432 conn(0x55636d3b9c90 msgr2=0x55636d3ed850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 -- 172.21.15.57:0/2605465432 shutdown_connections 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 -- 172.21.15.57:0/2605465432 wait complete. 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 -- start start 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.466+0000 7f6b4b575740 1 Processor -- start 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d3fba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3fc0e0 0x55636d3f93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55636d3fee10 con 0x55636d3f38d0 2024-04-21T08:50:23.471 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55636d2dfeb0 con 0x55636d3fc0e0 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55636d3fef90 con 0x55636d3f2b60 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4b575740 10 client.0 _ll_register_callbacks cb 0x55636d30cb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d3fba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d3fba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:54018/0 (socket says 172.21.15.57:54018) 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b3bfff640 1 -- 172.21.15.57:0/4182365000 learned_addr learned my addr 172.21.15.57:0/4182365000 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3fc0e0 0x55636d3f93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 -- 172.21.15.57:0/4182365000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 msgr2=0x55636d3fba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d3fba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 -- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3fc0e0 msgr2=0x55636d3f93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3fc0e0 0x55636d3f93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 -- 172.21.15.57:0/4182365000 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636d06a210 con 0x55636d3f38d0 2024-04-21T08:50:23.472 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b4259c640 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3f8e20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6b34012cd0 tx=0x7f6b3400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50955003b34ec99 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.467+0000 7f6b29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b29ffb640 1 -- 172.21.15.57:0/4182365000 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b340120d0 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b29ffb640 1 -- 172.21.15.57:0/4182365000 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6b340047a0 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b2c00f910 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b29ffb640 1 -- 172.21.15.57:0/4182365000 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b34025050 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 10 client.16542 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55636d136620 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b29ffb640 1 -- 172.21.15.57:0/4182365000 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b2c00f910 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b29ffb640 1 -- 172.21.15.57:0/4182365000 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6b3401a400 con 0x55636d3f38d0 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 10 client.16542 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 10 client.16542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 20 client.16542 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d40c1b0 0x55636d40e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 4 client.16542 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:23.473 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.468+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55636d36b880 con 0x55636d40c1b0 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.469+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d40c1b0 0x55636d40e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.469+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d40c1b0 0x55636d40e5a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb941a3a823e5202 server_cookie=f5994f27f1d6279f in_seq=0 out_seq=0 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.469+0000 7f6b29ffb640 10 client.16542 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b29ffb640 1 -- 172.21.15.57:0/4182365000 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55636d36b880 con 0x55636d40c1b0 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b29ffb640 10 client.16542 handle_command_reply: tid=0 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 client.16542 shutdown 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 msgr2=0x55636d3f8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3f8e20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6b34012cd0 tx=0x7f6b3400f790 comp rx=0 tx=0).stop 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 shutdown_connections 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d40c1b0 0x55636d40e5a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3fc0e0 0x55636d3f93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3f8e20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b29ffb640 0 client.16542 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 --2- 172.21.15.57:0/4182365000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d3fba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 >> 172.21.15.57:0/4182365000 conn(0x55636d3b9c90 msgr2=0x55636d3ef5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 shutdown_connections 2024-04-21T08:50:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.470+0000 7f6b4b575740 1 -- 172.21.15.57:0/4182365000 wait complete. 2024-04-21T08:50:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.471+0000 7f6b4b575740 20 client.16542 trim_cache size 0 max 16384 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 Processor -- start 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 -- start start 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d407d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3fc0e0 0x55636d3fb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441af3f0 con 0x55636d3fc0e0 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441b1c50 con 0x55636d3f38d0 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4b575740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441afa40 con 0x55636d3f2b60 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d407d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d407d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:54032/0 (socket says 172.21.15.57:54032) 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 -- 172.21.15.57:0/729733959 learned_addr learned my addr 172.21.15.57:0/729733959 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3fc0e0 0x55636d3fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 -- 172.21.15.57:0/729733959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 msgr2=0x55636d3f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 -- 172.21.15.57:0/729733959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3fc0e0 msgr2=0x55636d3fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3fc0e0 0x55636d3fb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.489+0000 7f6b3bfff640 1 -- 172.21.15.57:0/729733959 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636d40ec00 con 0x55636d3f2b60 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.490+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d407d40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0050d0 tx=0x7f6b2c009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3053b9e95056eede server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.490+0000 7f6b2bfff640 1 -- 172.21.15.57:0/729733959 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c011050 con 0x55636d3f2b60 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.490+0000 7f6b2bfff640 1 -- 172.21.15.57:0/729733959 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b2c030880 con 0x55636d3f2b60 2024-04-21T08:50:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.490+0000 7f6b2bfff640 1 -- 172.21.15.57:0/729733959 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c00fde0 con 0x55636d3f2b60 2024-04-21T08:50:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 -- 172.21.15.57:0/729733959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 msgr2=0x55636d407d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d407d40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0050d0 tx=0x7f6b2c009ce0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 -- 172.21.15.57:0/729733959 shutdown_connections 2024-04-21T08:50:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3fc0e0 0x55636d3fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f38d0 0x55636d3f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 --2- 172.21.15.57:0/729733959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3f2b60 0x55636d407d40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 -- 172.21.15.57:0/729733959 >> 172.21.15.57:0/729733959 conn(0x55636d3b9c90 msgr2=0x55636d3f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 -- 172.21.15.57:0/729733959 shutdown_connections 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 -- 172.21.15.57:0/729733959 wait complete. 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 -- start start 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.491+0000 7f6b4b575740 1 Processor -- start 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f2b60 0x55636d404690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441b0110 con 0x55636d3f38d0 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441b1580 con 0x55636d3f2b60 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b441b07e0 con 0x55636d3fc0e0 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 10 client.0 _ll_register_callbacks cb 0x7f6b34009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b3bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f2b60 0x55636d404690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3ffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46064/0 (socket says 172.21.15.57:46064) 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 -- 172.21.15.57:0/3116528878 learned_addr learned my addr 172.21.15.57:0/3116528878 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 -- 172.21.15.57:0/3116528878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 msgr2=0x55636d3fff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 -- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f2b60 msgr2=0x55636d404690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f2b60 0x55636d404690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 -- 172.21.15.57:0/3116528878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636d3e8f30 con 0x55636d3f38d0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4259c640 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3ffa00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6b3401a9c0 tx=0x7f6b340122e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfaa438ca7cbd19d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b29ffb640 1 -- 172.21.15.57:0/3116528878 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3402e060 con 0x55636d3f38d0 2024-04-21T08:50:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.492+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b2c030880 con 0x55636d3f38d0 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b29ffb640 1 -- 172.21.15.57:0/3116528878 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6b3401a580 con 0x55636d3f38d0 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b29ffb640 1 -- 172.21.15.57:0/3116528878 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6b34035050 con 0x55636d3f38d0 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 10 client.16548 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55636d3f1e30 con 0x55636d3f38d0 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b29ffb640 1 -- 172.21.15.57:0/3116528878 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b2c030880 con 0x55636d3f38d0 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b29ffb640 1 -- 172.21.15.57:0/3116528878 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6b340035f0 con 0x55636d3f38d0 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 10 client.16548 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 10 client.16548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 20 client.16548 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d472bd0 0x55636d474ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 4 client.16548 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55636d2f73d0 con 0x55636d472bd0 2024-04-21T08:50:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.493+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d472bd0 0x55636d474ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.494+0000 7f6b29ffb640 10 client.16548 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:23.494+0000 7f6b3bfff640 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d472bd0 0x55636d474ff0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a677478f281b9447 server_cookie=934097c61e5893d2 in_seq=0 out_seq=0 2024-04-21T08:50:24.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:24 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:50:24.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:24 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:24.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:24 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:50:24.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:24 smithi129 ceph-mon[31272]: pgmap v152: 129 pgs: 129 active+clean; 724 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 18 MiB/s wr, 1.09k op/s 2024-04-21T08:50:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:24 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:50:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:24 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:24 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:50:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:24 smithi057 ceph-mon[27968]: pgmap v152: 129 pgs: 129 active+clean; 724 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 18 MiB/s wr, 1.09k op/s 2024-04-21T08:50:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:24 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:50:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:24 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:50:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:24 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:50:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:24 smithi136 ceph-mon[31791]: pgmap v152: 129 pgs: 129 active+clean; 724 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 18 MiB/s wr, 1.09k op/s 2024-04-21T08:50:26.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:26 smithi129 ceph-mon[31272]: pgmap v153: 129 pgs: 129 active+clean; 724 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 17 MiB/s wr, 50 op/s 2024-04-21T08:50:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:26 smithi057 ceph-mon[27968]: pgmap v153: 129 pgs: 129 active+clean; 724 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 17 MiB/s wr, 50 op/s 2024-04-21T08:50:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:26 smithi136 ceph-mon[31791]: pgmap v153: 129 pgs: 129 active+clean; 724 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 17 MiB/s wr, 50 op/s 2024-04-21T08:50:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b29ffb640 1 -- 172.21.15.57:0/3116528878 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55636d2f73d0 con 0x55636d472bd0 2024-04-21T08:50:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b29ffb640 10 client.16548 handle_command_reply: tid=0 2024-04-21T08:50:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 client.16548 shutdown 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 msgr2=0x55636d3ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3ffa00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6b3401a9c0 tx=0x7f6b340122e0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 shutdown_connections 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55636d472bd0 0x55636d474ff0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55636d3fc0e0 0x55636d3fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b29ffb640 0 client.16548 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55636d3f38d0 0x55636d3ffa00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 --2- 172.21.15.57:0/3116528878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55636d3f2b60 0x55636d404690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 >> 172.21.15.57:0/3116528878 conn(0x55636d3b9c90 msgr2=0x55636d3eebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 shutdown_connections 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 1 -- 172.21.15.57:0/3116528878 wait complete. 2024-04-21T08:50:26.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.703+0000 7f6b4b575740 20 client.16548 trim_cache size 0 max 16384 2024-04-21T08:50:26.705 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:50:26.705 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:50:26.705 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:50:26.705 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b300a3dc0 msgr2=0x7f6b300a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b300a3dc0 0x7f6b300a6280 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0bb4f0 tx=0x7f6b3c090560 comp rx=0 tx=0).stop 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 msgr2=0x7f6b4419cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4419cf10 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c005e30 tx=0x7f6b3c035dd0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 shutdown_connections 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6b300a3dc0 0x7f6b300a6280 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b4410c520 0x7f6b4419cf10 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b44103b00 0x7f6b44198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 --2- 172.21.15.57:0/3953313175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b441031e0 0x7f6b44198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 >> 172.21.15.57:0/3953313175 conn(0x7f6b44077fd0 msgr2=0x7f6b440fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:26.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 shutdown_connections 2024-04-21T08:50:26.707 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:26.704+0000 7f6b488a5640 1 -- 172.21.15.57:0/3953313175 wait complete. 2024-04-21T08:50:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:28 smithi129 ceph-mon[31272]: pgmap v154: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 24 MiB/s wr, 1.51k op/s 2024-04-21T08:50:28.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:28 smithi057 ceph-mon[27968]: pgmap v154: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 24 MiB/s wr, 1.51k op/s 2024-04-21T08:50:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:28 smithi136 ceph-mon[31791]: pgmap v154: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 24 MiB/s wr, 1.51k op/s 2024-04-21T08:50:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:30 smithi129 ceph-mon[31272]: pgmap v155: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 8.4 MiB/s wr, 1.47k op/s 2024-04-21T08:50:30.588 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:30 smithi057 ceph-mon[27968]: pgmap v155: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 8.4 MiB/s wr, 1.47k op/s 2024-04-21T08:50:30.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:30 smithi136 ceph-mon[31791]: pgmap v155: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 8.4 MiB/s wr, 1.47k op/s 2024-04-21T08:50:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:32 smithi129 ceph-mon[31272]: pgmap v156: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 8.4 MiB/s wr, 1.47k op/s 2024-04-21T08:50:32.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:32 smithi057 ceph-mon[27968]: pgmap v156: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 8.4 MiB/s wr, 1.47k op/s 2024-04-21T08:50:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:32 smithi136 ceph-mon[31791]: pgmap v156: 129 pgs: 129 active+clean; 810 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 8.4 MiB/s wr, 1.47k op/s 2024-04-21T08:50:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:34 smithi129 ceph-mon[31272]: pgmap v157: 129 pgs: 129 active+clean; 779 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.9 MiB/s wr, 2.15k op/s 2024-04-21T08:50:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:34 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:34 smithi057 ceph-mon[27968]: pgmap v157: 129 pgs: 129 active+clean; 779 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.9 MiB/s wr, 2.15k op/s 2024-04-21T08:50:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:34 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:34 smithi136 ceph-mon[31791]: pgmap v157: 129 pgs: 129 active+clean; 779 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 8.9 MiB/s wr, 2.15k op/s 2024-04-21T08:50:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:34 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:50:35.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:35 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:35.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:35 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:35.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:35 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:50:35.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:35 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:35.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:35 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:35.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:35 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:50:35.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:35 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:50:35.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:35 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:50:35.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:35 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:50:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:36 smithi129 ceph-mon[31272]: pgmap v158: 129 pgs: 129 active+clean; 779 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.1 MiB/s wr, 2.14k op/s 2024-04-21T08:50:36.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:36 smithi057 ceph-mon[27968]: pgmap v158: 129 pgs: 129 active+clean; 779 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.1 MiB/s wr, 2.14k op/s 2024-04-21T08:50:36.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:36 smithi136 ceph-mon[31791]: pgmap v158: 129 pgs: 129 active+clean; 779 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 8.1 MiB/s wr, 2.14k op/s 2024-04-21T08:50:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:38 smithi129 ceph-mon[31272]: pgmap v159: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 16 MiB/s wr, 2.88k op/s 2024-04-21T08:50:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:38 smithi057 ceph-mon[27968]: pgmap v159: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 16 MiB/s wr, 2.88k op/s 2024-04-21T08:50:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:38 smithi136 ceph-mon[31791]: pgmap v159: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 16 MiB/s wr, 2.88k op/s 2024-04-21T08:50:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 -- 172.21.15.57:0/2358950825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3194102f70 msgr2=0x7f3194103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 --2- 172.21.15.57:0/2358950825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3194102f70 0x7f3194103350 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3184009900 tx=0x7f318402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 -- 172.21.15.57:0/2358950825 shutdown_connections 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 --2- 172.21.15.57:0/2358950825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f319410c340 0x7f319410e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 --2- 172.21.15.57:0/2358950825 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3194103890 0x7f319410bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 --2- 172.21.15.57:0/2358950825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3194102f70 0x7f3194103350 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 -- 172.21.15.57:0/2358950825 >> 172.21.15.57:0/2358950825 conn(0x7f31940fc770 msgr2=0x7f31940feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 -- 172.21.15.57:0/2358950825 shutdown_connections 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.796+0000 7f3198c51640 1 -- 172.21.15.57:0/2358950825 wait complete. 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 Processor -- start 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 -- start start 2024-04-21T08:50:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3194102f70 0x7f31941985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3194103890 0x7f3194198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 0x7f319419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3194111fa0 con 0x7f319410c340 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3194111e20 con 0x7f3194102f70 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.797+0000 7f3198c51640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3194112120 con 0x7f3194103890 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 0x7f319419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 0x7f319419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37064/0 (socket says 172.21.15.57:37064) 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 -- 172.21.15.57:0/1835466954 learned_addr learned my addr 172.21.15.57:0/1835466954 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f31937fe640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3194102f70 0x7f31941985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f318affd640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3194103890 0x7f3194198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 -- 172.21.15.57:0/1835466954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3194103890 msgr2=0x7f3194198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3194103890 0x7f3194198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 -- 172.21.15.57:0/1835466954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3194102f70 msgr2=0x7f31941985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3194102f70 0x7f31941985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 -- 172.21.15.57:0/1835466954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319419d5b0 con 0x7f319410c340 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3193fff640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 0x7f319419ce80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f318c00eb10 tx=0x7f318c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f31917fa640 1 -- 172.21.15.57:0/1835466954 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c004b20 con 0x7f319410c340 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319419d8a0 con 0x7f319410c340 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f31917fa640 1 -- 172.21.15.57:0/1835466954 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c004cc0 con 0x7f319410c340 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31941a5130 con 0x7f319410c340 2024-04-21T08:50:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.798+0000 7f31917fa640 1 -- 172.21.15.57:0/1835466954 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c018630 con 0x7f319410c340 2024-04-21T08:50:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.800+0000 7f31917fa640 1 -- 172.21.15.57:0/1835466954 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f318c005000 con 0x7f319410c340 2024-04-21T08:50:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 Processor -- start 2024-04-21T08:50:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 -- start start 2024-04-21T08:50:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd369810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd36a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd3756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5626dd2ce8d0 con 0x5626dd3732d0 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5626dd26be80 con 0x5626dd36aac0 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f319cb7d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5626dd2ca890 con 0x5626dd369d50 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f31917fa640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31800a3d10 0x7f31800a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd3756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f318affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd369810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f318affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd369810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:48140/0 (socket says 172.21.15.57:48140) 2024-04-21T08:50:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.801+0000 7f318affd640 1 -- 172.21.15.57:0/3839504044 learned_addr learned my addr 172.21.15.57:0/3839504044 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f31937fe640 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd36a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f3193fff640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31800a3d10 0x7f31800a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f31917fa640 1 -- 172.21.15.57:0/1835466954 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f318c0d11c0 con 0x7f319410c340 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f318affd640 1 -- 172.21.15.57:0/3839504044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 msgr2=0x5626dd36a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f318affd640 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd36a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f318affd640 1 -- 172.21.15.57:0/3839504044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 msgr2=0x5626dd3756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f318affd640 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd3756c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f318affd640 1 -- 172.21.15.57:0/3839504044 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626dd28d320 con 0x5626dd36aac0 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f318affd640 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd369810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f317c0058c0 tx=0x7f317c00b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba3a2b0798f1f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f3193fff640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31800a3d10 0x7f31800a61d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f318c099260 tx=0x7f318c0bd040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f3189ffb640 1 -- 172.21.15.57:0/3839504044 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c004020 con 0x5626dd36aac0 2024-04-21T08:50:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f3189ffb640 1 -- 172.21.15.57:0/3839504044 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f317c00fc10 con 0x5626dd36aac0 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.802+0000 7f3189ffb640 1 -- 172.21.15.57:0/3839504044 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c003830 con 0x5626dd36aac0 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 -- 172.21.15.57:0/3839504044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 msgr2=0x5626dd369810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd369810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f317c0058c0 tx=0x7f317c00b410 comp rx=0 tx=0).stop 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 -- 172.21.15.57:0/3839504044 shutdown_connections 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd3756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd36a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 --2- 172.21.15.57:0/3839504044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd369810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 -- 172.21.15.57:0/3839504044 >> 172.21.15.57:0/3839504044 conn(0x5626dd330c90 msgr2=0x5626dd364a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 -- 172.21.15.57:0/3839504044 shutdown_connections 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 -- 172.21.15.57:0/3839504044 wait complete. 2024-04-21T08:50:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 -- start start 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.803+0000 7f319cb7d740 1 Processor -- start 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd372c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd36ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd370560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5626dd376000 con 0x5626dd3732d0 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5626dd256eb0 con 0x5626dd36aac0 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5626dd376180 con 0x5626dd369d50 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 10 client.0 _ll_register_callbacks cb 0x5626dd283b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd370560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f31937fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd36ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd370560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37076/0 (socket says 172.21.15.57:37076) 2024-04-21T08:50:39.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 -- 172.21.15.57:0/2212887463 learned_addr learned my addr 172.21.15.57:0/2212887463 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f31937fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd36ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:48146/0 (socket says 172.21.15.57:48146) 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd372c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 -- 172.21.15.57:0/2212887463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 msgr2=0x5626dd372c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd372c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 -- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 msgr2=0x5626dd36ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd36ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 -- 172.21.15.57:0/2212887463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626dcfe1210 con 0x5626dd3732d0 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f318affd640 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd370560 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f317c005d00 tx=0x7f317c017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb1f969b2162770 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f3172ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f3172ffd640 1 -- 172.21.15.57:0/2212887463 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c00f760 con 0x5626dd3732d0 2024-04-21T08:50:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f3172ffd640 1 -- 172.21.15.57:0/2212887463 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f317c00f900 con 0x5626dd3732d0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.804+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f317c00fc10 con 0x5626dd3732d0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f3172ffd640 1 -- 172.21.15.57:0/2212887463 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f317c005030 con 0x5626dd3732d0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 10 client.16560 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5626dd0ad620 con 0x5626dd3732d0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f3172ffd640 1 -- 172.21.15.57:0/2212887463 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f317c00fc10 con 0x5626dd3732d0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f3172ffd640 1 -- 172.21.15.57:0/2212887463 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f317c02b020 con 0x5626dd3732d0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 10 client.16560 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 20 client.16560 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd383130 0x5626dd385520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 4 client.16560 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626dd2e2880 con 0x5626dd383130 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.805+0000 7f31937fe640 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd383130 0x5626dd385520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.806+0000 7f3172ffd640 10 client.16560 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.806+0000 7f31937fe640 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd383130 0x5626dd385520 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b778c3cdd51dccef server_cookie=780a4ad377459f03 in_seq=0 out_seq=0 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f3172ffd640 1 -- 172.21.15.57:0/2212887463 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5626dd2e2880 con 0x5626dd383130 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f3172ffd640 10 client.16560 handle_command_reply: tid=0 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 client.16560 shutdown 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 msgr2=0x5626dd370560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd370560 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f317c005d00 tx=0x7f317c017040 comp rx=0 tx=0).stop 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 shutdown_connections 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd383130 0x5626dd385520 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd3732d0 0x5626dd370560 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd36aac0 0x5626dd36ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2212887463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd369d50 0x5626dd372c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f3172ffd640 0 client.16560 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 >> 172.21.15.57:0/2212887463 conn(0x5626dd330c90 msgr2=0x5626dd366790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 shutdown_connections 2024-04-21T08:50:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 1 -- 172.21.15.57:0/2212887463 wait complete. 2024-04-21T08:50:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.808+0000 7f319cb7d740 20 client.16560 trim_cache size 0 max 16384 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.823+0000 7f319cb7d740 1 Processor -- start 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.823+0000 7f319cb7d740 1 -- start start 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd36f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd372710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd372c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f319cb7d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ad140 con 0x5626dd369d50 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f319cb7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f31941af9a0 con 0x5626dd3732d0 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f319cb7d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ad790 con 0x5626dd36aac0 2024-04-21T08:50:39.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd36f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd36f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37090/0 (socket says 172.21.15.57:37090) 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 -- 172.21.15.57:0/1896206929 learned_addr learned my addr 172.21.15.57:0/1896206929 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f31937fe640 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd372710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd372c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 -- 172.21.15.57:0/1896206929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 msgr2=0x5626dd372710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd372710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 -- 172.21.15.57:0/1896206929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 msgr2=0x5626dd372c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd372c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 -- 172.21.15.57:0/1896206929 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626dd385890 con 0x5626dd369d50 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f318affd640 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd36f8c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f317c00b750 tx=0x7f317c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c7aec266297572 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f31897fa640 1 -- 172.21.15.57:0/1896206929 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c00b410 con 0x5626dd369d50 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f31897fa640 1 -- 172.21.15.57:0/1896206929 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f317c005680 con 0x5626dd369d50 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.824+0000 7f31897fa640 1 -- 172.21.15.57:0/1896206929 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c004ac0 con 0x5626dd369d50 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 -- 172.21.15.57:0/1896206929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 msgr2=0x5626dd36f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd36f8c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f317c00b750 tx=0x7f317c005fd0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 -- 172.21.15.57:0/1896206929 shutdown_connections 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd372c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd372710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 --2- 172.21.15.57:0/1896206929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd36f8c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 -- 172.21.15.57:0/1896206929 >> 172.21.15.57:0/1896206929 conn(0x5626dd330c90 msgr2=0x5626dd363ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 -- 172.21.15.57:0/1896206929 shutdown_connections 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 -- 172.21.15.57:0/1896206929 wait complete. 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.825+0000 7f319cb7d740 1 -- start start 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 Processor -- start 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd37b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd37b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd376890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ade60 con 0x5626dd369d50 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f31941af2d0 con 0x5626dd3732d0 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f31941ae530 con 0x5626dd36aac0 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f319cb7d740 10 client.0 _ll_register_callbacks cb 0x7f317c00be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd37b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd37b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:37098/0 (socket says 172.21.15.57:37098) 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 -- 172.21.15.57:0/2724477137 learned_addr learned my addr 172.21.15.57:0/2724477137 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f31937fe640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd37b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd376890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 -- 172.21.15.57:0/2724477137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 msgr2=0x5626dd37b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd37b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 -- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 msgr2=0x5626dd376890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd376890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 -- 172.21.15.57:0/2724477137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626dd3770c0 con 0x5626dd369d50 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f318affd640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd37b380 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f317c00bfd0 tx=0x7f317c00bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e27468490a8da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.826+0000 7f3172ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f3172ffd640 1 -- 172.21.15.57:0/2724477137 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c004840 con 0x5626dd369d50 2024-04-21T08:50:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f3172ffd640 1 -- 172.21.15.57:0/2724477137 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f317c028040 con 0x5626dd369d50 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f317c005680 con 0x5626dd369d50 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f3172ffd640 1 -- 172.21.15.57:0/2724477137 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f317c04b050 con 0x5626dd369d50 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 10 client.16566 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5626dd3e9160 con 0x5626dd369d50 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f3172ffd640 1 -- 172.21.15.57:0/2724477137 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f317c005680 con 0x5626dd369d50 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f3172ffd640 1 -- 172.21.15.57:0/2724477137 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f317c02b040 con 0x5626dd369d50 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 10 client.16566 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 20 client.16566 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd3e9740 0x5626dd3ebb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:39.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 4 client.16566 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626dd26e3d0 con 0x5626dd3e9740 2024-04-21T08:50:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.827+0000 7f31937fe640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd3e9740 0x5626dd3ebb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.828+0000 7f31937fe640 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd3e9740 0x5626dd3ebb60 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7061931e6a7eddd6 server_cookie=44af30550060c49f in_seq=0 out_seq=0 2024-04-21T08:50:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:39.828+0000 7f3172ffd640 10 client.16566 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:40 smithi129 ceph-mon[31272]: pgmap v160: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.3 MiB/s wr, 1.42k op/s 2024-04-21T08:50:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:40 smithi057 ceph-mon[27968]: pgmap v160: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.3 MiB/s wr, 1.42k op/s 2024-04-21T08:50:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:40 smithi136 ceph-mon[31791]: pgmap v160: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.3 MiB/s wr, 1.42k op/s 2024-04-21T08:50:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:42 smithi057 ceph-mon[27968]: pgmap v161: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.3 MiB/s wr, 1.42k op/s 2024-04-21T08:50:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:42 smithi136 ceph-mon[31791]: pgmap v161: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.3 MiB/s wr, 1.42k op/s 2024-04-21T08:50:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:42 smithi129 ceph-mon[31272]: pgmap v161: 129 pgs: 129 active+clean; 869 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 8.3 MiB/s wr, 1.42k op/s 2024-04-21T08:50:44.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:44 smithi057 ceph-mon[27968]: pgmap v162: 129 pgs: 129 active+clean; 913 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 12 MiB/s wr, 1.62k op/s 2024-04-21T08:50:44.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:44 smithi136 ceph-mon[31791]: pgmap v162: 129 pgs: 129 active+clean; 913 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 12 MiB/s wr, 1.62k op/s 2024-04-21T08:50:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:44 smithi129 ceph-mon[31272]: pgmap v162: 129 pgs: 129 active+clean; 913 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 12 MiB/s wr, 1.62k op/s 2024-04-21T08:50:44.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f3172ffd640 1 -- 172.21.15.57:0/2724477137 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626dd26e3d0 con 0x5626dd3e9740 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f3172ffd640 10 client.16566 handle_command_reply: tid=0 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 client.16566 shutdown 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 msgr2=0x5626dd37b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd37b380 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f317c00bfd0 tx=0x7f317c00bb00 comp rx=0 tx=0).stop 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 shutdown_connections 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5626dd3e9740 0x5626dd3ebb60 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626dd3732d0 0x5626dd376890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5626dd36aac0 0x5626dd37b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f3172ffd640 0 client.16566 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 --2- 172.21.15.57:0/2724477137 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5626dd369d50 0x5626dd37b380 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 >> 172.21.15.57:0/2724477137 conn(0x5626dd330c90 msgr2=0x5626dd365cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 shutdown_connections 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.983+0000 7f319cb7d740 1 -- 172.21.15.57:0/2724477137 wait complete. 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f319cb7d740 20 client.16566 trim_cache size 0 max 16384 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:50:44.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31800a3d10 msgr2=0x7f31800a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31800a3d10 0x7f31800a61d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f318c099260 tx=0x7f318c0bd040 comp rx=0 tx=0).stop 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 msgr2=0x7f319419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 0x7f319419ce80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f318c00eb10 tx=0x7f318c00c2d0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 shutdown_connections 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31800a3d10 0x7f31800a61d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f319410c340 0x7f319419ce80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3194103890 0x7f3194198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 --2- 172.21.15.57:0/1835466954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3194102f70 0x7f31941985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 >> 172.21.15.57:0/1835466954 conn(0x7f31940fc770 msgr2=0x7f31940fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 shutdown_connections 2024-04-21T08:50:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:44.984+0000 7f3198c51640 1 -- 172.21.15.57:0/1835466954 wait complete. 2024-04-21T08:50:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:46 smithi057 ceph-mon[27968]: pgmap v163: 129 pgs: 129 active+clean; 913 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 12 MiB/s wr, 944 op/s 2024-04-21T08:50:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:46 smithi136 ceph-mon[31791]: pgmap v163: 129 pgs: 129 active+clean; 913 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 12 MiB/s wr, 944 op/s 2024-04-21T08:50:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:46 smithi129 ceph-mon[31272]: pgmap v163: 129 pgs: 129 active+clean; 913 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 12 MiB/s wr, 944 op/s 2024-04-21T08:50:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:48 smithi057 ceph-mon[27968]: pgmap v164: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 12 MiB/s wr, 2.30k op/s 2024-04-21T08:50:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:48 smithi136 ceph-mon[31791]: pgmap v164: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 12 MiB/s wr, 2.30k op/s 2024-04-21T08:50:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:48 smithi129 ceph-mon[31272]: pgmap v164: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 12 MiB/s wr, 2.30k op/s 2024-04-21T08:50:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:50 smithi136 ceph-mon[31791]: pgmap v165: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.56k op/s 2024-04-21T08:50:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:50 smithi129 ceph-mon[31272]: pgmap v165: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.56k op/s 2024-04-21T08:50:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:50 smithi057 ceph-mon[27968]: pgmap v165: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.56k op/s 2024-04-21T08:50:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:52 smithi136 ceph-mon[31791]: pgmap v166: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.56k op/s 2024-04-21T08:50:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:52 smithi129 ceph-mon[31272]: pgmap v166: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.56k op/s 2024-04-21T08:50:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:52 smithi057 ceph-mon[27968]: pgmap v166: 129 pgs: 129 active+clean; 893 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.6 MiB/s wr, 1.56k op/s 2024-04-21T08:50:53.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:50:53.099 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:50:53.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/724242674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a141031c0 msgr2=0x7f9a141035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/724242674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a141031c0 0x7f9a141035a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009900 tx=0x7f9a0402bd60 comp rx=0 tx=0).stop 2024-04-21T08:50:53.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/724242674 shutdown_connections 2024-04-21T08:50:53.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/724242674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1410c500 0x7f9a1410e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/724242674 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a14103ae0 0x7f9a1410bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/724242674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a141031c0 0x7f9a141035a0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/724242674 >> 172.21.15.57:0/724242674 conn(0x7f9a14078000 msgr2=0x7f9a14078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/724242674 shutdown_connections 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.211+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/724242674 wait complete. 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 Processor -- start 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 -- start start 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a141031c0 0x7f9a14198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 0x7f9a14198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1410c500 0x7f9a1419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9a14112190 con 0x7f9a14103ae0 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a14112010 con 0x7f9a1410c500 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a1a7b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a14112310 con 0x7f9a141031c0 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 0x7f9a14198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 0x7f9a14198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33076/0 (socket says 172.21.15.57:33076) 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a141031c0 0x7f9a14198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a137fe640 1 -- 172.21.15.57:0/2603379891 learned_addr learned my addr 172.21.15.57:0/2603379891 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.212+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1410c500 0x7f9a1419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a137fe640 1 -- 172.21.15.57:0/2603379891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a141031c0 msgr2=0x7f9a14198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a137fe640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a141031c0 0x7f9a14198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a137fe640 1 -- 172.21.15.57:0/2603379891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1410c500 msgr2=0x7f9a1419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a137fe640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1410c500 0x7f9a1419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a137fe640 1 -- 172.21.15.57:0/2603379891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1419d700 con 0x7f9a14103ae0 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a13fff640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a141031c0 0x7f9a14198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a137fe640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 0x7f9a14198c50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800e9f0 tx=0x7f9a0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a117fa640 1 -- 172.21.15.57:0/2603379891 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0800cd80 con 0x7f9a14103ae0 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a117fa640 1 -- 172.21.15.57:0/2603379891 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a080054d0 con 0x7f9a14103ae0 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1419d9f0 con 0x7f9a14103ae0 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a117fa640 1 -- 172.21.15.57:0/2603379891 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08010690 con 0x7f9a14103ae0 2024-04-21T08:50:53.216 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.213+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a141a5280 con 0x7f9a14103ae0 2024-04-21T08:50:53.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a117fa640 1 -- 172.21.15.57:0/2603379891 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9a08004e90 con 0x7f9a14103ae0 2024-04-21T08:50:53.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 Processor -- start 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 -- start start 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74013960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x563b73f781e0 con 0x563b7401ca80 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563b73f15390 con 0x563b74013580 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x563b73f741a0 con 0x563b74013ea0 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a117fa640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f99fc0a3d20 0x7f99fc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33082/0 (socket says 172.21.15.57:33082) 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a137fe640 1 -- 172.21.15.57:0/3883836083 learned_addr learned my addr 172.21.15.57:0/3883836083 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74013960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.216+0000 7f9a13fff640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a117fa640 1 -- 172.21.15.57:0/2603379891 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9a080d0fb0 con 0x7f9a14103ae0 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f99fc0a3d20 0x7f99fc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a13fff640 1 -- 172.21.15.57:0/3883836083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 msgr2=0x563b74013960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a13fff640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74013960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a13fff640 1 -- 172.21.15.57:0/3883836083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 msgr2=0x563b7401ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a13fff640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a13fff640 1 -- 172.21.15.57:0/3883836083 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b73d547d0 con 0x563b74013ea0 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a137fe640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74013960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a13fff640 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401c540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a040095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e3425792a41cb52 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:53.225 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f99fc0a3d20 0x7f99fc0a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009fd0 tx=0x7f9a0c004ab0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f99fa7fc640 1 -- 172.21.15.57:0/3883836083 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04043070 con 0x563b74013ea0 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f99fa7fc640 1 -- 172.21.15.57:0/3883836083 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0402c910 con 0x563b74013ea0 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.217+0000 7f99fa7fc640 1 -- 172.21.15.57:0/3883836083 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04003990 con 0x563b74013ea0 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3883836083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 msgr2=0x563b7401c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401c540 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a040095f0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3883836083 shutdown_connections 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401c540 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3883836083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74013960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3883836083 >> 172.21.15.57:0/3883836083 conn(0x563b7400dcb0 msgr2=0x563b7400e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3883836083 shutdown_connections 2024-04-21T08:50:53.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3883836083 wait complete. 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.218+0000 7f9a1d3a0740 1 -- start start 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 Processor -- start 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74019ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x563b7401f810 con 0x563b7401ca80 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563b73f312f0 con 0x563b74013580 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x563b7401f990 con 0x563b74013ea0 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a1d3a0740 10 client.0 _ll_register_callbacks cb 0x563b73f3e580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55922/0 (socket says 172.21.15.57:55922) 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33084/0 (socket says 172.21.15.57:33084) 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 -- 172.21.15.57:0/3775760855 learned_addr learned my addr 172.21.15.57:0/3775760855 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74019ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 -- 172.21.15.57:0/3775760855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 msgr2=0x563b7401a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 -- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 msgr2=0x563b74019ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74019ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a18d4f640 1 -- 172.21.15.57:0/3775760855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b73c8a3f0 con 0x563b7401ca80 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.219+0000 7f9a137fe640 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401a540 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009f40 tx=0x7f9a0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9929ea30bb97d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f99f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:53.228 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f99f37fe640 1 -- 172.21.15.57:0/3775760855 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c025070 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f99f37fe640 1 -- 172.21.15.57:0/3775760855 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9a0c010dc0 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a0402c910 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f99f37fe640 1 -- 172.21.15.57:0/3775760855 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a0c021440 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a1d3a0740 10 client.16578 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563b73c4a0f0 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f99f37fe640 1 -- 172.21.15.57:0/3775760855 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0402c910 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f99f37fe640 1 -- 172.21.15.57:0/3775760855 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9a0c029570 con 0x563b7401ca80 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a1d3a0740 10 client.16578 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a1d3a0740 10 client.16578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.220+0000 7f9a1d3a0740 20 client.16578 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.221+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b7402ca90 0x563b7402eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.221+0000 7f9a1d3a0740 4 client.16578 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:53.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.221+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b73f85c00 con 0x563b7402ca90 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.221+0000 7f9a13fff640 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b7402ca90 0x563b7402eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.221+0000 7f9a13fff640 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b7402ca90 0x563b7402eef0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52a7695a8f4f1f0a server_cookie=5f8f3d276bb44c89 in_seq=0 out_seq=0 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.221+0000 7f99f37fe640 10 client.16578 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.222+0000 7f99f37fe640 1 -- 172.21.15.57:0/3775760855 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b73f85c00 con 0x563b7402ca90 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.222+0000 7f99f37fe640 10 client.16578 handle_command_reply: tid=0 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.222+0000 7f9a1d3a0740 1 client.16578 shutdown 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 msgr2=0x563b7401a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401a540 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009f40 tx=0x7f9a0c004a40 comp rx=0 tx=0).stop 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 shutdown_connections 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b7402ca90 0x563b7402eef0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b7401ca80 0x563b7401a540 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7401a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f99f37fe640 0 client.16578 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/3775760855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013580 0x563b74019ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 >> 172.21.15.57:0/3775760855 conn(0x563b7400dcb0 msgr2=0x563b7400fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 shutdown_connections 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/3775760855 wait complete. 2024-04-21T08:50:53.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.223+0000 7f9a1d3a0740 20 client.16578 trim_cache size 0 max 16384 2024-04-21T08:50:53.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 Processor -- start 2024-04-21T08:50:53.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 -- start start 2024-04-21T08:50:53.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7401c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013ea0 0x563b7400e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b7401ca80 0x563b740286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141ad210 con 0x563b74013580 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141afa70 con 0x563b74013ea0 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141ad860 con 0x563b7401ca80 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7401c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7401c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33096/0 (socket says 172.21.15.57:33096) 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a13fff640 1 -- 172.21.15.57:0/133166739 learned_addr learned my addr 172.21.15.57:0/133166739 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.242+0000 7f9a137fe640 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013ea0 0x563b7400e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b7401ca80 0x563b740286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a13fff640 1 -- 172.21.15.57:0/133166739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b7401ca80 msgr2=0x563b740286d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a13fff640 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b7401ca80 0x563b740286d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a13fff640 1 -- 172.21.15.57:0/133166739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013ea0 msgr2=0x563b7400e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a13fff640 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013ea0 0x563b7400e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a13fff640 1 -- 172.21.15.57:0/133166739 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b7400d7e0 con 0x563b74013580 2024-04-21T08:50:53.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f9a13fff640 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7401c1f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a0402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac7216d03abc3cf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:53.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f99f9ffb640 1 -- 172.21.15.57:0/133166739 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04055070 con 0x563b74013580 2024-04-21T08:50:53.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f99f9ffb640 1 -- 172.21.15.57:0/133166739 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0402cc20 con 0x563b74013580 2024-04-21T08:50:53.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.243+0000 7f99f9ffb640 1 -- 172.21.15.57:0/133166739 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04043040 con 0x563b74013580 2024-04-21T08:50:53.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/133166739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 msgr2=0x563b7401c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7401c1f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9a04000c00 tx=0x7f9a0402e040 comp rx=0 tx=0).stop 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/133166739 shutdown_connections 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b7401ca80 0x563b740286d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b74013ea0 0x563b7400e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/133166739 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7401c1f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/133166739 >> 172.21.15.57:0/133166739 conn(0x563b7400dcb0 msgr2=0x563b74018390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/133166739 shutdown_connections 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.244+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/133166739 wait complete. 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 -- start start 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 Processor -- start 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7408b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7408b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b7401ca80 0x563b7401ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141adf30 con 0x563b74013580 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141af3a0 con 0x563b7401ca80 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a141ae600 con 0x563b74013ea0 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.245+0000 7f9a1d3a0740 10 client.0 _ll_register_callbacks cb 0x7f9a0c0135e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7408b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7408b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a13fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7408b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33104/0 (socket says 172.21.15.57:33104) 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a13fff640 1 -- 172.21.15.57:0/368730843 learned_addr learned my addr 172.21.15.57:0/368730843 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7408b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55932/0 (socket says 172.21.15.57:55932) 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a18d4f640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b7401ca80 0x563b7401ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 -- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b7401ca80 msgr2=0x563b7401ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b7401ca80 0x563b7401ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 -- 172.21.15.57:0/368730843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 msgr2=0x563b7408b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7408b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 -- 172.21.15.57:0/368730843 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b740206a0 con 0x563b74013ea0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f9a137fe640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7408b870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a0801c000 tx=0x7f9a0809c5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c6cf64db989ddd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f99f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.246+0000 7f99f37fe640 1 -- 172.21.15.57:0/368730843 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b1020 con 0x563b74013ea0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.247+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b73f7fa70 con 0x563b74013ea0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.247+0000 7f99f37fe640 1 -- 172.21.15.57:0/368730843 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9a080b11c0 con 0x563b74013ea0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.247+0000 7f99f37fe640 1 -- 172.21.15.57:0/368730843 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9a080b13c0 con 0x563b74013ea0 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.247+0000 7f9a1d3a0740 10 client.25954 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:53.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.247+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563b74092bb0 con 0x563b74013ea0 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f99f37fe640 1 -- 172.21.15.57:0/368730843 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563b73f7fa70 con 0x563b74013ea0 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f99f37fe640 1 -- 172.21.15.57:0/368730843 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9a0809cdf0 con 0x563b74013ea0 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a1d3a0740 10 client.25954 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a1d3a0740 10 client.25954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a1d3a0740 20 client.25954 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b74093330 0x563b74095750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a1d3a0740 4 client.25954 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b73efb590 con 0x563b74093330 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a13fff640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b74093330 0x563b74095750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f99f37fe640 10 client.25954 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:53.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.248+0000 7f9a13fff640 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b74093330 0x563b74095750 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb043528366de43b server_cookie=ce61c7dab18bf8d8 in_seq=0 out_seq=0 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f99f37fe640 1 -- 172.21.15.57:0/368730843 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x563b73efb590 con 0x563b74093330 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f99f37fe640 10 client.25954 handle_command_reply: tid=0 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 client.25954 shutdown 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 msgr2=0x563b7408b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7408b870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a0801c000 tx=0x7f9a0809c5b0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 shutdown_connections 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563b74093330 0x563b74095750 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b7401ca80 0x563b7401ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563b74013ea0 0x563b7408b870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f99f37fe640 0 client.25954 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 --2- 172.21.15.57:0/368730843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563b74013580 0x563b7408b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 >> 172.21.15.57:0/368730843 conn(0x563b7400dcb0 msgr2=0x563b74019a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 shutdown_connections 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 1 -- 172.21.15.57:0/368730843 wait complete. 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.250+0000 7f9a1d3a0740 20 client.25954 trim_cache size 0 max 16384 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (8578 inodes in the stack)", 2024-04-21T08:50:53.255 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout: "76c668b2-07a6-444b-87fc-10f228b0bd24": { 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "76c668b2-07a6-444b-87fc-10f228b0bd24", 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:50:53.256 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:50:53.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f99fc0a3d20 msgr2=0x7f99fc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f99fc0a3d20 0x7f99fc0a61e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c009fd0 tx=0x7f9a0c004ab0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 msgr2=0x7f9a14198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:53.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 0x7f9a14198c50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800e9f0 tx=0x7f9a0800edf0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 shutdown_connections 2024-04-21T08:50:53.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f99fc0a3d20 0x7f99fc0a61e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a1410c500 0x7f9a1419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a14103ae0 0x7f9a14198c50 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 --2- 172.21.15.57:0/2603379891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a141031c0 0x7f9a14198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 >> 172.21.15.57:0/2603379891 conn(0x7f9a14078000 msgr2=0x7f9a140fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 shutdown_connections 2024-04-21T08:50:53.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:50:53.251+0000 7f9a1a7b2640 1 -- 172.21.15.57:0/2603379891 wait complete. 2024-04-21T08:50:53.262 INFO:tasks.cephfs.filesystem:scrub status for tag:76c668b2-07a6-444b-87fc-10f228b0bd24 - {'path': '/', 'tag': '76c668b2-07a6-444b-87fc-10f228b0bd24', 'options': 'recursive,force'} 2024-04-21T08:50:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:54 smithi136 ceph-mon[31791]: pgmap v167: 129 pgs: 129 active+clean; 900 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.3 MiB/s wr, 2.33k op/s 2024-04-21T08:50:54.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:54 smithi129 ceph-mon[31272]: pgmap v167: 129 pgs: 129 active+clean; 900 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.3 MiB/s wr, 2.33k op/s 2024-04-21T08:50:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:54 smithi057 ceph-mon[27968]: pgmap v167: 129 pgs: 129 active+clean; 900 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 5.3 MiB/s wr, 2.33k op/s 2024-04-21T08:50:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:56 smithi136 ceph-mon[31791]: pgmap v168: 129 pgs: 129 active+clean; 900 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 2.12k op/s 2024-04-21T08:50:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:56 smithi129 ceph-mon[31272]: pgmap v168: 129 pgs: 129 active+clean; 900 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 2.12k op/s 2024-04-21T08:50:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:56 smithi057 ceph-mon[27968]: pgmap v168: 129 pgs: 129 active+clean; 900 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 2.12k op/s 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 -- 172.21.15.57:0/602825702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99241010f0 msgr2=0x7f99241014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 --2- 172.21.15.57:0/602825702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99241010f0 0x7f99241014d0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9914009900 tx=0x7f991402bc30 comp rx=0 tx=0).stop 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 -- 172.21.15.57:0/602825702 shutdown_connections 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 --2- 172.21.15.57:0/602825702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9924106170 0x7f992410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 --2- 172.21.15.57:0/602825702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924101aa0 0x7f9924105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 --2- 172.21.15.57:0/602825702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f99241010f0 0x7f99241014d0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 -- 172.21.15.57:0/602825702 >> 172.21.15.57:0/602825702 conn(0x7f9924077fd0 msgr2=0x7f99240783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 -- 172.21.15.57:0/602825702 shutdown_connections 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 -- 172.21.15.57:0/602825702 wait complete. 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.078+0000 7f992946d640 1 Processor -- start 2024-04-21T08:50:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 -- start start 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99241010f0 0x7f99241975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 0x7f9924197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992419be60 0x7f992419c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f99241110e0 con 0x7f9924106170 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9924110f60 con 0x7f99241010f0 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f992946d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9924111260 con 0x7f992419be60 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 0x7f9924197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f9922ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99241010f0 0x7f99241975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 0x7f9924197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33120/0 (socket says 172.21.15.57:33120) 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 -- 172.21.15.57:0/448142432 learned_addr learned my addr 172.21.15.57:0/448142432 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99237fe640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992419be60 0x7f992419c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 -- 172.21.15.57:0/448142432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992419be60 msgr2=0x7f992419c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992419be60 0x7f992419c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 -- 172.21.15.57:0/448142432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99241010f0 msgr2=0x7f99241975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99241010f0 0x7f99241975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 -- 172.21.15.57:0/448142432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992419c950 con 0x7f9924106170 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f9922ffd640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99241010f0 0x7f99241975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.079+0000 7f99227fc640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 0x7f9924197ae0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f991800ead0 tx=0x7f991800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.080+0000 7f990bfff640 1 -- 172.21.15.57:0/448142432 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991800bbd0 con 0x7f9924106170 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.080+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992419cc40 con 0x7f9924106170 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.080+0000 7f990bfff640 1 -- 172.21.15.57:0/448142432 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f991800bd70 con 0x7f9924106170 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.080+0000 7f990bfff640 1 -- 172.21.15.57:0/448142432 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918017410 con 0x7f9924106170 2024-04-21T08:50:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.080+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9924069970 con 0x7f9924106170 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 Processor -- start 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 -- start start 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f990bfff640 1 -- 172.21.15.57:0/448142432 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9918004e70 con 0x7f9924106170 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470ab620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b3470108d0 con 0x55b3470ac8d0 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b346fade80 con 0x55b3470b50e0 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f992c068740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b34700c890 con 0x55b3470abb60 2024-04-21T08:50:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f99237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f99237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470ab620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33130/0 (socket says 172.21.15.57:33130) 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f99237fe640 1 -- 172.21.15.57:0/1996170599 learned_addr learned my addr 172.21.15.57:0/1996170599 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f9922ffd640 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.083+0000 7f99227fc640 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99237fe640 1 -- 172.21.15.57:0/1996170599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 msgr2=0x55b3470abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99237fe640 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99237fe640 1 -- 172.21.15.57:0/1996170599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 msgr2=0x55b3470b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99237fe640 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99237fe640 1 -- 172.21.15.57:0/1996170599 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b346fcf320 con 0x55b3470ac8d0 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f9922ffd640 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f990bfff640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f990c0a3dc0 0x7f990c0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99237fe640 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470ab620 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f991c009f50 tx=0x7f991c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d7e60533212081 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f9922ffd640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f990c0a3dc0 0x7f990c0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f990bfff640 1 -- 172.21.15.57:0/448142432 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f99180d2280 con 0x7f9924106170 2024-04-21T08:50:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99097fa640 1 -- 172.21.15.57:0/1996170599 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c004d20 con 0x55b3470ac8d0 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99097fa640 1 -- 172.21.15.57:0/1996170599 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f991c004050 con 0x55b3470ac8d0 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.084+0000 7f99097fa640 1 -- 172.21.15.57:0/1996170599 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c005280 con 0x55b3470ac8d0 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 -- 172.21.15.57:0/1996170599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 msgr2=0x55b3470ab620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f9922ffd640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f990c0a3dc0 0x7f990c0a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9914006010 tx=0x7f9914005e40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470ab620 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f991c009f50 tx=0x7f991c0118d0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 -- 172.21.15.57:0/1996170599 shutdown_connections 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 --2- 172.21.15.57:0/1996170599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470ab620 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 -- 172.21.15.57:0/1996170599 >> 172.21.15.57:0/1996170599 conn(0x55b347072c90 msgr2=0x55b3470a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 -- 172.21.15.57:0/1996170599 shutdown_connections 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 -- 172.21.15.57:0/1996170599 wait complete. 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 -- start start 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.085+0000 7f992c068740 1 Processor -- start 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470abb60 0x55b3470b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470ac8d0 0x55b3470b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b3470b7e10 con 0x55b3470b50e0 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b346f98eb0 con 0x55b3470abb60 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b3470b7f90 con 0x55b3470ac8d0 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 10 client.0 _ll_register_callbacks cb 0x55b346fc5b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99227fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470ac8d0 0x55b3470b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f9922ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470abb60 0x55b3470b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470b23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33132/0 (socket says 172.21.15.57:33132) 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 -- 172.21.15.57:0/3939179359 learned_addr learned my addr 172.21.15.57:0/3939179359 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 -- 172.21.15.57:0/3939179359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470ac8d0 msgr2=0x55b3470b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470ac8d0 0x55b3470b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 -- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470abb60 msgr2=0x55b3470b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470abb60 0x55b3470b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 -- 172.21.15.57:0/3939179359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b346d23210 con 0x55b3470b50e0 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99227fc640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470ac8d0 0x55b3470b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470abb60 0x55b3470b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f99237fe640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470b23c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f991c005c40 tx=0x7f991c009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebfc03882cce836 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f9906ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f9906ff5640 1 -- 172.21.15.57:0/3939179359 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c028070 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f9906ff5640 1 -- 172.21.15.57:0/3939179359 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f991c004c00 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.086+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f991c004050 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f9906ff5640 1 -- 172.21.15.57:0/3939179359 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f991c004e00 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 10 client.16596 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b346def620 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f9906ff5640 1 -- 172.21.15.57:0/3939179359 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f991c004050 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f9906ff5640 1 -- 172.21.15.57:0/3939179359 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f991c024600 con 0x55b3470b50e0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 10 client.16596 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 10 client.16596 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 20 client.16596 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3470c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 4 client.16596 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b347024880 con 0x55b3470c50a0 2024-04-21T08:50:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.087+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3470c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.088+0000 7f9922ffd640 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3470c7490 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42579745ffed137 server_cookie=6fe292d5c8a3aedf in_seq=0 out_seq=0 2024-04-21T08:50:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.088+0000 7f9906ff5640 10 client.16596 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f9906ff5640 1 -- 172.21.15.57:0/3939179359 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b347024880 con 0x55b3470c50a0 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f9906ff5640 10 client.16596 handle_command_reply: tid=0 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 client.16596 shutdown 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 msgr2=0x55b3470b23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470b23c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f991c005c40 tx=0x7f991c009d20 comp rx=0 tx=0).stop 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 shutdown_connections 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3470c7490 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470b23c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f9906ff5640 0 client.16596 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470ac8d0 0x55b3470b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 --2- 172.21.15.57:0/3939179359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470abb60 0x55b3470b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 >> 172.21.15.57:0/3939179359 conn(0x55b347072c90 msgr2=0x55b3470a85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 shutdown_connections 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.089+0000 7f992c068740 1 -- 172.21.15.57:0/3939179359 wait complete. 2024-04-21T08:50:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.090+0000 7f992c068740 20 client.16596 trim_cache size 0 max 16384 2024-04-21T08:50:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 Processor -- start 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 -- start start 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470a8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f99241aaef0 con 0x55b3470ac8d0 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99241ad750 con 0x55b3470b50e0 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f992c068740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f99241ab540 con 0x55b3470abb60 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470a8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470a8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33138/0 (socket says 172.21.15.57:33138) 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 -- 172.21.15.57:0/1018773756 learned_addr learned my addr 172.21.15.57:0/1018773756 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f9922ffd640 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99237fe640 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 -- 172.21.15.57:0/1018773756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 msgr2=0x55b3470c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470c0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 -- 172.21.15.57:0/1018773756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 msgr2=0x55b3470b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 -- 172.21.15.57:0/1018773756 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3470aae30 con 0x55b3470ac8d0 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f99227fc640 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470a8340 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f99180cfc70 tx=0x7f99180d5ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fd90ce7c66d595 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.105+0000 7f9908ff9640 1 -- 172.21.15.57:0/1018773756 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991809a210 con 0x55b3470ac8d0 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f9908ff9640 1 -- 172.21.15.57:0/1018773756 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f991809df70 con 0x55b3470ac8d0 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f9908ff9640 1 -- 172.21.15.57:0/1018773756 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991809d8d0 con 0x55b3470ac8d0 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 -- 172.21.15.57:0/1018773756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 msgr2=0x55b3470a8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470a8340 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f99180cfc70 tx=0x7f99180d5ac0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 -- 172.21.15.57:0/1018773756 shutdown_connections 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470b50e0 0x55b3470b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470ac8d0 0x55b3470a8340 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 --2- 172.21.15.57:0/1018773756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 -- 172.21.15.57:0/1018773756 >> 172.21.15.57:0/1018773756 conn(0x55b347072c90 msgr2=0x55b3470a7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 -- 172.21.15.57:0/1018773756 shutdown_connections 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 -- 172.21.15.57:0/1018773756 wait complete. 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.106+0000 7f992c068740 1 -- start start 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 Processor -- start 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470ac8d0 0x55b3470aec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470af190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f99241abc10 con 0x55b3470b50e0 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99241ad080 con 0x55b3470ac8d0 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f99241ac2e0 con 0x55b3470abb60 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f992c068740 10 client.0 _ll_register_callbacks cb 0x7f99180d5ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:50:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f9922ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f9922ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:56014/0 (socket says 172.21.15.57:56014) 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33150/0 (socket says 172.21.15.57:33150) 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f9922ffd640 1 -- 172.21.15.57:0/1267382451 learned_addr learned my addr 172.21.15.57:0/1267382451 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99227fc640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470ac8d0 0x55b3470aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 -- 172.21.15.57:0/1267382451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 msgr2=0x55b3470ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 -- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470ac8d0 msgr2=0x55b3470aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470ac8d0 0x55b3470aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 -- 172.21.15.57:0/1267382451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3470bc410 con 0x55b3470b50e0 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f9922ffd640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f99237fe640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470af190 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f991c005d50 tx=0x7f991c004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e17925182b1c9a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:50:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.107+0000 7f9906ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f9906ff5640 1 -- 172.21.15.57:0/1267382451 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c028040 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f9906ff5640 1 -- 172.21.15.57:0/1267382451 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f991c003d40 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f991809df70 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f9906ff5640 1 -- 172.21.15.57:0/1267382451 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f991c009590 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 10 client.16602 fetch_fsmap learned FSMap version 21 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b3470bd720 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f9906ff5640 1 -- 172.21.15.57:0/1267382451 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f991809df70 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f9906ff5640 1 -- 172.21.15.57:0/1267382451 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f991c0054b0 con 0x55b3470b50e0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 10 client.16602 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 10 client.16602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 20 client.16602 populate_metadata read hostname 'smithi057' 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3471360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 4 client.16602 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b346fb03d0 con 0x55b3470c50a0 2024-04-21T08:50:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.108+0000 7f9922ffd640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3471360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:50:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.109+0000 7f9922ffd640 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3471360d0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3034d6d8f937da79 server_cookie=f05eaaadecba77e in_seq=0 out_seq=0 2024-04-21T08:50:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:50:58.109+0000 7f9906ff5640 10 client.16602 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:50:58.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:50:58 smithi136 ceph-mon[31791]: pgmap v169: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 2.71k op/s 2024-04-21T08:50:58.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:50:58 smithi129 ceph-mon[31272]: pgmap v169: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 2.71k op/s 2024-04-21T08:50:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:50:58 smithi057 ceph-mon[27968]: pgmap v169: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.0 MiB/s wr, 2.71k op/s 2024-04-21T08:51:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:00 smithi136 ceph-mon[31791]: pgmap v170: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 1.35k op/s 2024-04-21T08:51:00.635 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:00 smithi129 ceph-mon[31272]: pgmap v170: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 1.35k op/s 2024-04-21T08:51:00.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:00 smithi057 ceph-mon[27968]: pgmap v170: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 1.35k op/s 2024-04-21T08:51:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:02 smithi136 ceph-mon[31791]: pgmap v171: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 1.35k op/s 2024-04-21T08:51:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:02 smithi129 ceph-mon[31272]: pgmap v171: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 1.35k op/s 2024-04-21T08:51:02.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:02 smithi057 ceph-mon[27968]: pgmap v171: 129 pgs: 129 active+clean; 964 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 1.35k op/s 2024-04-21T08:51:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:04 smithi129 ceph-mon[31272]: pgmap v172: 129 pgs: 129 active+clean; 1019 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 2.31k op/s 2024-04-21T08:51:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:04 smithi057 ceph-mon[27968]: pgmap v172: 129 pgs: 129 active+clean; 1019 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 2.31k op/s 2024-04-21T08:51:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:04 smithi136 ceph-mon[31791]: pgmap v172: 129 pgs: 129 active+clean; 1019 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 11 MiB/s wr, 2.31k op/s 2024-04-21T08:51:05.380 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.380+0000 7f9906ff5640 1 -- 172.21.15.57:0/1267382451 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b346fb03d0 con 0x55b3470c50a0 2024-04-21T08:51:05.380 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.380+0000 7f9906ff5640 10 client.16602 handle_command_reply: tid=0 2024-04-21T08:51:05.381 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.380+0000 7f992c068740 1 client.16602 shutdown 2024-04-21T08:51:05.381 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 msgr2=0x55b3470af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:05.381 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f992c068740 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470af190 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f991c005d50 tx=0x7f991c004bd0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 shutdown_connections 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f992c068740 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b3470c50a0 0x55b3471360d0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f992c068740 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b3470b50e0 0x55b3470af190 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f9906ff5640 0 client.16602 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.381+0000 7f992c068740 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b3470ac8d0 0x55b3470aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.382+0000 7f992c068740 1 --2- 172.21.15.57:0/1267382451 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b3470abb60 0x55b3470ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.382+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 >> 172.21.15.57:0/1267382451 conn(0x55b347072c90 msgr2=0x55b3470a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:05.382 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.382+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 shutdown_connections 2024-04-21T08:51:05.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.382+0000 7f992c068740 1 -- 172.21.15.57:0/1267382451 wait complete. 2024-04-21T08:51:05.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.382+0000 7f992c068740 20 client.16602 trim_cache size 0 max 16384 2024-04-21T08:51:05.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:51:05.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:51:05.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:51:05.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f990c0a3dc0 msgr2=0x7f990c0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f990c0a3dc0 0x7f990c0a6280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9914006010 tx=0x7f9914005e40 comp rx=0 tx=0).stop 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 msgr2=0x7f9924197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 0x7f9924197ae0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f991800ead0 tx=0x7f991800ef90 comp rx=0 tx=0).stop 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 shutdown_connections 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f990c0a3dc0 0x7f990c0a6280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992419be60 0x7f992419c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9924106170 0x7f9924197ae0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 --2- 172.21.15.57:0/448142432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99241010f0 0x7f99241975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 >> 172.21.15.57:0/448142432 conn(0x7f9924077fd0 msgr2=0x7f99240fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 shutdown_connections 2024-04-21T08:51:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:05.383+0000 7f992946d640 1 -- 172.21.15.57:0/448142432 wait complete. 2024-04-21T08:51:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:06 smithi129 ceph-mon[31272]: pgmap v173: 129 pgs: 129 active+clean; 1019 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 10 MiB/s wr, 1.54k op/s 2024-04-21T08:51:06.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:06 smithi057 ceph-mon[27968]: pgmap v173: 129 pgs: 129 active+clean; 1019 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 10 MiB/s wr, 1.54k op/s 2024-04-21T08:51:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:06 smithi136 ceph-mon[31791]: pgmap v173: 129 pgs: 129 active+clean; 1019 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 10 MiB/s wr, 1.54k op/s 2024-04-21T08:51:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:08 smithi129 ceph-mon[31272]: pgmap v174: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 2.80k op/s 2024-04-21T08:51:08.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:08 smithi057 ceph-mon[27968]: pgmap v174: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 2.80k op/s 2024-04-21T08:51:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:08 smithi136 ceph-mon[31791]: pgmap v174: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 12 MiB/s wr, 2.80k op/s 2024-04-21T08:51:10.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:10 smithi129 ceph-mon[31272]: pgmap v175: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 2.22k op/s 2024-04-21T08:51:10.881 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:10 smithi057 ceph-mon[27968]: pgmap v175: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 2.22k op/s 2024-04-21T08:51:10.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:10 smithi136 ceph-mon[31791]: pgmap v175: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 2.22k op/s 2024-04-21T08:51:12.682 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:12 smithi129 ceph-mon[31272]: pgmap v176: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 2.22k op/s 2024-04-21T08:51:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:12 smithi057 ceph-mon[27968]: pgmap v176: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 2.22k op/s 2024-04-21T08:51:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:12 smithi136 ceph-mon[31791]: pgmap v176: 129 pgs: 129 active+clean; 1006 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.1 MiB/s wr, 2.22k op/s 2024-04-21T08:51:13.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:13 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:13.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:13 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:13.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:13 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:14 smithi129 ceph-mon[31272]: pgmap v177: 129 pgs: 129 active+clean; 1017 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 2.93k op/s 2024-04-21T08:51:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:14 smithi057 ceph-mon[27968]: pgmap v177: 129 pgs: 129 active+clean; 1017 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 2.93k op/s 2024-04-21T08:51:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:14 smithi136 ceph-mon[31791]: pgmap v177: 129 pgs: 129 active+clean; 1017 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 7.0 MiB/s wr, 2.93k op/s 2024-04-21T08:51:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:16 smithi129 ceph-mon[31272]: pgmap v178: 129 pgs: 129 active+clean; 1017 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 1.97k op/s 2024-04-21T08:51:16.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:16 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:51:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:16 smithi057 ceph-mon[27968]: pgmap v178: 129 pgs: 129 active+clean; 1017 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 1.97k op/s 2024-04-21T08:51:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:16 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:51:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:16 smithi136 ceph-mon[31791]: pgmap v178: 129 pgs: 129 active+clean; 1017 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 1.97k op/s 2024-04-21T08:51:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:16 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:51:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 -- 172.21.15.57:0/3893302865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa180102b80 msgr2=0x7fa180106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 --2- 172.21.15.57:0/3893302865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa180102b80 0x7fa180106bc0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa170009900 tx=0x7fa17002bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 -- 172.21.15.57:0/3893302865 shutdown_connections 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 --2- 172.21.15.57:0/3893302865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1801072a0 0x7fa18010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 --2- 172.21.15.57:0/3893302865 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa180102b80 0x7fa180106bc0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 --2- 172.21.15.57:0/3893302865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1801021d0 0x7fa1801025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.480+0000 7fa186aba640 1 -- 172.21.15.57:0/3893302865 >> 172.21.15.57:0/3893302865 conn(0x7fa1800793e0 msgr2=0x7fa1800797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 -- 172.21.15.57:0/3893302865 shutdown_connections 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 -- 172.21.15.57:0/3893302865 wait complete. 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 Processor -- start 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 -- start start 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 0x7fa180074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1801072a0 0x7fa180074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa180074cd0 0x7fa1801a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa180112210 con 0x7fa1801021d0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa180112090 con 0x7fa180074cd0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa186aba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa180112390 con 0x7fa1801072a0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa184856640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 0x7fa180074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.481+0000 7fa184856640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 0x7fa180074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36426/0 (socket says 172.21.15.57:36426) 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 -- 172.21.15.57:0/857946369 learned_addr learned my addr 172.21.15.57:0/857946369 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa185057640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa180074cd0 0x7fa1801a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa177fff640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1801072a0 0x7fa180074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 -- 172.21.15.57:0/857946369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1801072a0 msgr2=0x7fa180074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1801072a0 0x7fa180074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 -- 172.21.15.57:0/857946369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa180074cd0 msgr2=0x7fa1801a68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa180074cd0 0x7fa1801a68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 -- 172.21.15.57:0/857946369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1801a6f40 con 0x7fa1801021d0 2024-04-21T08:51:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa185057640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa180074cd0 0x7fa1801a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa184856640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 0x7fa180074250 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9f0 tx=0x7fa17c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa175ffb640 1 -- 172.21.15.57:0/857946369 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c00cd80 con 0x7fa1801021d0 2024-04-21T08:51:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa175ffb640 1 -- 172.21.15.57:0/857946369 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17c0108e0 con 0x7fa1801021d0 2024-04-21T08:51:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1801a7230 con 0x7fa1801021d0 2024-04-21T08:51:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa175ffb640 1 -- 172.21.15.57:0/857946369 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17c018980 con 0x7fa1801021d0 2024-04-21T08:51:18.486 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.482+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1801a7750 con 0x7fa1801021d0 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.485+0000 7fa175ffb640 1 -- 172.21.15.57:0/857946369 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa17c010430 con 0x7fa1801021d0 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 Processor -- start 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 -- start start 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02c9a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02c9afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa175ffb640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1680a3dc0 0x7fa1680a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ae02bff8d0 con 0x55ae02ca40e0 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae02b9ce80 con 0x55ae02c9b8d0 2024-04-21T08:51:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa189786740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ae02bfb890 con 0x55ae02c9ab60 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa185057640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1680a3dc0 0x7fa1680a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36440/0 (socket says 172.21.15.57:36440) 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 -- 172.21.15.57:0/89119837 learned_addr learned my addr 172.21.15.57:0/89119837 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa175ffb640 1 -- 172.21.15.57:0/857946369 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa17c0d0ef0 con 0x7fa1801021d0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa184856640 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02c9afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa185057640 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02c9a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 -- 172.21.15.57:0/89119837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 msgr2=0x55ae02c9afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02c9afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 -- 172.21.15.57:0/89119837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 msgr2=0x55ae02c9a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02c9a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.486+0000 7fa177fff640 1 -- 172.21.15.57:0/89119837 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae02bbe320 con 0x55ae02ca40e0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.487+0000 7fa185057640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1680a3dc0 0x7fa1680a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa178005fd0 tx=0x7fa178011690 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.487+0000 7fa177fff640 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca64d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa170000c00 tx=0x7fa170005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929329a52cdbda96 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.487+0000 7fa166ffd640 1 -- 172.21.15.57:0/89119837 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170044020 con 0x55ae02ca40e0 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.487+0000 7fa166ffd640 1 -- 172.21.15.57:0/89119837 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17002cc90 con 0x55ae02ca40e0 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.487+0000 7fa166ffd640 1 -- 172.21.15.57:0/89119837 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700055b0 con 0x55ae02ca40e0 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- 172.21.15.57:0/89119837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 msgr2=0x55ae02ca64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca64d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa170000c00 tx=0x7fa170005dc0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- 172.21.15.57:0/89119837 shutdown_connections 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca64d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02c9afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- 172.21.15.57:0/89119837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02c9a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- 172.21.15.57:0/89119837 >> 172.21.15.57:0/89119837 conn(0x55ae02c61c90 msgr2=0x55ae02c95850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- 172.21.15.57:0/89119837 shutdown_connections 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- 172.21.15.57:0/89119837 wait complete. 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- start start 2024-04-21T08:51:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 Processor -- start 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02ca3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ae02ca6db0 con 0x55ae02ca40e0 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae02b87eb0 con 0x55ae02c9ab60 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.488+0000 7fa189786740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ae02ca6f30 con 0x55ae02c9b8d0 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa189786740 10 client.0 _ll_register_callbacks cb 0x55ae02bb4b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa184856640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02ca3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36442/0 (socket says 172.21.15.57:36442) 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 -- 172.21.15.57:0/1324400232 learned_addr learned my addr 172.21.15.57:0/1324400232 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa177fff640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 -- 172.21.15.57:0/1324400232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 msgr2=0x55ae02ca0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 -- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 msgr2=0x55ae02ca3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02ca3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 -- 172.21.15.57:0/1324400232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae02912210 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa184856640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02ca3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa185057640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca1320 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa178003e40 tx=0x7fa17801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21c41d711df64bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa15ffff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa15ffff640 1 -- 172.21.15.57:0/1324400232 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178024070 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa15ffff640 1 -- 172.21.15.57:0/1324400232 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa17801f660 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa17002cc90 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa15ffff640 1 -- 172.21.15.57:0/1324400232 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa178033050 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa189786740 10 client.16614 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.489+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ae029de620 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa15ffff640 1 -- 172.21.15.57:0/1324400232 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa17002cc90 con 0x55ae02ca40e0 2024-04-21T08:51:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa15ffff640 1 -- 172.21.15.57:0/1324400232 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa1780286f0 con 0x55ae02ca40e0 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa189786740 10 client.16614 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa189786740 10 client.16614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa189786740 20 client.16614 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa189786740 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02cb40f0 0x55ae02cb6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa189786740 4 client.16614 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae02c13880 con 0x55ae02cb40f0 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa184856640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02cb40f0 0x55ae02cb6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa15ffff640 10 client.16614 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.490+0000 7fa184856640 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02cb40f0 0x55ae02cb6500 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b77fd62775a5c74 server_cookie=91a1a41a4fb56016 in_seq=0 out_seq=0 2024-04-21T08:51:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa15ffff640 1 -- 172.21.15.57:0/1324400232 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae02c13880 con 0x55ae02cb40f0 2024-04-21T08:51:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa15ffff640 10 client.16614 handle_command_reply: tid=0 2024-04-21T08:51:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 client.16614 shutdown 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 msgr2=0x55ae02ca1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca1320 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa178003e40 tx=0x7fa17801d000 comp rx=0 tx=0).stop 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 shutdown_connections 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02cb40f0 0x55ae02cb6500 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca1320 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 --2- 172.21.15.57:0/1324400232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02ca3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa15ffff640 0 client.16614 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 >> 172.21.15.57:0/1324400232 conn(0x55ae02c61c90 msgr2=0x55ae02c975a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 shutdown_connections 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 1 -- 172.21.15.57:0/1324400232 wait complete. 2024-04-21T08:51:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.492+0000 7fa189786740 20 client.16614 trim_cache size 0 max 16384 2024-04-21T08:51:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.507+0000 7fa189786740 1 Processor -- start 2024-04-21T08:51:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.507+0000 7fa189786740 1 -- start start 2024-04-21T08:51:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa189786740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02cafcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa189786740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa189786740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa189786740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801af2f0 con 0x55ae02ca40e0 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa189786740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801b1b50 con 0x55ae02c9b8d0 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa189786740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801af940 con 0x55ae02c9ab60 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36452/0 (socket says 172.21.15.57:36452) 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 -- 172.21.15.57:0/1463417606 learned_addr learned my addr 172.21.15.57:0/1463417606 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa177fff640 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa184856640 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02cafcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 -- 172.21.15.57:0/1463417606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 msgr2=0x55ae02cafcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02cafcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 -- 172.21.15.57:0/1463417606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 msgr2=0x55ae02ca34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 -- 172.21.15.57:0/1463417606 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae02c99e30 con 0x55ae02ca40e0 2024-04-21T08:51:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa177fff640 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.508+0000 7fa185057640 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca3a30 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa178028d00 tx=0x7fa178021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebec6636aaa73363 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa1667fc640 1 -- 172.21.15.57:0/1463417606 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17803b020 con 0x55ae02ca40e0 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa1667fc640 1 -- 172.21.15.57:0/1463417606 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa178011c90 con 0x55ae02ca40e0 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa1667fc640 1 -- 172.21.15.57:0/1463417606 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178024040 con 0x55ae02ca40e0 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 -- 172.21.15.57:0/1463417606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 msgr2=0x55ae02ca3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca3a30 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa178028d00 tx=0x7fa178021000 comp rx=0 tx=0).stop 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 -- 172.21.15.57:0/1463417606 shutdown_connections 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02ca3a30 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9b8d0 0x55ae02ca34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 --2- 172.21.15.57:0/1463417606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9ab60 0x55ae02cafcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 -- 172.21.15.57:0/1463417606 >> 172.21.15.57:0/1463417606 conn(0x55ae02c61c90 msgr2=0x55ae02c95380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 -- 172.21.15.57:0/1463417606 shutdown_connections 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.509+0000 7fa189786740 1 -- 172.21.15.57:0/1463417606 wait complete. 2024-04-21T08:51:18.512 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 -- start start 2024-04-21T08:51:18.513 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 Processor -- start 2024-04-21T08:51:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02cac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02cac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.515 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02c9e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801b0010 con 0x55ae02ca40e0 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801b1480 con 0x55ae02c9ab60 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801b06e0 con 0x55ae02c9b8d0 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa189786740 10 client.0 _ll_register_callbacks cb 0x7fa178003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa185057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02c9e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa184856640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02cac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa185057640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02c9e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36460/0 (socket says 172.21.15.57:36460) 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa184856640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02cac190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44208/0 (socket says 172.21.15.57:44208) 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa185057640 1 -- 172.21.15.57:0/4052444003 learned_addr learned my addr 172.21.15.57:0/4052444003 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.510+0000 7fa177fff640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02cac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa185057640 1 -- 172.21.15.57:0/4052444003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 msgr2=0x55ae02cac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa185057640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02cac6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.516 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa185057640 1 -- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 msgr2=0x55ae02cac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa185057640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02cac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa185057640 1 -- 172.21.15.57:0/4052444003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae02c9ea20 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa184856640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02cac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa185057640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02c9e1f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa1780048d0 tx=0x7fa178004ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffa878e71cb8a55 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa15ffff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa15ffff640 1 -- 172.21.15.57:0/4052444003 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178045060 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa15ffff640 1 -- 172.21.15.57:0/4052444003 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fa178028610 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa178011c90 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa15ffff640 1 -- 172.21.15.57:0/4052444003 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa17804c050 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa189786740 10 client.16620 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ae02ca52a0 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa15ffff640 1 -- 172.21.15.57:0/4052444003 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa178011c90 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.511+0000 7fa15ffff640 1 -- 172.21.15.57:0/4052444003 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fa178005040 con 0x55ae02ca40e0 2024-04-21T08:51:18.517 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa189786740 10 client.16620 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa189786740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa189786740 20 client.16620 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa189786740 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02d1a950 0x55ae02d1cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa189786740 4 client.16620 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae02b9f3d0 con 0x55ae02d1a950 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa184856640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02d1a950 0x55ae02d1cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa184856640 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02d1a950 0x55ae02d1cd70 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=929753b1cf94a95d server_cookie=8bc47a41715d3658 in_seq=0 out_seq=0 2024-04-21T08:51:18.518 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:18.512+0000 7fa15ffff640 10 client.16620 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:18 smithi129 ceph-mon[31272]: pgmap v179: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.3 MiB/s wr, 2.42k op/s 2024-04-21T08:51:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:18 smithi057 ceph-mon[27968]: pgmap v179: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.3 MiB/s wr, 2.42k op/s 2024-04-21T08:51:18.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:18 smithi136 ceph-mon[31791]: pgmap v179: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 8.3 MiB/s wr, 2.42k op/s 2024-04-21T08:51:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:20 smithi129 ceph-mon[31272]: pgmap v180: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.1 MiB/s wr, 1.16k op/s 2024-04-21T08:51:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:20 smithi057 ceph-mon[27968]: pgmap v180: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.1 MiB/s wr, 1.16k op/s 2024-04-21T08:51:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:20 smithi136 ceph-mon[31791]: pgmap v180: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.1 MiB/s wr, 1.16k op/s 2024-04-21T08:51:22.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:22 smithi129 ceph-mon[31272]: pgmap v181: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.1 MiB/s wr, 1.16k op/s 2024-04-21T08:51:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:22 smithi057 ceph-mon[27968]: pgmap v181: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.1 MiB/s wr, 1.16k op/s 2024-04-21T08:51:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:22 smithi136 ceph-mon[31791]: pgmap v181: 129 pgs: 129 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.1 MiB/s wr, 1.16k op/s 2024-04-21T08:51:23.263 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:23.263 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:23.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.466+0000 7f9ee58b2640 1 -- 172.21.15.57:0/742103444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee011b730 msgr2=0x7f9ee011db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.466+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/742103444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee011b730 0x7f9ee011db20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009cd0 tx=0x7f9ed802c120 comp rx=0 tx=0).stop 2024-04-21T08:51:23.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.472+0000 7f9ee58b2640 1 -- 172.21.15.57:0/742103444 shutdown_connections 2024-04-21T08:51:23.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.472+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/742103444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee011b730 0x7f9ee011db20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.472+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/742103444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee010e7b0 0x7f9ee010ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.472+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/742103444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ee01106d0 0x7f9ee010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.472+0000 7f9ee58b2640 1 -- 172.21.15.57:0/742103444 >> 172.21.15.57:0/742103444 conn(0x7f9ee006d070 msgr2=0x7f9ee006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.475+0000 7f9ee58b2640 1 -- 172.21.15.57:0/742103444 shutdown_connections 2024-04-21T08:51:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.479+0000 7f9ee58b2640 1 -- 172.21.15.57:0/742103444 wait complete. 2024-04-21T08:51:23.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.479+0000 7f9ee58b2640 1 Processor -- start 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 -- start start 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee010e7b0 0x7f9ee01194e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 0x7f9ee0119a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ee011b730 0x7f9ee0112710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0121060 con 0x7f9ee010e7b0 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0120ee0 con 0x7f9ee01106d0 2024-04-21T08:51:23.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.481+0000 7f9ee58b2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01211e0 con 0x7f9ee011b730 2024-04-21T08:51:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.483+0000 7f9ede7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 0x7f9ee0119a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.483+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ee011b730 0x7f9ee0112710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.483+0000 7f9ede7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 0x7f9ee0119a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:52248/0 (socket says 172.21.15.57:52248) 2024-04-21T08:51:23.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.483+0000 7f9ede7fc640 1 -- 172.21.15.57:0/2994565346 learned_addr learned my addr 172.21.15.57:0/2994565346 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.484+0000 7f9ede7fc640 1 -- 172.21.15.57:0/2994565346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ee011b730 msgr2=0x7f9ee0112710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.484+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ee011b730 0x7f9ee0112710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.484+0000 7f9ede7fc640 1 -- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee010e7b0 msgr2=0x7f9ee01194e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.484+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee010e7b0 0x7f9ee01194e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.484+0000 7f9ede7fc640 1 -- 172.21.15.57:0/2994565346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee0112fd0 con 0x7f9ee01106d0 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.485+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 0x7f9ee0119a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000d950 tx=0x7f9ed000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.486+0000 7f9ee48b0640 1 -- 172.21.15.57:0/2994565346 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0017070 con 0x7f9ee01106d0 2024-04-21T08:51:23.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.486+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee01132c0 con 0x7f9ee01106d0 2024-04-21T08:51:23.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.486+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee01172a0 con 0x7f9ee01106d0 2024-04-21T08:51:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 Processor -- start 2024-04-21T08:51:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 -- start start 2024-04-21T08:51:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 0x55e78ac49960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac52540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac54e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e78abae1e0 con 0x55e78ac49ea0 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e78ab4b390 con 0x55e78ac52a80 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee8576740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e78abaa1a0 con 0x55e78ac49580 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee48b0640 1 -- 172.21.15.57:0/2994565346 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed0009c30 con 0x7f9ee01106d0 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.490+0000 7f9ee48b0640 1 -- 172.21.15.57:0/2994565346 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed000f620 con 0x7f9ee01106d0 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.491+0000 7f9ede7fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac54e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.491+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 0x55e78ac49960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.491+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 0x55e78ac49960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:52032/0 (socket says 172.21.15.57:52032) 2024-04-21T08:51:23.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.491+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3771846818 learned_addr learned my addr 172.21.15.57:0/3771846818 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.493+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3771846818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 msgr2=0x55e78ac54e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.493+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac54e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.493+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3771846818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 msgr2=0x55e78ac52540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.493+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac52540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.493+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3771846818 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e78a98a7d0 con 0x55e78ac49580 2024-04-21T08:51:23.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.496+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 0x55e78ac49960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009cd0 tx=0x7f9ed8032940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc7df300afc533f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.496+0000 7f9ee48b0640 1 -- 172.21.15.57:0/2994565346 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ed000f980 con 0x7f9ee01106d0 2024-04-21T08:51:23.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.497+0000 7f9ec5ffb640 1 -- 172.21.15.57:0/3771846818 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8039050 con 0x55e78ac49580 2024-04-21T08:51:23.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.498+0000 7f9ee48b0640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9ec80a3f30 0x7f9ec80a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.498+0000 7f9ee48b0640 1 -- 172.21.15.57:0/2994565346 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9ed00d74f0 con 0x7f9ee01106d0 2024-04-21T08:51:23.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.499+0000 7f9ec5ffb640 1 -- 172.21.15.57:0/3771846818 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed8045070 con 0x55e78ac49580 2024-04-21T08:51:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.500+0000 7f9ee8576740 1 -- 172.21.15.57:0/3771846818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 msgr2=0x55e78ac49960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:23.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.500+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 0x55e78ac49960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009cd0 tx=0x7f9ed8032940 comp rx=0 tx=0).stop 2024-04-21T08:51:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.502+0000 7f9ee8576740 1 -- 172.21.15.57:0/3771846818 shutdown_connections 2024-04-21T08:51:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.502+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac54e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.502+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac52540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.502+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3771846818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49580 0x55e78ac49960 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.502+0000 7f9ee8576740 1 -- 172.21.15.57:0/3771846818 >> 172.21.15.57:0/3771846818 conn(0x55e78ac43cb0 msgr2=0x55e78ac44090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:23.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.505+0000 7f9edeffd640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9ec80a3f30 0x7f9ec80a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.510 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.508+0000 7f9ee8576740 1 -- 172.21.15.57:0/3771846818 shutdown_connections 2024-04-21T08:51:23.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.513+0000 7f9ee8576740 1 -- 172.21.15.57:0/3771846818 wait complete. 2024-04-21T08:51:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.513+0000 7f9ee8576740 1 -- start start 2024-04-21T08:51:23.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.513+0000 7f9ee8576740 1 Processor -- start 2024-04-21T08:51:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49580 0x55e78ac4f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac4fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e78ac558b0 con 0x55e78ac49580 2024-04-21T08:51:23.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e78ab672f0 con 0x55e78ac52a80 2024-04-21T08:51:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e78ac55a30 con 0x55e78ac49ea0 2024-04-21T08:51:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.514+0000 7f9ee8576740 10 client.0 _ll_register_callbacks cb 0x55e78ab74580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.515+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.516+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:52256/0 (socket says 172.21.15.57:52256) 2024-04-21T08:51:23.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.516+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3975252131 learned_addr learned my addr 172.21.15.57:0/3975252131 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:23.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.517+0000 7f9edeffd640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9ec80a3f30 0x7f9ec80a63f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55e78ac49050 tx=0x7f9ed4006bd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.518+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac4fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.520+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3975252131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 msgr2=0x55e78ac4fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:23.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.520+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac4fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.520+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3975252131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49580 msgr2=0x55e78ac4f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:23.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.520+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49580 0x55e78ac4f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:23.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.520+0000 7f9edf7fe640 1 -- 172.21.15.57:0/3975252131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78a8c03f0 con 0x55e78ac52a80 2024-04-21T08:51:23.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.524+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac503a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9ed802e010 tx=0x7f9ed8009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=708f7fc4f7de79a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:23.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.524+0000 7f9ebeffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:23.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.528+0000 7f9ebeffd640 1 -- 172.21.15.57:0/3975252131 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8032340 con 0x55e78ac52a80 2024-04-21T08:51:23.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.528+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ed8045070 con 0x55e78ac52a80 2024-04-21T08:51:23.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.528+0000 7f9ebeffd640 1 -- 172.21.15.57:0/3975252131 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9ed80324e0 con 0x55e78ac52a80 2024-04-21T08:51:23.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.534+0000 7f9ebeffd640 1 -- 172.21.15.57:0/3975252131 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9ed8039020 con 0x55e78ac52a80 2024-04-21T08:51:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.536+0000 7f9ee8576740 10 client.26102 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:23.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.536+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e78a8800f0 con 0x55e78ac52a80 2024-04-21T08:51:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.541+0000 7f9ebeffd640 1 -- 172.21.15.57:0/3975252131 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed8045070 con 0x55e78ac52a80 2024-04-21T08:51:23.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.541+0000 7f9ebeffd640 1 -- 172.21.15.57:0/3975252131 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9ed8049500 con 0x55e78ac52a80 2024-04-21T08:51:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.543+0000 7f9ee8576740 10 client.26102 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.543+0000 7f9ee8576740 10 client.26102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.543+0000 7f9ee8576740 20 client.26102 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:23.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.543+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78ac62860 0x55e78ac64c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.543+0000 7f9ee8576740 4 client.26102 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:23.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.543+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e78abbbc00 con 0x55e78ac62860 2024-04-21T08:51:23.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.552+0000 7f9edeffd640 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78ac62860 0x55e78ac64c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.556+0000 7f9edeffd640 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78ac62860 0x55e78ac64c50 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c6e1a410f4480d8 server_cookie=eca0ebce3f325867 in_seq=0 out_seq=0 2024-04-21T08:51:23.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:23.557+0000 7f9ebeffd640 10 client.26102 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:24.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa15ffff640 1 -- 172.21.15.57:0/4052444003 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ae02b9f3d0 con 0x55ae02d1a950 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa15ffff640 10 client.16620 handle_command_reply: tid=0 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 client.16620 shutdown 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 msgr2=0x55ae02c9e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02c9e1f0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa1780048d0 tx=0x7fa178004ea0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 shutdown_connections 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae02d1a950 0x55ae02d1cd70 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae02ca40e0 0x55ae02c9e1f0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae02c9b8d0 0x55ae02cac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 --2- 172.21.15.57:0/4052444003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae02c9ab60 0x55ae02cac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa15ffff640 0 client.16620 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 >> 172.21.15.57:0/4052444003 conn(0x55ae02c61c90 msgr2=0x55ae02c9f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 shutdown_connections 2024-04-21T08:51:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 1 -- 172.21.15.57:0/4052444003 wait complete. 2024-04-21T08:51:24.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.553+0000 7fa189786740 20 client.16620 trim_cache size 0 max 16384 2024-04-21T08:51:24.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ebeffd640 1 -- 172.21.15.57:0/3975252131 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e78abbbc00 con 0x55e78ac62860 2024-04-21T08:51:24.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ebeffd640 10 client.26102 handle_command_reply: tid=0 2024-04-21T08:51:24.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:51:24.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:51:24.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:51:24.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 client.26102 shutdown 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 msgr2=0x55e78ac503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac503a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9ed802e010 tx=0x7f9ed8009860 comp rx=0 tx=0).stop 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 shutdown_connections 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78ac62860 0x55e78ac64c50 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac52a80 0x55e78ac503a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac4fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 --2- 172.21.15.57:0/3975252131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49580 0x55e78ac4f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ebeffd640 0 client.26102 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 >> 172.21.15.57:0/3975252131 conn(0x55e78ac43cb0 msgr2=0x55e78ac45a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 shutdown_connections 2024-04-21T08:51:24.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 1 -- 172.21.15.57:0/3975252131 wait complete. 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1680a3dc0 msgr2=0x7fa1680a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1680a3dc0 0x7fa1680a6280 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa178005fd0 tx=0x7fa178011690 comp rx=0 tx=0).stop 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 msgr2=0x7fa180074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 0x7fa180074250 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9f0 tx=0x7fa17c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 shutdown_connections 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa1680a3dc0 0x7fa1680a6280 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa180074cd0 0x7fa1801a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.558 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1801072a0 0x7fa180074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 --2- 172.21.15.57:0/857946369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa1801021d0 0x7fa180074250 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 >> 172.21.15.57:0/857946369 conn(0x7fa1800793e0 msgr2=0x7fa1800ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.554+0000 7f9ee8576740 20 client.26102 trim_cache size 0 max 16384 2024-04-21T08:51:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 shutdown_connections 2024-04-21T08:51:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:24.554+0000 7fa186aba640 1 -- 172.21.15.57:0/857946369 wait complete. 2024-04-21T08:51:24.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 Processor -- start 2024-04-21T08:51:24.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 -- start start 2024-04-21T08:51:24.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac52050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac5e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 0x55e78ac67c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc00df50 con 0x55e78ac52a80 2024-04-21T08:51:24.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc0107b0 con 0x55e78ac49580 2024-04-21T08:51:24.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9ee8576740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc00e5a0 con 0x55e78ac49ea0 2024-04-21T08:51:24.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 0x55e78ac67c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9edf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 0x55e78ac67c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39422/0 (socket says 172.21.15.57:39422) 2024-04-21T08:51:24.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9edf7fe640 1 -- 172.21.15.57:0/2016678044 learned_addr learned my addr 172.21.15.57:0/2016678044 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:24.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.580+0000 7f9edeffd640 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac52050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac5e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9edf7fe640 1 -- 172.21.15.57:0/2016678044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 msgr2=0x55e78ac5e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac5e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9edf7fe640 1 -- 172.21.15.57:0/2016678044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 msgr2=0x55e78ac52050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac52050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9edf7fe640 1 -- 172.21.15.57:0/2016678044 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e78ac43ae0 con 0x55e78ac52a80 2024-04-21T08:51:24.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac5e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 0x55e78ac67c70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9ed802c730 tx=0x7f9ed8032fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3e21e75552a569 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9ec57fa640 1 -- 172.21.15.57:0/2016678044 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80493f0 con 0x55e78ac52a80 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9ec57fa640 1 -- 172.21.15.57:0/2016678044 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed8045040 con 0x55e78ac52a80 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.581+0000 7f9ec57fa640 1 -- 172.21.15.57:0/2016678044 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed80325a0 con 0x55e78ac52a80 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 -- 172.21.15.57:0/2016678044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 msgr2=0x55e78ac67c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 0x55e78ac67c70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9ed802c730 tx=0x7f9ed8032fa0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 -- 172.21.15.57:0/2016678044 shutdown_connections 2024-04-21T08:51:24.595 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac52a80 0x55e78ac67c70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac49ea0 0x55e78ac5e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 --2- 172.21.15.57:0/2016678044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac52050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 -- 172.21.15.57:0/2016678044 >> 172.21.15.57:0/2016678044 conn(0x55e78ac43cb0 msgr2=0x55e78ac5dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 -- 172.21.15.57:0/2016678044 shutdown_connections 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 -- 172.21.15.57:0/2016678044 wait complete. 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.582+0000 7f9ee8576740 1 -- start start 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 Processor -- start 2024-04-21T08:51:24.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac67fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac65360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac52a80 0x55e78ac658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc00ec70 con 0x55e78ac49ea0 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc0100e0 con 0x55e78ac49580 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9ecc00f340 con 0x55e78ac52a80 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ee8576740 10 client.0 _ll_register_callbacks cb 0x7f9ed001e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac65360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac67fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac65360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39436/0 (socket says 172.21.15.57:39436) 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 -- 172.21.15.57:0/1383443703 learned_addr learned my addr 172.21.15.57:0/1383443703 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9edf7fe640 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac52a80 0x55e78ac658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 -- 172.21.15.57:0/1383443703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac52a80 msgr2=0x55e78ac658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac52a80 0x55e78ac658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 -- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 msgr2=0x55e78ac67fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac67fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 -- 172.21.15.57:0/1383443703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e78ac660a0 con 0x55e78ac49ea0 2024-04-21T08:51:24.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ede7fc640 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac65360 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9ed009f4a0 tx=0x7f9ed0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc5a5b68af1abc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.583+0000 7f9ebeffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ebeffd640 1 -- 172.21.15.57:0/1383443703 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00a2a30 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ebeffd640 1 -- 172.21.15.57:0/1383443703 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f9ed00d4b00 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ed8045040 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ebeffd640 1 -- 172.21.15.57:0/1383443703 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f9ed00a1210 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 10 client.16629 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e78acc8a40 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ebeffd640 1 -- 172.21.15.57:0/1383443703 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ed8045040 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ebeffd640 1 -- 172.21.15.57:0/1383443703 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f9ed00a3330 con 0x55e78ac49ea0 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 10 client.16629 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 10 client.16629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 20 client.16629 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:24.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78acc9210 0x55e78accb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 4 client.16629 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e78abb5a70 con 0x55e78acc9210 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.584+0000 7f9edeffd640 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78acc9210 0x55e78accb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9edeffd640 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78acc9210 0x55e78accb630 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51b7233c55f97f68 server_cookie=c9d6bebbbbb07439 in_seq=0 out_seq=0 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ebeffd640 10 client.16629 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ebeffd640 1 -- 172.21.15.57:0/1383443703 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e78abb5a70 con 0x55e78acc9210 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ebeffd640 10 client.16629 handle_command_reply: tid=0 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 client.16629 shutdown 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 msgr2=0x55e78ac65360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac65360 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9ed009f4a0 tx=0x7f9ed0005bd0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 shutdown_connections 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e78acc9210 0x55e78accb630 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e78ac52a80 0x55e78ac658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e78ac49ea0 0x55e78ac65360 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ebeffd640 0 client.16629 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 --2- 172.21.15.57:0/1383443703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e78ac49580 0x55e78ac67fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.585+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 >> 172.21.15.57:0/1383443703 conn(0x55e78ac43cb0 msgr2=0x55e78ac5e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 shutdown_connections 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee8576740 1 -- 172.21.15.57:0/1383443703 wait complete. 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee8576740 20 client.16629 trim_cache size 0 max 16384 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:51:24.601 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:51:24.602 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:51:24.602 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9ec80a3f30 msgr2=0x7f9ec80a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9ec80a3f30 0x7f9ec80a63f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x55e78ac49050 tx=0x7f9ed4006bd0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 msgr2=0x7f9ee0119a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 0x7f9ee0119a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000d950 tx=0x7f9ed000de10 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 shutdown_connections 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9ec80a3f30 0x7f9ec80a63f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ee011b730 0x7f9ee0112710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee01106d0 0x7f9ee0119a20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 --2- 172.21.15.57:0/2994565346 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ee010e7b0 0x7f9ee01194e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 >> 172.21.15.57:0/2994565346 conn(0x7f9ee006d070 msgr2=0x7f9ee0109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 shutdown_connections 2024-04-21T08:51:24.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.586+0000 7f9ee58b2640 1 -- 172.21.15.57:0/2994565346 wait complete. 2024-04-21T08:51:24.604 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:51:24.604 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:51:24.711 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.709+0000 7f4a87869640 1 -- 172.21.15.57:0/2549231050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8010c5a0 msgr2=0x7f4a8010e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.709+0000 7f4a87869640 1 --2- 172.21.15.57:0/2549231050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8010c5a0 0x7f4a8010e9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009ca0 tx=0x7f4a7802c180 comp rx=0 tx=0).stop 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 -- 172.21.15.57:0/2549231050 shutdown_connections 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 --2- 172.21.15.57:0/2549231050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a8010c5a0 0x7f4a8010e9d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 --2- 172.21.15.57:0/2549231050 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a80103af0 0x7f4a8010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 --2- 172.21.15.57:0/2549231050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a801031d0 0x7f4a801035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 -- 172.21.15.57:0/2549231050 >> 172.21.15.57:0/2549231050 conn(0x7f4a800771d0 msgr2=0x7f4a800775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 -- 172.21.15.57:0/2549231050 shutdown_connections 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 -- 172.21.15.57:0/2549231050 wait complete. 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 Processor -- start 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 -- start start 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a801031d0 0x7f4a801986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a80103af0 0x7f4a80198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.710+0000 7f4a87869640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 0x7f4a8019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a87869640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a801121c0 con 0x7f4a8010c5a0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a87869640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a80112040 con 0x7f4a80103af0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a87869640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a80112340 con 0x7f4a801031d0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 0x7f4a8019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 0x7f4a8019cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39464/0 (socket says 172.21.15.57:39464) 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85605640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a801031d0 0x7f4a801986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 -- 172.21.15.57:0/3663931485 learned_addr learned my addr 172.21.15.57:0/3663931485 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a84e04640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a80103af0 0x7f4a80198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 -- 172.21.15.57:0/3663931485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a801031d0 msgr2=0x7f4a801986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a801031d0 0x7f4a801986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 -- 172.21.15.57:0/3663931485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a80103af0 msgr2=0x7f4a80198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a80103af0 0x7f4a80198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 -- 172.21.15.57:0/3663931485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8019d6d0 con 0x7f4a8010c5a0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a85e06640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 0x7f4a8019cfa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4a7802c790 tx=0x7f4a78032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78039040 con 0x7f4a8010c5a0 2024-04-21T08:51:24.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a78044070 con 0x7f4a8010c5a0 2024-04-21T08:51:24.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8019d960 con 0x7f4a8010c5a0 2024-04-21T08:51:24.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a780327c0 con 0x7f4a8010c5a0 2024-04-21T08:51:24.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.711+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a801a5250 con 0x7f4a8010c5a0 2024-04-21T08:51:24.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.712+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a80069230 con 0x7f4a8010c5a0 2024-04-21T08:51:24.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.714+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a78003b70 con 0x7f4a8010c5a0 2024-04-21T08:51:24.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.714+0000 7f4a767fc640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a6c0a3d20 0x7f4a6c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:24.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.714+0000 7f4a85605640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a6c0a3d20 0x7f4a6c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:24.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.714+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f4a780f2e00 con 0x7f4a8010c5a0 2024-04-21T08:51:24.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.715+0000 7f4a85605640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a6c0a3d20 0x7f4a6c0a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009bd0 tx=0x7f4a7c005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:24.719 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:24.720+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a780bf250 con 0x7f4a8010c5a0 2024-04-21T08:51:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:24 smithi129 ceph-mon[31272]: pgmap v182: 129 pgs: 129 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 1.56k op/s 2024-04-21T08:51:24.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:24 smithi057 ceph-mon[27968]: pgmap v182: 129 pgs: 129 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 1.56k op/s 2024-04-21T08:51:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:24 smithi136 ceph-mon[31791]: pgmap v182: 129 pgs: 129 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 1.56k op/s 2024-04-21T08:51:25.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.003+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a80107690 con 0x7f4a8010c5a0 2024-04-21T08:51:25.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.004+0000 7f4a767fc640 1 -- 172.21.15.57:0/3663931485 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f4a780beb50 con 0x7f4a8010c5a0 2024-04-21T08:51:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:51:25.004 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:51:25.004 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:51:25.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a6c0a3d20 msgr2=0x7f4a6c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a6c0a3d20 0x7f4a6c0a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009bd0 tx=0x7f4a7c005f90 comp rx=0 tx=0).stop 2024-04-21T08:51:25.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 msgr2=0x7f4a8019cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 0x7f4a8019cfa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4a7802c790 tx=0x7f4a78032340 comp rx=0 tx=0).stop 2024-04-21T08:51:25.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 shutdown_connections 2024-04-21T08:51:25.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a6c0a3d20 0x7f4a6c0a61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a8010c5a0 0x7f4a8019cfa0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a80103af0 0x7f4a80198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 --2- 172.21.15.57:0/3663931485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a801031d0 0x7f4a801986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 >> 172.21.15.57:0/3663931485 conn(0x7f4a800771d0 msgr2=0x7f4a800fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 shutdown_connections 2024-04-21T08:51:25.007 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.006+0000 7f4a87869640 1 -- 172.21.15.57:0/3663931485 wait complete. 2024-04-21T08:51:25.017 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:51:25.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 -- 172.21.15.57:0/1291947411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210106160 msgr2=0x7f021010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 --2- 172.21.15.57:0/1291947411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210106160 0x7f021010dd30 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0208009ca0 tx=0x7f020802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 -- 172.21.15.57:0/1291947411 shutdown_connections 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 --2- 172.21.15.57:0/1291947411 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210106160 0x7f021010dd30 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 --2- 172.21.15.57:0/1291947411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0210101a40 0x7f0210105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 --2- 172.21.15.57:0/1291947411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0210101090 0x7f0210101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.124+0000 7f0215006640 1 -- 172.21.15.57:0/1291947411 >> 172.21.15.57:0/1291947411 conn(0x7f0210077f50 msgr2=0x7f0210078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.125+0000 7f0215006640 1 -- 172.21.15.57:0/1291947411 shutdown_connections 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.125+0000 7f0215006640 1 -- 172.21.15.57:0/1291947411 wait complete. 2024-04-21T08:51:25.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.125+0000 7f0215006640 1 Processor -- start 2024-04-21T08:51:25.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.125+0000 7f0215006640 1 -- start start 2024-04-21T08:51:25.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 0x7f02101986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0210106160 0x7f0210198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021019cf60 0x7f021019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02101121e0 con 0x7f0210101090 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0210112060 con 0x7f021019cf60 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0210112360 con 0x7f0210106160 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 0x7f02101986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 0x7f02101986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39492/0 (socket says 172.21.15.57:39492) 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 -- 172.21.15.57:0/808394832 learned_addr learned my addr 172.21.15.57:0/808394832 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020e59c640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0210106160 0x7f0210198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020f59e640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021019cf60 0x7f021019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 -- 172.21.15.57:0/808394832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0210106160 msgr2=0x7f0210198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0210106160 0x7f0210198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 -- 172.21.15.57:0/808394832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021019cf60 msgr2=0x7f021019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021019cf60 0x7f021019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 -- 172.21.15.57:0/808394832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021019da50 con 0x7f0210101090 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f020ed9d640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 0x7f02101986a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f020000c920 tx=0x7f020000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f01f7fff640 1 -- 172.21.15.57:0/808394832 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200007c70 con 0x7f0210101090 2024-04-21T08:51:25.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.126+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f021019dd40 con 0x7f0210101090 2024-04-21T08:51:25.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.127+0000 7f01f7fff640 1 -- 172.21.15.57:0/808394832 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0200007e10 con 0x7f0210101090 2024-04-21T08:51:25.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.127+0000 7f01f7fff640 1 -- 172.21.15.57:0/808394832 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200018820 con 0x7f0210101090 2024-04-21T08:51:25.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.127+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101a5580 con 0x7f0210101090 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f01f7fff640 1 -- 172.21.15.57:0/808394832 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f020000f450 con 0x7f0210101090 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 Processor -- start 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 -- start start 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f5b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 0x562021f64890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562021ec01f0 con 0x562021f5bd60 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562021e5d4b0 con 0x562021f64e60 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.129+0000 7f0217bf7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562021ebc1b0 con 0x562021f5b3b0 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 0x562021f64890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 0x562021f64890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39502/0 (socket says 172.21.15.57:39502) 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 -- 172.21.15.57:0/907788670 learned_addr learned my addr 172.21.15.57:0/907788670 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020f59e640 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f5b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020e59c640 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f01f7fff640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f01f80a3d20 0x7f01f80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 -- 172.21.15.57:0/907788670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 msgr2=0x562021f5b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f5b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 -- 172.21.15.57:0/907788670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 msgr2=0x562021f652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 -- 172.21.15.57:0/907788670 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562021e4f1b0 con 0x562021f5bd60 2024-04-21T08:51:25.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020f59e640 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f5b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020f59e640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f01f80a3d20 0x7f01f80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f01f7fff640 1 -- 172.21.15.57:0/808394832 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f02000d0bd0 con 0x7f0210101090 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f020ed9d640 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 0x562021f64890 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f02000cdf30 tx=0x7f020001def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0373b08936d6d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f01f57fa640 1 -- 172.21.15.57:0/907788670 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020009c170 con 0x562021f5bd60 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.130+0000 7f01f57fa640 1 -- 172.21.15.57:0/907788670 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0200067d10 con 0x562021f5bd60 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f01f57fa640 1 -- 172.21.15.57:0/907788670 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020006c160 con 0x562021f5bd60 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f020f59e640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f01f80a3d20 0x7f01f80a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0208004af0 tx=0x7f020802e040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 -- 172.21.15.57:0/907788670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 msgr2=0x562021f64890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 0x562021f64890 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f02000cdf30 tx=0x7f020001def0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 -- 172.21.15.57:0/907788670 shutdown_connections 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5bd60 0x562021f64890 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 --2- 172.21.15.57:0/907788670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f5b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 -- 172.21.15.57:0/907788670 >> 172.21.15.57:0/907788670 conn(0x562021f55ba0 msgr2=0x562021f55f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.136 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 -- 172.21.15.57:0/907788670 shutdown_connections 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.131+0000 7f0217bf7740 1 -- 172.21.15.57:0/907788670 wait complete. 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 -- start start 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 Processor -- start 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f6b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5bd60 0x562021f68580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f68ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562021f6bd40 con 0x562021f5b3b0 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562021e79500 con 0x562021f64e60 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562021f6bec0 con 0x562021f5bd60 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f0217bf7740 10 client.0 _ll_register_callbacks cb 0x562021e47b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f6b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f6b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39518/0 (socket says 172.21.15.57:39518) 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 -- 172.21.15.57:0/2461597940 learned_addr learned my addr 172.21.15.57:0/2461597940 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020e59c640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5bd60 0x562021f68580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020f59e640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f68ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 -- 172.21.15.57:0/2461597940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5bd60 msgr2=0x562021f68580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5bd60 0x562021f68580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 -- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 msgr2=0x562021f68ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f68ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 -- 172.21.15.57:0/2461597940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562021bd23f0 con 0x562021f5b3b0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f020ed9d640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f6b1f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f02000a4f50 tx=0x7f02000d0f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12640f4584bd4f1b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.132+0000 7f01f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f01f2ff5640 1 -- 172.21.15.57:0/2461597940 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020009c170 con 0x562021f5b3b0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f01f2ff5640 1 -- 172.21.15.57:0/2461597940 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f02000b0f70 con 0x562021f5b3b0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0200067d10 con 0x562021f5b3b0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f01f2ff5640 1 -- 172.21.15.57:0/2461597940 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f020009cf10 con 0x562021f5b3b0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 10 client.16647 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562021c9c7d0 con 0x562021f5b3b0 2024-04-21T08:51:25.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f01f2ff5640 1 -- 172.21.15.57:0/2461597940 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0200067d10 con 0x562021f5b3b0 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f01f2ff5640 1 -- 172.21.15.57:0/2461597940 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f020006c080 con 0x562021f5b3b0 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 10 client.16647 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 20 client.16647 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021f78de0 0x562021f7b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 4 client.16647 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562021e5f930 con 0x562021f78de0 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.133+0000 7f020e59c640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021f78de0 0x562021f7b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.134+0000 7f020e59c640 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021f78de0 0x562021f7b1d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25cff672593a3d09 server_cookie=34f37d0fe765a76 in_seq=0 out_seq=0 2024-04-21T08:51:25.139 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.134+0000 7f01f2ff5640 10 client.16647 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f01f2ff5640 1 -- 172.21.15.57:0/2461597940 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562021e5f930 con 0x562021f78de0 2024-04-21T08:51:25.140 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f01f2ff5640 10 client.16647 handle_command_reply: tid=0 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 client.16647 shutdown 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 msgr2=0x562021f6b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f6b1f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f02000a4f50 tx=0x7f02000d0f40 comp rx=0 tx=0).stop 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 shutdown_connections 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021f78de0 0x562021f7b1d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f64e60 0x562021f68ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f01f2ff5640 0 client.16647 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5bd60 0x562021f68580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2461597940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f6b1f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 >> 172.21.15.57:0/2461597940 conn(0x562021f55ba0 msgr2=0x562021f578f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 shutdown_connections 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 1 -- 172.21.15.57:0/2461597940 wait complete. 2024-04-21T08:51:25.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.135+0000 7f0217bf7740 20 client.16647 trim_cache size 0 max 16384 2024-04-21T08:51:25.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 Processor -- start 2024-04-21T08:51:25.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 -- start start 2024-04-21T08:51:25.152 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f74c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f54c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 0x562021f67430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02101aaf80 con 0x562021f64e60 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02101ad7e0 con 0x562021f5bd60 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f0217bf7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f02101ab5d0 con 0x562021f5b3b0 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 0x562021f67430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 0x562021f67430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39522/0 (socket says 172.21.15.57:39522) 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020e59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f54c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 -- 172.21.15.57:0/2698681564 learned_addr learned my addr 172.21.15.57:0/2698681564 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020ed9d640 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f74c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 -- 172.21.15.57:0/2698681564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 msgr2=0x562021f74c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f74c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 -- 172.21.15.57:0/2698681564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 msgr2=0x562021f54c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.153 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f54c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020f59e640 1 -- 172.21.15.57:0/2698681564 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562021f5bb00 con 0x562021f64e60 2024-04-21T08:51:25.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.152+0000 7f020ed9d640 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f74c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:25.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f020f59e640 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 0x562021f67430 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0208037040 tx=0x7f0208041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bf93cfc805ea20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f01f4ff9640 1 -- 172.21.15.57:0/2698681564 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208039040 con 0x562021f64e60 2024-04-21T08:51:25.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f01f4ff9640 1 -- 172.21.15.57:0/2698681564 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0208053070 con 0x562021f64e60 2024-04-21T08:51:25.154 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f01f4ff9640 1 -- 172.21.15.57:0/2698681564 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208041470 con 0x562021f64e60 2024-04-21T08:51:25.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 -- 172.21.15.57:0/2698681564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 msgr2=0x562021f67430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 0x562021f67430 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0208037040 tx=0x7f0208041000 comp rx=0 tx=0).stop 2024-04-21T08:51:25.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 -- 172.21.15.57:0/2698681564 shutdown_connections 2024-04-21T08:51:25.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f64e60 0x562021f67430 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f54c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 --2- 172.21.15.57:0/2698681564 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f5b3b0 0x562021f74c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 -- 172.21.15.57:0/2698681564 >> 172.21.15.57:0/2698681564 conn(0x562021f55ba0 msgr2=0x562021f66d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 -- 172.21.15.57:0/2698681564 shutdown_connections 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.153+0000 7f0217bf7740 1 -- 172.21.15.57:0/2698681564 wait complete. 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 -- start start 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 Processor -- start 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f71560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f69f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f64e60 0x562021f6a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02101abca0 con 0x562021f5b3b0 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02101ad110 con 0x562021f5bd60 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f02101ac370 con 0x562021f64e60 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f0217bf7740 10 client.0 _ll_register_callbacks cb 0x7f0204005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f71560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f71560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39538/0 (socket says 172.21.15.57:39538) 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 -- 172.21.15.57:0/1290105777 learned_addr learned my addr 172.21.15.57:0/1290105777 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020e59c640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f69f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020f59e640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f64e60 0x562021f6a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 -- 172.21.15.57:0/1290105777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f64e60 msgr2=0x562021f6a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f64e60 0x562021f6a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 -- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 msgr2=0x562021f69f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f69f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.154+0000 7f020ed9d640 1 -- 172.21.15.57:0/1290105777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562021f6ac90 con 0x562021f5b3b0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f020ed9d640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f71560 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0200098c70 tx=0x7f020000a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad86a2080e0bfdd server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f01f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f01f2ff5640 1 -- 172.21.15.57:0/1290105777 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020009c170 con 0x562021f5b3b0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f01f2ff5640 1 -- 172.21.15.57:0/1290105777 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f020000fa90 con 0x562021f5b3b0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0208053070 con 0x562021f5b3b0 2024-04-21T08:51:25.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f01f2ff5640 1 -- 172.21.15.57:0/1290105777 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f020009c350 con 0x562021f5b3b0 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 10 client.16653 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562021fdf0d0 con 0x562021f5b3b0 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f01f2ff5640 1 -- 172.21.15.57:0/1290105777 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0208053070 con 0x562021f5b3b0 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f01f2ff5640 1 -- 172.21.15.57:0/1290105777 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f02000a4ff0 con 0x562021f5b3b0 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 10 client.16653 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 10 client.16653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 20 client.16653 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021fdf460 0x562021fe1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 4 client.16653 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.155+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562021ecdc10 con 0x562021fdf460 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.156+0000 7f020e59c640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021fdf460 0x562021fe1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.156+0000 7f020e59c640 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021fdf460 0x562021fe1850 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e852d70b8c0b918 server_cookie=40e9bc2a5edb2b52 in_seq=0 out_seq=0 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.156+0000 7f01f2ff5640 10 client.16653 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:25.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f01f2ff5640 1 -- 172.21.15.57:0/1290105777 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562021ecdc10 con 0x562021fdf460 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f01f2ff5640 10 client.16653 handle_command_reply: tid=0 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 client.16653 shutdown 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 msgr2=0x562021f71560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f71560 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0200098c70 tx=0x7f020000a5e0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 shutdown_connections 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562021fdf460 0x562021fe1850 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562021f64e60 0x562021f6a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562021f5bd60 0x562021f69f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f01f2ff5640 0 client.16653 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 --2- 172.21.15.57:0/1290105777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562021f5b3b0 0x562021f71560 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 >> 172.21.15.57:0/1290105777 conn(0x562021f55ba0 msgr2=0x562021f675b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 shutdown_connections 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 1 -- 172.21.15.57:0/1290105777 wait complete. 2024-04-21T08:51:25.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.157+0000 7f0217bf7740 20 client.16653 trim_cache size 0 max 16384 2024-04-21T08:51:25.163 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:51:25.163 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f01f80a3d20 msgr2=0x7f01f80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f01f80a3d20 0x7f01f80a61e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0208004af0 tx=0x7f020802e040 comp rx=0 tx=0).stop 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 msgr2=0x7f02101986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 0x7f02101986a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f020000c920 tx=0x7f020000cde0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 shutdown_connections 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f01f80a3d20 0x7f01f80a61e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021019cf60 0x7f021019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0210106160 0x7f0210198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 --2- 172.21.15.57:0/808394832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0210101090 0x7f02101986a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 >> 172.21.15.57:0/808394832 conn(0x7f0210077f50 msgr2=0x7f02100fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 shutdown_connections 2024-04-21T08:51:25.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:25.158+0000 7f0215006640 1 -- 172.21.15.57:0/808394832 wait complete. 2024-04-21T08:51:25.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3663931485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:25.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3663931485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3663931485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:51:26.169 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:51:26.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:51:26.169 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:51:26.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 -- 172.21.15.57:0/3526855024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 msgr2=0x7fd9201035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 --2- 172.21.15.57:0/3526855024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd9201035b0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd908009900 tx=0x7fd90802bd60 comp rx=0 tx=0).stop 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 -- 172.21.15.57:0/3526855024 shutdown_connections 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 --2- 172.21.15.57:0/3526855024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd92010c510 0x7fd92010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 --2- 172.21.15.57:0/3526855024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd920103af0 0x7fd92010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 --2- 172.21.15.57:0/3526855024 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd9201035b0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 -- 172.21.15.57:0/3526855024 >> 172.21.15.57:0/3526855024 conn(0x7fd920077f50 msgr2=0x7fd920078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 -- 172.21.15.57:0/3526855024 shutdown_connections 2024-04-21T08:51:26.282 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.281+0000 7fd926a55640 1 -- 172.21.15.57:0/3526855024 wait complete. 2024-04-21T08:51:26.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 Processor -- start 2024-04-21T08:51:26.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 -- start start 2024-04-21T08:51:26.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd92019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd920103af0 0x7fd92019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd92010c510 0x7fd9201985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd920112180 con 0x7fd9201031d0 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd920112000 con 0x7fd92010c510 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd926a55640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd920112300 con 0x7fd920103af0 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd91ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd92019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd91ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd92019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39564/0 (socket says 172.21.15.57:39564) 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd91ffff640 1 -- 172.21.15.57:0/453247286 learned_addr learned my addr 172.21.15.57:0/453247286 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd924ff2640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd92010c510 0x7fd9201985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.285 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd920103af0 0x7fd92019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.285 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.282+0000 7fd91ffff640 1 -- 172.21.15.57:0/453247286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd920103af0 msgr2=0x7fd92019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91ffff640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd920103af0 0x7fd92019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91ffff640 1 -- 172.21.15.57:0/453247286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd92010c510 msgr2=0x7fd9201985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91ffff640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd92010c510 0x7fd9201985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91ffff640 1 -- 172.21.15.57:0/453247286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd920198d20 con 0x7fd9201031d0 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91ffff640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd92019e3f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd908005940 tx=0x7fd90802d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91d7fa640 1 -- 172.21.15.57:0/453247286 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908043020 con 0x7fd9201031d0 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91d7fa640 1 -- 172.21.15.57:0/453247286 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd90802dd20 con 0x7fd9201031d0 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd920198fb0 con 0x7fd9201031d0 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd91d7fa640 1 -- 172.21.15.57:0/453247286 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd908003b80 con 0x7fd9201031d0 2024-04-21T08:51:26.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.283+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9201a52b0 con 0x7fd9201031d0 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.286+0000 7fd91d7fa640 1 -- 172.21.15.57:0/453247286 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd908046020 con 0x7fd9201031d0 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.286+0000 7fd92964a740 1 Processor -- start 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.286+0000 7fd92964a740 1 -- start start 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.286+0000 7fd92964a740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df499618e0 0x55df49961cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.286+0000 7fd92964a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df4996a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.286+0000 7fd92964a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 0x55df4996d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd92964a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55df498c5f00 con 0x55df4996ae10 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd92964a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55df49863590 con 0x55df49962290 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd92964a740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55df498c1ec0 con 0x55df499618e0 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 0x55df4996d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 0x55df4996d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39576/0 (socket says 172.21.15.57:39576) 2024-04-21T08:51:26.290 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3858017534 learned_addr learned my addr 172.21.15.57:0/3858017534 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91ffff640 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df4996a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd924ff2640 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df499618e0 0x55df49961cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3858017534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df499618e0 msgr2=0x55df49961cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df499618e0 0x55df49961cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3858017534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 msgr2=0x55df4996a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df4996a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3858017534 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df496a07e0 con 0x55df4996ae10 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91d7fa640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd90c0a3d20 0x7fd90c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd924ff2640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd90c0a3d20 0x7fd90c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91d7fa640 1 -- 172.21.15.57:0/453247286 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd9080f2c10 con 0x7fd9201031d0 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 0x55df4996d260 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd91000dc50 tx=0x7fd91000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de8c45361c1f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd9167fc640 1 -- 172.21.15.57:0/3858017534 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910015070 con 0x55df4996ae10 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd9167fc640 1 -- 172.21.15.57:0/3858017534 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9100035f0 con 0x55df4996ae10 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.287+0000 7fd9167fc640 1 -- 172.21.15.57:0/3858017534 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9100116a0 con 0x55df4996ae10 2024-04-21T08:51:26.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 -- 172.21.15.57:0/3858017534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 msgr2=0x55df4996d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd924ff2640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd90c0a3d20 0x7fd90c0a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd91800ae40 tx=0x7fd91800a4d0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 0x55df4996d260 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd91000dc50 tx=0x7fd91000f730 comp rx=0 tx=0).stop 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 -- 172.21.15.57:0/3858017534 shutdown_connections 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4996ae10 0x55df4996d260 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df4996a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 --2- 172.21.15.57:0/3858017534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df499618e0 0x55df49961cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 -- 172.21.15.57:0/3858017534 >> 172.21.15.57:0/3858017534 conn(0x55df4995bfb0 msgr2=0x55df4995c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 -- 172.21.15.57:0/3858017534 shutdown_connections 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.288+0000 7fd92964a740 1 -- 172.21.15.57:0/3858017534 wait complete. 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 -- start start 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 Processor -- start 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49967d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 0x55df499682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4996ae10 0x55df49968810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55df4996dd80 con 0x55df49962290 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55df4987f580 con 0x55df499618e0 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55df4996df00 con 0x55df4996ae10 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd92964a740 10 client.0 _ll_register_callbacks cb 0x55df49885650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 0x55df499682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 0x55df499682d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39580/0 (socket says 172.21.15.57:39580) 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3795280028 learned_addr learned my addr 172.21.15.57:0/3795280028 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91ffff640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49967d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.296 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd924ff2640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4996ae10 0x55df49968810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3795280028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4996ae10 msgr2=0x55df49968810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4996ae10 0x55df49968810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.289+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 msgr2=0x55df49967d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49967d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd91f7fe640 1 -- 172.21.15.57:0/3795280028 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df49700f70 con 0x55df49962290 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd924ff2640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4996ae10 0x55df49968810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd91ffff640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49967d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 0x55df499682d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd910000c00 tx=0x7fd91000d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d5da8854a7171c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/3795280028 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910028060 con 0x55df49962290 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/3795280028 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd910004380 con 0x55df49962290 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9100035f0 con 0x55df49962290 2024-04-21T08:51:26.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/3795280028 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd91002f050 con 0x55df49962290 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd92964a740 10 client.16665 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55df49598170 con 0x55df49962290 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/3795280028 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9100035f0 con 0x55df49962290 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/3795280028 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd910011970 con 0x55df49962290 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.290+0000 7fd92964a740 10 client.16665 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd92964a740 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd92964a740 20 client.16665 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd92964a740 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4997b230 0x55df4997d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd92964a740 4 client.16665 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df498d3920 con 0x55df4997b230 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd91ffff640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4997b230 0x55df4997d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd91ffff640 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4997b230 0x55df4997d620 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63547fa4f3c002d4 server_cookie=850e843c63df415e in_seq=0 out_seq=0 2024-04-21T08:51:26.298 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.291+0000 7fd8ff7fe640 10 client.16665 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/3795280028 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df498d3920 con 0x55df4997b230 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd8ff7fe640 10 client.16665 handle_command_reply: tid=0 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 client.16665 shutdown 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 msgr2=0x55df499682d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 0x55df499682d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd910000c00 tx=0x7fd91000d9a0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 shutdown_connections 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4997b230 0x55df4997d620 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4996ae10 0x55df49968810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df49962290 0x55df499682d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd8ff7fe640 0 client.16665 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 --2- 172.21.15.57:0/3795280028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49967d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 >> 172.21.15.57:0/3795280028 conn(0x55df4995bfb0 msgr2=0x55df4995dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 shutdown_connections 2024-04-21T08:51:26.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 1 -- 172.21.15.57:0/3795280028 wait complete. 2024-04-21T08:51:26.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.292+0000 7fd92964a740 20 client.16665 trim_cache size 0 max 16384 2024-04-21T08:51:26.309 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 Processor -- start 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 -- start start 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4995bfb0 0x55df49976c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 0x55df4995da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df49982bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201ad250 con 0x55df499618e0 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201afab0 con 0x55df49962290 2024-04-21T08:51:26.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd92964a740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201ad8a0 con 0x55df4995bfb0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 0x55df4995da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 0x55df4995da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39588/0 (socket says 172.21.15.57:39588) 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 -- 172.21.15.57:0/2861462160 learned_addr learned my addr 172.21.15.57:0/2861462160 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd924ff2640 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df49982bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91ffff640 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4995bfb0 0x55df49976c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 -- 172.21.15.57:0/2861462160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4995bfb0 msgr2=0x55df49976c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4995bfb0 0x55df49976c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 -- 172.21.15.57:0/2861462160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 msgr2=0x55df49982bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df49982bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 -- 172.21.15.57:0/2861462160 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df49960c10 con 0x55df499618e0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91ffff640 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4995bfb0 0x55df49976c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.310+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 0x55df4995da10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd910005bc0 tx=0x7fd91000d930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d49db6c62a42e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd915ffb640 1 -- 172.21.15.57:0/2861462160 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910028070 con 0x55df499618e0 2024-04-21T08:51:26.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd915ffb640 1 -- 172.21.15.57:0/2861462160 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9100054d0 con 0x55df499618e0 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd915ffb640 1 -- 172.21.15.57:0/2861462160 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910015070 con 0x55df499618e0 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 -- 172.21.15.57:0/2861462160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 msgr2=0x55df4995da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 0x55df4995da10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd910005bc0 tx=0x7fd91000d930 comp rx=0 tx=0).stop 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 -- 172.21.15.57:0/2861462160 shutdown_connections 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df49962290 0x55df49982bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df499618e0 0x55df4995da10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 --2- 172.21.15.57:0/2861462160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df4995bfb0 0x55df49976c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 -- 172.21.15.57:0/2861462160 >> 172.21.15.57:0/2861462160 conn(0x55df499669d0 msgr2=0x55df4995d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 -- 172.21.15.57:0/2861462160 shutdown_connections 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.311+0000 7fd92964a740 1 -- 172.21.15.57:0/2861462160 wait complete. 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 -- start start 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 Processor -- start 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 0x55df49973420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49970120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.320 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df49962290 0x55df49970660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201adf70 con 0x55df4995bfb0 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201af3e0 con 0x55df499618e0 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd9201ae640 con 0x55df49962290 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd92964a740 10 client.0 _ll_register_callbacks cb 0x7fd9080f6630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd91ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 0x55df49973420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd91ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 0x55df49973420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39598/0 (socket says 172.21.15.57:39598) 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd91ffff640 1 -- 172.21.15.57:0/142718479 learned_addr learned my addr 172.21.15.57:0/142718479 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49970120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.312+0000 7fd924ff2640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df49962290 0x55df49970660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91f7fe640 1 -- 172.21.15.57:0/142718479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df49962290 msgr2=0x55df49970660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df49962290 0x55df49970660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91f7fe640 1 -- 172.21.15.57:0/142718479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 msgr2=0x55df49973420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 0x55df49973420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.321 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91f7fe640 1 -- 172.21.15.57:0/142718479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df49970da0 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91ffff640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 0x55df49973420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd91f7fe640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49970120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55df49977a40 tx=0x7fd9100045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c31ed1b32c3fb06 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/142718479 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910015070 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9100054d0 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/142718479 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fd910005230 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/142718479 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd910028040 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd92964a740 10 client.26150 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55df499e23d0 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/142718479 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9100054d0 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.313+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/142718479 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd910011440 con 0x55df499618e0 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd92964a740 10 client.26150 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd92964a740 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:26.322 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd92964a740 20 client.26150 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd92964a740 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4996ae10 0x55df499e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd92964a740 4 client.26150 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55df498cd790 con 0x55df4996ae10 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd91ffff640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4996ae10 0x55df499e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd8ff7fe640 10 client.26150 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.314+0000 7fd91ffff640 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4996ae10 0x55df499e4a40 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf978e5defac12e3 server_cookie=8817156da6a5914a in_seq=0 out_seq=0 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd8ff7fe640 1 -- 172.21.15.57:0/142718479 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55df498cd790 con 0x55df4996ae10 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd8ff7fe640 10 client.26150 handle_command_reply: tid=0 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 client.26150 shutdown 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 msgr2=0x55df49970120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49970120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55df49977a40 tx=0x7fd9100045b0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 shutdown_connections 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55df4996ae10 0x55df499e4a40 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55df49962290 0x55df49970660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd8ff7fe640 0 client.26150 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55df499618e0 0x55df49970120 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 --2- 172.21.15.57:0/142718479 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55df4995bfb0 0x55df49973420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.315+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 >> 172.21.15.57:0/142718479 conn(0x55df499669d0 msgr2=0x55df4995db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 shutdown_connections 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd92964a740 1 -- 172.21.15.57:0/142718479 wait complete. 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd92964a740 20 client.26150 trim_cache size 0 max 16384 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:51:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "8b767c17-e785-4e87-bdf4-567a8488f353", 2024-04-21T08:51:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:51:26.325 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:51:26.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd90c0a3d20 msgr2=0x7fd90c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd90c0a3d20 0x7fd90c0a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd91800ae40 tx=0x7fd91800a4d0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 msgr2=0x7fd92019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd92019e3f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd908005940 tx=0x7fd90802d920 comp rx=0 tx=0).stop 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 shutdown_connections 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd90c0a3d20 0x7fd90c0a61e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd92010c510 0x7fd9201985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd920103af0 0x7fd92019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 --2- 172.21.15.57:0/453247286 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9201031d0 0x7fd92019e3f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 >> 172.21.15.57:0/453247286 conn(0x7fd920077f50 msgr2=0x7fd9200fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 shutdown_connections 2024-04-21T08:51:26.326 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.316+0000 7fd926a55640 1 -- 172.21.15.57:0/453247286 wait complete. 2024-04-21T08:51:26.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:26.327 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 -- 172.21.15.57:0/1163299216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4102180 msgr2=0x7f8ff4102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 --2- 172.21.15.57:0/1163299216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4102180 0x7f8ff4102560 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009900 tx=0x7f8fe402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 -- 172.21.15.57:0/1163299216 shutdown_connections 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 --2- 172.21.15.57:0/1163299216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ff4107250 0x7f8ff410ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 --2- 172.21.15.57:0/1163299216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4102b30 0x7f8ff4106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 --2- 172.21.15.57:0/1163299216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4102180 0x7f8ff4102560 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 -- 172.21.15.57:0/1163299216 >> 172.21.15.57:0/1163299216 conn(0x7f8ff40fda20 msgr2=0x7f8ff40ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.444 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 -- 172.21.15.57:0/1163299216 shutdown_connections 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 -- 172.21.15.57:0/1163299216 wait complete. 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.442+0000 7f8ff930d640 1 Processor -- start 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 -- start start 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ff4102180 0x7f8ff4074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 0x7f8ff4074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4074cd0 0x7f8ff41a6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff4112210 con 0x7f8ff4107250 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff4112090 con 0x7f8ff4074cd0 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff930d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff4112390 con 0x7f8ff4102180 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 0x7f8ff4074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 0x7f8ff4074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39614/0 (socket says 172.21.15.57:39614) 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 -- 172.21.15.57:0/818787453 learned_addr learned my addr 172.21.15.57:0/818787453 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4074cd0 0x7f8ff41a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ff4102180 0x7f8ff4074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 -- 172.21.15.57:0/818787453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ff4102180 msgr2=0x7f8ff4074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ff4102180 0x7f8ff4074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 -- 172.21.15.57:0/818787453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4074cd0 msgr2=0x7f8ff41a6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4074cd0 0x7f8ff41a6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff27fc640 1 -- 172.21.15.57:0/818787453 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff41a7020 con 0x7f8ff4107250 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.443+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4074cd0 0x7f8ff41a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.444+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 0x7f8ff4074790 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800e9c0 tx=0x7f8fe800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.444+0000 7f8fdbfff640 1 -- 172.21.15.57:0/818787453 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe800cd80 con 0x7f8ff4107250 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.444+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff41a7310 con 0x7f8ff4107250 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.444+0000 7f8fdbfff640 1 -- 172.21.15.57:0/818787453 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe80054d0 con 0x7f8ff4107250 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.444+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff41a78b0 con 0x7f8ff4107250 2024-04-21T08:51:26.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.444+0000 7f8fdbfff640 1 -- 172.21.15.57:0/818787453 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe8010430 con 0x7f8ff4107250 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.446+0000 7f8fdbfff640 1 -- 172.21.15.57:0/818787453 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8fe8004fb0 con 0x7f8ff4107250 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.446+0000 7f8ffbfce740 1 Processor -- start 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.446+0000 7f8ffbfce740 1 -- start start 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.446+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eed7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.446+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 0x55957eee0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 0x55957eee2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55957ee3c1e0 con 0x55957eee0a80 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55957edd9390 con 0x55957eed7ea0 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55957ee381a0 con 0x55957eed7580 2024-04-21T08:51:26.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 0x55957eee2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 0x55957eee0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 0x55957eee0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:52440/0 (socket says 172.21.15.57:52440) 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 0x55957eee2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39616/0 (socket says 172.21.15.57:39616) 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/4051340157 learned_addr learned my addr 172.21.15.57:0/4051340157 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eed7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8fdbfff640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8fdc0a3d50 0x7f8fdc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8fdc0a3d50 0x7f8fdc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/4051340157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 msgr2=0x55957eed7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eed7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/4051340157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 msgr2=0x55957eee2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 0x55957eee2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/4051340157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55957ec187d0 con 0x55957eed7ea0 2024-04-21T08:51:26.453 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8fdbfff640 1 -- 172.21.15.57:0/818787453 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8fe80d0fe0 con 0x7f8ff4107250 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 0x55957eee2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 0x55957eee0540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009900 tx=0x7f8fe4005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5ba991c844c9f56 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.447+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8fdc0a3d50 0x7f8fdc0a6210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0094d0 tx=0x7f8fec004ab0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8fd97fa640 1 -- 172.21.15.57:0/4051340157 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4044020 con 0x55957eed7ea0 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8fd97fa640 1 -- 172.21.15.57:0/4051340157 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe402cc90 con 0x55957eed7ea0 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8fd97fa640 1 -- 172.21.15.57:0/4051340157 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40046c0 con 0x55957eed7ea0 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 -- 172.21.15.57:0/4051340157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 msgr2=0x55957eee0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 0x55957eee0540 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009900 tx=0x7f8fe4005dd0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 -- 172.21.15.57:0/4051340157 shutdown_connections 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eee0a80 0x55957eee2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7ea0 0x55957eee0540 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/4051340157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eed7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 -- 172.21.15.57:0/4051340157 >> 172.21.15.57:0/4051340157 conn(0x55957eed1cb0 msgr2=0x55957eed2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 -- 172.21.15.57:0/4051340157 shutdown_connections 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.448+0000 7f8ffbfce740 1 -- 172.21.15.57:0/4051340157 wait complete. 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 -- start start 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 Processor -- start 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eedda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eeddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eede4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55957eee3810 con 0x55957eed7ea0 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55957edf52f0 con 0x55957eee0a80 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55957eee3990 con 0x55957eed7580 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ffbfce740 10 client.0 _ll_register_callbacks cb 0x55957ee02580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eeddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eeddfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39620/0 (socket says 172.21.15.57:39620) 2024-04-21T08:51:26.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 -- 172.21.15.57:0/778300142 learned_addr learned my addr 172.21.15.57:0/778300142 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eede4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eedda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 -- 172.21.15.57:0/778300142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 msgr2=0x55957eedda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eedda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 -- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 msgr2=0x55957eede4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eede4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff27fc640 1 -- 172.21.15.57:0/778300142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55957eb4e3f0 con 0x55957eed7ea0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.449+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eedda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eeddfa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c6a0 tx=0x7f8fe80d4640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd798ffc79d5aaa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8fd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/778300142 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe809c760 con 0x55957eed7ea0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/778300142 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8fe80bd0d0 con 0x55957eed7ea0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe402cc90 con 0x55957eed7ea0 2024-04-21T08:51:26.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/778300142 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8fe809c580 con 0x55957eed7ea0 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8ffbfce740 10 client.16683 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55957eb0e0f0 con 0x55957eed7ea0 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/778300142 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe402cc90 con 0x55957eed7ea0 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.450+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/778300142 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8fe80d30f0 con 0x55957eed7ea0 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ffbfce740 10 client.16683 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ffbfce740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ffbfce740 20 client.16683 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957eef0a20 0x55957eef2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ffbfce740 4 client.16683 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55957ee49c00 con 0x55957eef0a20 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957eef0a20 0x55957eef2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957eef0a20 0x55957eef2e10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e423551606ce138 server_cookie=f894b19d71e2c125 in_seq=0 out_seq=0 2024-04-21T08:51:26.457 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.451+0000 7f8fd6ff5640 10 client.16683 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/778300142 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55957ee49c00 con 0x55957eef0a20 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8fd6ff5640 10 client.16683 handle_command_reply: tid=0 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 client.16683 shutdown 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 msgr2=0x55957eeddfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eeddfa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c6a0 tx=0x7f8fe80d4640 comp rx=0 tx=0).stop 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 shutdown_connections 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957eef0a20 0x55957eef2e10 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eede4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eeddfa0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/778300142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7580 0x55957eedda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8fd6ff5640 0 client.16683 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 >> 172.21.15.57:0/778300142 conn(0x55957eed1cb0 msgr2=0x55957eed3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 shutdown_connections 2024-04-21T08:51:26.459 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 1 -- 172.21.15.57:0/778300142 wait complete. 2024-04-21T08:51:26.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.454+0000 7f8ffbfce740 20 client.16683 trim_cache size 0 max 16384 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 Processor -- start 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 -- start start 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 0x55957eee0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7ea0 0x55957eeec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eedc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41af440 con 0x55957eed7580 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41b1c10 con 0x55957eee0a80 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41afa00 con 0x55957eed7ea0 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 0x55957eee0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 0x55957eee0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39630/0 (socket says 172.21.15.57:39630) 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ff37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eedc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.474+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/1680026936 learned_addr learned my addr 172.21.15.57:0/1680026936 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7ea0 0x55957eeec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/1680026936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7ea0 msgr2=0x55957eeec6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7ea0 0x55957eeec6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/1680026936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 msgr2=0x55957eedc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eedc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff2ffd640 1 -- 172.21.15.57:0/1680026936 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55957eedbfb0 con 0x55957eed7580 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 0x55957eee0190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4005ec0 tx=0x7f8fe402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b2ba9c192b33dc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8fd8ff9640 1 -- 172.21.15.57:0/1680026936 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe404d070 con 0x55957eed7580 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8fd8ff9640 1 -- 172.21.15.57:0/1680026936 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe402cc90 con 0x55957eed7580 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.475+0000 7f8fd8ff9640 1 -- 172.21.15.57:0/1680026936 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4044040 con 0x55957eed7580 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1680026936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 msgr2=0x55957eee0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 0x55957eee0190 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4005ec0 tx=0x7f8fe402bc30 comp rx=0 tx=0).stop 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1680026936 shutdown_connections 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eee0a80 0x55957eedc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eed7ea0 0x55957eeec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1680026936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7580 0x55957eee0190 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1680026936 >> 172.21.15.57:0/1680026936 conn(0x55957eed1cb0 msgr2=0x55957eed0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1680026936 shutdown_connections 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1680026936 wait complete. 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 -- start start 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.476+0000 7f8ffbfce740 1 Processor -- start 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7580 0x55957eedae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eedb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eee0a80 0x55957ef55e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41b00d0 con 0x55957eed7ea0 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41b1540 con 0x55957eed7580 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41b07a0 con 0x55957eee0a80 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 10 client.0 _ll_register_callbacks cb 0x7f8fe809df50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:26.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eedb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7580 0x55957eedae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eedb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:39644/0 (socket says 172.21.15.57:39644) 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 -- 172.21.15.57:0/1538803558 learned_addr learned my addr 172.21.15.57:0/1538803558 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff37fe640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eee0a80 0x55957ef55e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 -- 172.21.15.57:0/1538803558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eee0a80 msgr2=0x55957ef55e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eee0a80 0x55957ef55e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 -- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7580 msgr2=0x55957eedae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7580 0x55957eedae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 -- 172.21.15.57:0/1538803558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55957ef56380 con 0x55957eed7ea0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7580 0x55957eedae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ff27fc640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eedb380 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80ce480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00428bc220f4fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8fd6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/1538803558 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe809c760 con 0x55957eed7ea0 2024-04-21T08:51:26.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/1538803558 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f8fe80a5600 con 0x55957eed7ea0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe402cc90 con 0x55957eed7ea0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.477+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/1538803558 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8fe80046c0 con 0x55957eed7ea0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 10 client.16689 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55957ef57690 con 0x55957eed7ea0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/1538803558 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe402cc90 con 0x55957eed7ea0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/1538803558 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8fe8061ba0 con 0x55957eed7ea0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 10 client.16689 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 20 client.16689 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957ef57db0 0x55957ef5a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 4 client.16689 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55957ee43a70 con 0x55957ef57db0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.478+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957ef57db0 0x55957ef5a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.479+0000 7f8fd6ff5640 10 client.16689 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.479+0000 7f8ff2ffd640 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957ef57db0 0x55957ef5a1d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e505aac89a0bb483 server_cookie=34bd50a89f00439a in_seq=0 out_seq=0 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8fd6ff5640 1 -- 172.21.15.57:0/1538803558 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55957ee43a70 con 0x55957ef57db0 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8fd6ff5640 10 client.16689 handle_command_reply: tid=0 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 client.16689 shutdown 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 msgr2=0x55957eedb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eedb380 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8000c00 tx=0x7f8fe80ce480 comp rx=0 tx=0).stop 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 shutdown_connections 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55957ef57db0 0x55957ef5a1d0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55957eee0a80 0x55957ef55e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55957eed7ea0 0x55957eedb380 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8fd6ff5640 0 client.16689 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 --2- 172.21.15.57:0/1538803558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55957eed7580 0x55957eedae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 >> 172.21.15.57:0/1538803558 conn(0x55957eed1cb0 msgr2=0x55957eedda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 shutdown_connections 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 1 -- 172.21.15.57:0/1538803558 wait complete. 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ffbfce740 20 client.16689 trim_cache size 0 max 16384 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (2158 inodes in the stack)", 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "8b767c17-e785-4e87-bdf4-567a8488f353": { 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:51:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "8b767c17-e785-4e87-bdf4-567a8488f353", 2024-04-21T08:51:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:51:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:51:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:51:26.487 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:51:26.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8fdc0a3d50 msgr2=0x7f8fdc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ff930d640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8fdc0a3d50 0x7f8fdc0a6210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fec0094d0 tx=0x7f8fec004ab0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 msgr2=0x7f8ff4074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.481+0000 7f8ff930d640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 0x7f8ff4074790 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800e9c0 tx=0x7f8fe800ee80 comp rx=0 tx=0).stop 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 shutdown_connections 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8fdc0a3d50 0x7f8fdc0a6210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ff4074cd0 0x7f8ff41a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff4107250 0x7f8ff4074790 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 --2- 172.21.15.57:0/818787453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ff4102180 0x7f8ff4074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 >> 172.21.15.57:0/818787453 conn(0x7f8ff40fda20 msgr2=0x7f8ff40fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 shutdown_connections 2024-04-21T08:51:26.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:26.482+0000 7f8ff930d640 1 -- 172.21.15.57:0/818787453 wait complete. 2024-04-21T08:51:26.492 INFO:tasks.cephfs.filesystem:scrub status for tag:8b767c17-e785-4e87-bdf4-567a8488f353 - {'path': '/', 'tag': '8b767c17-e785-4e87-bdf4-567a8488f353', 'options': 'recursive,force'} 2024-04-21T08:51:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:26 smithi129 ceph-mon[31272]: pgmap v183: 129 pgs: 129 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 9.5 MiB/s wr, 849 op/s 2024-04-21T08:51:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:26 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:51:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:26 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:26 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:51:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:26 smithi057 ceph-mon[27968]: pgmap v183: 129 pgs: 129 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 9.5 MiB/s wr, 849 op/s 2024-04-21T08:51:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:26 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:51:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:26 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:26 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:51:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:26 smithi136 ceph-mon[31791]: pgmap v183: 129 pgs: 129 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 9.5 MiB/s wr, 849 op/s 2024-04-21T08:51:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:26 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:51:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:26 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:51:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:26 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:51:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:28 smithi129 ceph-mon[31272]: pgmap v184: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 10 MiB/s wr, 1.84k op/s 2024-04-21T08:51:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:28 smithi057 ceph-mon[27968]: pgmap v184: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 10 MiB/s wr, 1.84k op/s 2024-04-21T08:51:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:28 smithi136 ceph-mon[31791]: pgmap v184: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 10 MiB/s wr, 1.84k op/s 2024-04-21T08:51:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:30 smithi129 ceph-mon[31272]: pgmap v185: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 1.54k op/s 2024-04-21T08:51:30.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:30 smithi057 ceph-mon[27968]: pgmap v185: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 1.54k op/s 2024-04-21T08:51:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:30 smithi136 ceph-mon[31791]: pgmap v185: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 1.54k op/s 2024-04-21T08:51:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:32 smithi129 ceph-mon[31272]: pgmap v186: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 1.54k op/s 2024-04-21T08:51:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:32 smithi057 ceph-mon[27968]: pgmap v186: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 1.54k op/s 2024-04-21T08:51:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:32 smithi136 ceph-mon[31791]: pgmap v186: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.0 MiB/s wr, 1.54k op/s 2024-04-21T08:51:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:34 smithi129 ceph-mon[31272]: pgmap v187: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 2.04k op/s 2024-04-21T08:51:34.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:34 smithi057 ceph-mon[27968]: pgmap v187: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 2.04k op/s 2024-04-21T08:51:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:34 smithi136 ceph-mon[31791]: pgmap v187: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.6 MiB/s wr, 2.04k op/s 2024-04-21T08:51:35.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:35 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:35.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:35 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:35.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:35 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:51:36.740 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:36 smithi057 ceph-mon[27968]: pgmap v188: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 1.64k op/s 2024-04-21T08:51:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:36 smithi129 ceph-mon[31272]: pgmap v188: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 1.64k op/s 2024-04-21T08:51:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:36 smithi136 ceph-mon[31791]: pgmap v188: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 1.64k op/s 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 -- 172.21.15.57:0/1954221868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face8106170 msgr2=0x7face810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 --2- 172.21.15.57:0/1954221868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face8106170 0x7face810dd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7face0009ca0 tx=0x7face002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 -- 172.21.15.57:0/1954221868 shutdown_connections 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 --2- 172.21.15.57:0/1954221868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face8106170 0x7face810dd40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 --2- 172.21.15.57:0/1954221868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face8101aa0 0x7face8105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 --2- 172.21.15.57:0/1954221868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 0x7face81014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 -- 172.21.15.57:0/1954221868 >> 172.21.15.57:0/1954221868 conn(0x7face8077fd0 msgr2=0x7face80783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 -- 172.21.15.57:0/1954221868 shutdown_connections 2024-04-21T08:51:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.694+0000 7faceedbd640 1 -- 172.21.15.57:0/1954221868 wait complete. 2024-04-21T08:51:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 Processor -- start 2024-04-21T08:51:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 -- start start 2024-04-21T08:51:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 0x7face81a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face8106170 0x7face81a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 0x7face819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7face81110c0 con 0x7face8106170 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7face8110f40 con 0x7face819a6a0 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faceedbd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7face8111240 con 0x7face81010f0 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faced35a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 0x7face819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faced35a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 0x7face819ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:48208/0 (socket says 172.21.15.57:48208) 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7faced35a640 1 -- 172.21.15.57:0/1454034900 learned_addr learned my addr 172.21.15.57:0/1454034900 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.695+0000 7face7fff640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face8106170 0x7face81a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7facecb59640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 0x7face81a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7faced35a640 1 -- 172.21.15.57:0/1454034900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 msgr2=0x7face81a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7faced35a640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 0x7face81a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7faced35a640 1 -- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face8106170 msgr2=0x7face81a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7faced35a640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face8106170 0x7face81a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7faced35a640 1 -- 172.21.15.57:0/1454034900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face819b2a0 con 0x7face819a6a0 2024-04-21T08:51:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7facecb59640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 0x7face81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:51:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7faced35a640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 0x7face819ab20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7face0005e30 tx=0x7face00346d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7face5ffb640 1 -- 172.21.15.57:0/1454034900 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0041020 con 0x7face819a6a0 2024-04-21T08:51:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7face5ffb640 1 -- 172.21.15.57:0/1454034900 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face0032970 con 0x7face819a6a0 2024-04-21T08:51:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.696+0000 7face5ffb640 1 -- 172.21.15.57:0/1454034900 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0039040 con 0x7face819a6a0 2024-04-21T08:51:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.697+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face819b530 con 0x7face819a6a0 2024-04-21T08:51:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.697+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face81a7620 con 0x7face819a6a0 2024-04-21T08:51:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.699+0000 7facf19b4740 1 Processor -- start 2024-04-21T08:51:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.701+0000 7face5ffb640 1 -- 172.21.15.57:0/1454034900 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7face0044020 con 0x7face819a6a0 2024-04-21T08:51:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.701+0000 7facf19b4740 1 -- start start 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7face5ffb640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7facd80a3dc0 0x7facd80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7facf19b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126508d0 0x55bd1264f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7facf19b4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd1264ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7facf19b4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd1265b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7facf19b4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55bd125b48d0 con 0x55bd126590e0 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7facf19b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bd12551e80 con 0x55bd126508d0 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7facf19b4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55bd125b0890 con 0x55bd1264fb60 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7face5ffb640 1 -- 172.21.15.57:0/1454034900 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7face00f26e0 con 0x7face819a6a0 2024-04-21T08:51:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7face7fff640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7facd80a3dc0 0x7facd80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7faced35a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd1265b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7face7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd1264ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7face7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd1264ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:39710/0 (socket says 172.21.15.57:39710) 2024-04-21T08:51:37.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.702+0000 7face7fff640 1 -- 172.21.15.57:0/227323095 learned_addr learned my addr 172.21.15.57:0/227323095 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7facecb59640 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126508d0 0x55bd1264f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 -- 172.21.15.57:0/227323095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126508d0 msgr2=0x55bd1264f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126508d0 0x55bd1264f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 -- 172.21.15.57:0/227323095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 msgr2=0x55bd1265b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd1265b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 -- 172.21.15.57:0/227323095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd12573320 con 0x55bd1264fb60 2024-04-21T08:51:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7facd80a3dc0 0x7facd80a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7facd4011930 tx=0x7facd4009b60 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7face7fff640 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd1264ffe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7facd4009d90 tx=0x7facd401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d83884606c1713ac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7facd2ffd640 1 -- 172.21.15.57:0/227323095 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd4024070 con 0x55bd1264fb60 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7facd2ffd640 1 -- 172.21.15.57:0/227323095 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd4011570 con 0x55bd1264fb60 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.703+0000 7facd2ffd640 1 -- 172.21.15.57:0/227323095 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd401f930 con 0x55bd1264fb60 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 -- 172.21.15.57:0/227323095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 msgr2=0x55bd1264ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd1264ffe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7facd4009d90 tx=0x7facd401d000 comp rx=0 tx=0).stop 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 -- 172.21.15.57:0/227323095 shutdown_connections 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd1265b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd1264ffe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 --2- 172.21.15.57:0/227323095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126508d0 0x55bd1264f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 -- 172.21.15.57:0/227323095 >> 172.21.15.57:0/227323095 conn(0x55bd12616c90 msgr2=0x55bd1264a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 -- 172.21.15.57:0/227323095 shutdown_connections 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.704+0000 7facf19b4740 1 -- 172.21.15.57:0/227323095 wait complete. 2024-04-21T08:51:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.705+0000 7facf19b4740 1 -- start start 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.705+0000 7facf19b4740 1 Processor -- start 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.705+0000 7facf19b4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd12658ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.705+0000 7facf19b4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12655e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.705+0000 7facf19b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 0x55bd12656460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7facf19b4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55bd1265bdb0 con 0x55bd126508d0 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7facf19b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bd1253ceb0 con 0x55bd126590e0 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7facf19b4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55bd1265bf30 con 0x55bd1264fb60 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 0x55bd12656460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7face7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12655e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 0x55bd12656460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:48226/0 (socket says 172.21.15.57:48226) 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 -- 172.21.15.57:0/3520583089 learned_addr learned my addr 172.21.15.57:0/3520583089 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7facf19b4740 10 client.0 _ll_register_callbacks cb 0x55bd12569b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7facecb59640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd12658ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 -- 172.21.15.57:0/3520583089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 msgr2=0x55bd12658ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd12658ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 -- 172.21.15.57:0/3520583089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 msgr2=0x55bd12655e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12655e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7faced35a640 1 -- 172.21.15.57:0/3520583089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd122c7210 con 0x55bd126590e0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.706+0000 7face7fff640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12655e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7faced35a640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 0x55bd12656460 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7face00be130 tx=0x7face00be2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52ad1a067a8d2fb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facc7fff640 1 -- 172.21.15.57:0/3520583089 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0003e30 con 0x55bd126590e0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facc7fff640 1 -- 172.21.15.57:0/3520583089 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7face003dab0 con 0x55bd126590e0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facd4011570 con 0x55bd126590e0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facc7fff640 1 -- 172.21.15.57:0/3520583089 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7face00beb30 con 0x55bd126590e0 2024-04-21T08:51:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facf19b4740 10 client.26177 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.707+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bd12393620 con 0x55bd126590e0 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facc7fff640 1 -- 172.21.15.57:0/3520583089 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd4011570 con 0x55bd126590e0 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facc7fff640 1 -- 172.21.15.57:0/3520583089 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7face003c030 con 0x55bd126590e0 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facf19b4740 10 client.26177 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facf19b4740 10 client.26177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facf19b4740 20 client.26177 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facf19b4740 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd12668f20 0x55bd1266b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facf19b4740 4 client.26177 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd125c8880 con 0x55bd12668f20 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.708+0000 7facecb59640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd12668f20 0x55bd1266b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.709+0000 7facecb59640 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd12668f20 0x55bd1266b370 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49f8c07c2c686a51 server_cookie=47af6c0b7d006df4 in_seq=0 out_seq=0 2024-04-21T08:51:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.709+0000 7facc7fff640 10 client.26177 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facc7fff640 1 -- 172.21.15.57:0/3520583089 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd125c8880 con 0x55bd12668f20 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facc7fff640 10 client.26177 handle_command_reply: tid=0 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 client.26177 shutdown 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 msgr2=0x55bd12656460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 0x55bd12656460 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7face00be130 tx=0x7face00be2c0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 shutdown_connections 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd12668f20 0x55bd1266b370 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd126590e0 0x55bd12656460 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12655e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 --2- 172.21.15.57:0/3520583089 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd1264fb60 0x55bd12658ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 >> 172.21.15.57:0/3520583089 conn(0x55bd12616c90 msgr2=0x55bd1264c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facc7fff640 0 client.26177 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 shutdown_connections 2024-04-21T08:51:37.715 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.712+0000 7facf19b4740 1 -- 172.21.15.57:0/3520583089 wait complete. 2024-04-21T08:51:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.713+0000 7facf19b4740 20 client.26177 trim_cache size 0 max 16384 2024-04-21T08:51:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 Processor -- start 2024-04-21T08:51:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 -- start start 2024-04-21T08:51:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd12664cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12658580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126590e0 0x55bd12658ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7face81af410 con 0x55bd126508d0 2024-04-21T08:51:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7face81b1c70 con 0x55bd1264fb60 2024-04-21T08:51:37.731 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facf19b4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7face81afa60 con 0x55bd126590e0 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7face7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12658580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facecb59640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd12664cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7faced35a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126590e0 0x55bd12658ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7face7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12658580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41072/0 (socket says 172.21.15.57:41072) 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7facecb59640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd12664cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:48232/0 (socket says 172.21.15.57:48232) 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.730+0000 7face7fff640 1 -- 172.21.15.57:0/3366278993 learned_addr learned my addr 172.21.15.57:0/3366278993 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7face7fff640 1 -- 172.21.15.57:0/3366278993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126590e0 msgr2=0x55bd12658ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7face7fff640 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126590e0 0x55bd12658ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7face7fff640 1 -- 172.21.15.57:0/3366278993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 msgr2=0x55bd12664cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7face7fff640 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd12664cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7face7fff640 1 -- 172.21.15.57:0/3366278993 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1264ee30 con 0x55bd126508d0 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7faced35a640 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126590e0 0x55bd12658ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7facecb59640 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd12664cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:37.732 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.731+0000 7face7fff640 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12658580 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7facd4023040 tx=0x7facd401f470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3108bb7da788a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.732+0000 7facd27fc640 1 -- 172.21.15.57:0/3366278993 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd4024040 con 0x55bd126508d0 2024-04-21T08:51:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.732+0000 7facd27fc640 1 -- 172.21.15.57:0/3366278993 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd40097a0 con 0x55bd126508d0 2024-04-21T08:51:37.733 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.732+0000 7facd27fc640 1 -- 172.21.15.57:0/3366278993 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd40207c0 con 0x55bd126508d0 2024-04-21T08:51:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 -- 172.21.15.57:0/3366278993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 msgr2=0x55bd12658580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12658580 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7facd4023040 tx=0x7facd401f470 comp rx=0 tx=0).stop 2024-04-21T08:51:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 -- 172.21.15.57:0/3366278993 shutdown_connections 2024-04-21T08:51:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126590e0 0x55bd12658ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126508d0 0x55bd12658580 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 --2- 172.21.15.57:0/3366278993 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd12664cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 -- 172.21.15.57:0/3366278993 >> 172.21.15.57:0/3366278993 conn(0x55bd12616c90 msgr2=0x55bd12651060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 -- 172.21.15.57:0/3366278993 shutdown_connections 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 -- 172.21.15.57:0/3366278993 wait complete. 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 -- start start 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.733+0000 7facf19b4740 1 Processor -- start 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd126611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126508d0 0x55bd12661700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd126530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7face81b0130 con 0x55bd126590e0 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7face81b15a0 con 0x55bd1264fb60 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7face81b0800 con 0x55bd126508d0 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facf19b4740 10 client.0 _ll_register_callbacks cb 0x7face00ba050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd126611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd126611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:48244/0 (socket says 172.21.15.57:48244) 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 -- 172.21.15.57:0/3945616884 learned_addr learned my addr 172.21.15.57:0/3945616884 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7faced35a640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd126530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7face7fff640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126508d0 0x55bd12661700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 -- 172.21.15.57:0/3945616884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126508d0 msgr2=0x55bd12661700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126508d0 0x55bd12661700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 -- 172.21.15.57:0/3945616884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 msgr2=0x55bd126530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.734+0000 7facecb59640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd126530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facecb59640 1 -- 172.21.15.57:0/3945616884 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd12653630 con 0x55bd1264fb60 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facecb59640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd126611c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7facdc0095f0 tx=0x7facdc00f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=854b2dd800331cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facc7fff640 1 -- 172.21.15.57:0/3945616884 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc01c820 con 0x55bd1264fb60 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facc7fff640 1 -- 172.21.15.57:0/3945616884 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7facdc004020 con 0x55bd1264fb60 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facd40097a0 con 0x55bd1264fb60 2024-04-21T08:51:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facc7fff640 1 -- 172.21.15.57:0/3945616884 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7facdc02c050 con 0x55bd1264fb60 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facf19b4740 10 client.26183 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.735+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bd12659d20 con 0x55bd1264fb60 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facc7fff640 1 -- 172.21.15.57:0/3945616884 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facdc020030 con 0x55bd1264fb60 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facc7fff640 1 -- 172.21.15.57:0/3945616884 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7facdc0243d0 con 0x55bd1264fb60 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facf19b4740 10 client.26183 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facf19b4740 10 client.26183 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facf19b4740 20 client.26183 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facf19b4740 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd126cf500 0x55bd126d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facf19b4740 4 client.26183 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7facd40097a0 con 0x55bd126cf500 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7face7fff640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd126cf500 0x55bd126d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7facc7fff640 10 client.26183 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:37.736+0000 7face7fff640 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd126cf500 0x55bd126d18f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=257723decbcc254 server_cookie=e41c980e92171bd0 in_seq=0 out_seq=0 2024-04-21T08:51:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:37 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:37 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:37 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:51:37.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:37 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:37.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:37 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:37.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:37 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:51:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:37 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:51:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:37 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:51:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:37 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:51:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:38 smithi129 ceph-mon[31272]: pgmap v189: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.5 MiB/s wr, 2.57k op/s 2024-04-21T08:51:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:38 smithi057 ceph-mon[27968]: pgmap v189: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.5 MiB/s wr, 2.57k op/s 2024-04-21T08:51:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:38 smithi136 ceph-mon[31791]: pgmap v189: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.5 MiB/s wr, 2.57k op/s 2024-04-21T08:51:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:40 smithi129 ceph-mon[31272]: pgmap v190: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 1.83k op/s 2024-04-21T08:51:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:40 smithi057 ceph-mon[27968]: pgmap v190: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 1.83k op/s 2024-04-21T08:51:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:40 smithi136 ceph-mon[31791]: pgmap v190: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 1.83k op/s 2024-04-21T08:51:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:42 smithi129 ceph-mon[31272]: pgmap v191: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 1.67k op/s 2024-04-21T08:51:42.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:42 smithi057 ceph-mon[27968]: pgmap v191: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 1.67k op/s 2024-04-21T08:51:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:42 smithi136 ceph-mon[31791]: pgmap v191: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.0 MiB/s wr, 1.67k op/s 2024-04-21T08:51:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.433+0000 7facc7fff640 1 -- 172.21.15.57:0/3945616884 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7facd40097a0 con 0x55bd126cf500 2024-04-21T08:51:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.433+0000 7facc7fff640 10 client.26183 handle_command_reply: tid=0 2024-04-21T08:51:44.433 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.433+0000 7facf19b4740 1 client.26183 shutdown 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 msgr2=0x55bd126611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd126611c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7facdc0095f0 tx=0x7facdc00f7f0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 shutdown_connections 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bd126cf500 0x55bd126d18f0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bd126590e0 0x55bd126530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bd126508d0 0x55bd12661700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 --2- 172.21.15.57:0/3945616884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bd1264fb60 0x55bd126611c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 >> 172.21.15.57:0/3945616884 conn(0x55bd12616c90 msgr2=0x55bd12654740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:44.434 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facc7fff640 0 client.26183 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 shutdown_connections 2024-04-21T08:51:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 1 -- 172.21.15.57:0/3945616884 wait complete. 2024-04-21T08:51:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.434+0000 7facf19b4740 20 client.26183 trim_cache size 0 max 16384 2024-04-21T08:51:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:51:44.439 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:51:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:51:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:51:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7facd80a3dc0 msgr2=0x7facd80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7facd80a3dc0 0x7facd80a6280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7facd4011930 tx=0x7facd4009b60 comp rx=0 tx=0).stop 2024-04-21T08:51:44.440 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 msgr2=0x7face819ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 0x7face819ab20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7face0005e30 tx=0x7face00346d0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 shutdown_connections 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7facd80a3dc0 0x7facd80a6280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7face819a6a0 0x7face819ab20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7face8106170 0x7face81a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 --2- 172.21.15.57:0/1454034900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7face81010f0 0x7face81a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 >> 172.21.15.57:0/1454034900 conn(0x7face8077fd0 msgr2=0x7face8100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 shutdown_connections 2024-04-21T08:51:44.441 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:44.435+0000 7faceedbd640 1 -- 172.21.15.57:0/1454034900 wait complete. 2024-04-21T08:51:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:44 smithi129 ceph-mon[31272]: pgmap v192: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.3 MiB/s wr, 2.77k op/s 2024-04-21T08:51:44.838 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:44 smithi057 ceph-mon[27968]: pgmap v192: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.3 MiB/s wr, 2.77k op/s 2024-04-21T08:51:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:44 smithi136 ceph-mon[31791]: pgmap v192: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.3 MiB/s wr, 2.77k op/s 2024-04-21T08:51:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:46 smithi129 ceph-mon[31272]: pgmap v193: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.6 MiB/s wr, 2.27k op/s 2024-04-21T08:51:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:46 smithi057 ceph-mon[27968]: pgmap v193: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.6 MiB/s wr, 2.27k op/s 2024-04-21T08:51:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:46 smithi136 ceph-mon[31791]: pgmap v193: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 8.6 MiB/s wr, 2.27k op/s 2024-04-21T08:51:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:48 smithi129 ceph-mon[31272]: pgmap v194: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.3 MiB/s wr, 3.44k op/s 2024-04-21T08:51:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:48 smithi057 ceph-mon[27968]: pgmap v194: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.3 MiB/s wr, 3.44k op/s 2024-04-21T08:51:48.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:48 smithi136 ceph-mon[31791]: pgmap v194: 129 pgs: 129 active+clean; 1.2 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 9.3 MiB/s wr, 3.44k op/s 2024-04-21T08:51:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:50 smithi129 ceph-mon[31272]: pgmap v195: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 3.44k op/s 2024-04-21T08:51:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:50 smithi057 ceph-mon[27968]: pgmap v195: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 3.44k op/s 2024-04-21T08:51:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:50 smithi136 ceph-mon[31791]: pgmap v195: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.3 MiB/s wr, 3.44k op/s 2024-04-21T08:51:52.826 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:52 smithi129 ceph-mon[31272]: pgmap v196: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 3.19k op/s 2024-04-21T08:51:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:52 smithi057 ceph-mon[27968]: pgmap v196: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 3.19k op/s 2024-04-21T08:51:52.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:52 smithi136 ceph-mon[31791]: pgmap v196: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 3.19k op/s 2024-04-21T08:51:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:54 smithi129 ceph-mon[31272]: pgmap v197: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 3.98k op/s 2024-04-21T08:51:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:54 smithi057 ceph-mon[27968]: pgmap v197: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 3.98k op/s 2024-04-21T08:51:54.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:54 smithi136 ceph-mon[31791]: pgmap v197: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.8 MiB/s wr, 3.98k op/s 2024-04-21T08:51:56.493 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:51:56.493 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:51:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.715+0000 7f6259c93640 1 -- 172.21.15.57:0/1235050049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625411b730 msgr2=0x7f625411db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.715+0000 7f6259c93640 1 --2- 172.21.15.57:0/1235050049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625411b730 0x7f625411db20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6244009900 tx=0x7f624402bc60 comp rx=0 tx=0).stop 2024-04-21T08:51:56.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.720+0000 7f6259c93640 1 -- 172.21.15.57:0/1235050049 shutdown_connections 2024-04-21T08:51:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.720+0000 7f6259c93640 1 --2- 172.21.15.57:0/1235050049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625411b730 0x7f625411db20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.720+0000 7f6259c93640 1 --2- 172.21.15.57:0/1235050049 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f625410e7b0 0x7f625410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.720+0000 7f6259c93640 1 --2- 172.21.15.57:0/1235050049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f625410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.720+0000 7f6259c93640 1 -- 172.21.15.57:0/1235050049 >> 172.21.15.57:0/1235050049 conn(0x7f625406d070 msgr2=0x7f625406d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.725+0000 7f6259c93640 1 -- 172.21.15.57:0/1235050049 shutdown_connections 2024-04-21T08:51:56.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.727+0000 7f6259c93640 1 -- 172.21.15.57:0/1235050049 wait complete. 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.731+0000 7f6259c93640 1 Processor -- start 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 -- start start 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625410e7b0 0x7f6254118d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f6254111db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f625411b730 0x7f62541122f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6254120f70 con 0x7f625411b730 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6254120df0 con 0x7f62541106d0 2024-04-21T08:51:56.734 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.733+0000 7f6259c93640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f62541210f0 con 0x7f625410e7b0 2024-04-21T08:51:56.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.736+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f6254111db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.737 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.736+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f6254111db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:49414/0 (socket says 172.21.15.57:49414) 2024-04-21T08:51:56.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.736+0000 7f6252ffd640 1 -- 172.21.15.57:0/1338230353 learned_addr learned my addr 172.21.15.57:0/1338230353 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:56.738 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.737+0000 7f6253fff640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f625411b730 0x7f62541122f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.743+0000 7f6252ffd640 1 -- 172.21.15.57:0/1338230353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625410e7b0 msgr2=0x7f6254118d30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:51:56.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.743+0000 7f6252ffd640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625410e7b0 0x7f6254118d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.743+0000 7f6252ffd640 1 -- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f625411b730 msgr2=0x7f62541122f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.743+0000 7f6252ffd640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f625411b730 0x7f62541122f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.743+0000 7f6252ffd640 1 -- 172.21.15.57:0/1338230353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62541128e0 con 0x7f62541106d0 2024-04-21T08:51:56.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.748+0000 7f6252ffd640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f6254111db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f624c00ebc0 tx=0x7f624c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.756+0000 7f6250ff9640 1 -- 172.21.15.57:0/1338230353 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c010630 con 0x7f62541106d0 2024-04-21T08:51:56.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.756+0000 7f6250ff9640 1 -- 172.21.15.57:0/1338230353 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c010c80 con 0x7f62541106d0 2024-04-21T08:51:56.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.756+0000 7f6250ff9640 1 -- 172.21.15.57:0/1338230353 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c018b60 con 0x7f62541106d0 2024-04-21T08:51:56.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.757+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6254112bd0 con 0x7f62541106d0 2024-04-21T08:51:56.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.757+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6254116b80 con 0x7f62541106d0 2024-04-21T08:51:56.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.761+0000 7f625c966740 1 Processor -- start 2024-04-21T08:51:56.765 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.765+0000 7f6250ff9640 1 -- 172.21.15.57:0/1338230353 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f624c01f020 con 0x7f62541106d0 2024-04-21T08:51:56.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.766+0000 7f6250ff9640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f623c0a3f20 0x7f623c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.766+0000 7f6250ff9640 1 -- 172.21.15.57:0/1338230353 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f624c0d0e10 con 0x7f62541106d0 2024-04-21T08:51:56.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.768+0000 7f6253fff640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f623c0a3f20 0x7f623c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 -- start start 2024-04-21T08:51:56.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5bb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5c4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56467d5201e0 con 0x56467d5c4a80 2024-04-21T08:51:56.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56467d4bd390 con 0x56467d5bbea0 2024-04-21T08:51:56.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.769+0000 7f625c966740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56467d51c1a0 con 0x56467d5bb580 2024-04-21T08:51:56.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.770+0000 7f6253fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.770+0000 7f6253fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49924/0 (socket says 172.21.15.57:49924) 2024-04-21T08:51:56.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.770+0000 7f6253fff640 1 -- 172.21.15.57:0/1411138789 learned_addr learned my addr 172.21.15.57:0/1411138789 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:56.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.773+0000 7f6253fff640 1 -- 172.21.15.57:0/1411138789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 msgr2=0x56467d5bb960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:56.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.773+0000 7f6253fff640 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5bb960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.773+0000 7f6253fff640 1 -- 172.21.15.57:0/1411138789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 msgr2=0x56467d5c4540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:51:56.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.773+0000 7f6253fff640 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5c4540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.773+0000 7f6253fff640 1 -- 172.21.15.57:0/1411138789 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56467d2fc7d0 con 0x56467d5c4a80 2024-04-21T08:51:56.774 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.775+0000 7f6253fff640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f623c0a3f20 0x7f623c0a63e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6244006010 tx=0x7f624403d000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.775+0000 7f6253fff640 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c6e70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f624402c1f0 tx=0x7f624402c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a894df3ae9d170 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.776+0000 7f6239ffb640 1 -- 172.21.15.57:0/1411138789 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244005410 con 0x56467d5c4a80 2024-04-21T08:51:56.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.777+0000 7f6239ffb640 1 -- 172.21.15.57:0/1411138789 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6244051070 con 0x56467d5c4a80 2024-04-21T08:51:56.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.778+0000 7f625c966740 1 -- 172.21.15.57:0/1411138789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 msgr2=0x56467d5c6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.778+0000 7f625c966740 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c6e70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f624402c1f0 tx=0x7f624402c8e0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.779+0000 7f625c966740 1 -- 172.21.15.57:0/1411138789 shutdown_connections 2024-04-21T08:51:56.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.779+0000 7f625c966740 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c6e70 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.779+0000 7f625c966740 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.779+0000 7f625c966740 1 --2- 172.21.15.57:0/1411138789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5bb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.779+0000 7f625c966740 1 -- 172.21.15.57:0/1411138789 >> 172.21.15.57:0/1411138789 conn(0x56467d5b5cb0 msgr2=0x56467d5b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.784+0000 7f625c966740 1 -- 172.21.15.57:0/1411138789 shutdown_connections 2024-04-21T08:51:56.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.784+0000 7f625c966740 1 -- 172.21.15.57:0/1411138789 wait complete. 2024-04-21T08:51:56.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.784+0000 7f625c966740 1 -- start start 2024-04-21T08:51:56.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 Processor -- start 2024-04-21T08:51:56.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bbea0 0x56467d5c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56467d5c7860 con 0x56467d5bbea0 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56467d4d92f0 con 0x56467d5c4a80 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56467d5c79e0 con 0x56467d5bb580 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f625c966740 10 client.0 _ll_register_callbacks cb 0x56467d4e6580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f6253fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f6253fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:49430/0 (socket says 172.21.15.57:49430) 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.785+0000 7f6253fff640 1 -- 172.21.15.57:0/1173014409 learned_addr learned my addr 172.21.15.57:0/1173014409 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:56.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6252ffd640 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bbea0 0x56467d5c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6253fff640 1 -- 172.21.15.57:0/1173014409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 msgr2=0x56467d5c1a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6253fff640 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5c1a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6253fff640 1 -- 172.21.15.57:0/1173014409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bbea0 msgr2=0x56467d5c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6253fff640 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bbea0 0x56467d5c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6253fff640 1 -- 172.21.15.57:0/1173014409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56467d2323f0 con 0x56467d5c4a80 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.786+0000 7f6253fff640 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5c24c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6244005ab0 tx=0x7f62440048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3499d31aeb7e900f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.787+0000 7f6232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.787+0000 7f6232ffd640 1 -- 172.21.15.57:0/1173014409 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244059070 con 0x56467d5c4a80 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.787+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6244051070 con 0x56467d5c4a80 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.787+0000 7f6232ffd640 1 -- 172.21.15.57:0/1173014409 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f6244003800 con 0x56467d5c4a80 2024-04-21T08:51:56.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.788+0000 7f6232ffd640 1 -- 172.21.15.57:0/1173014409 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6244065050 con 0x56467d5c4a80 2024-04-21T08:51:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.790+0000 7f625c966740 10 client.26192 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:56.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.790+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56467d1f20f0 con 0x56467d5c4a80 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f6232ffd640 1 -- 172.21.15.57:0/1173014409 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6244051070 con 0x56467d5c4a80 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f6232ffd640 1 -- 172.21.15.57:0/1173014409 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6244040040 con 0x56467d5c4a80 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f625c966740 10 client.26192 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f625c966740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f625c966740 20 client.26192 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f625c966740 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d5f6980 0x56467d5f8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f625c966740 4 client.26192 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:56.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.799+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56467d52dc00 con 0x56467d5f6980 2024-04-21T08:51:56.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.801+0000 7f62537fe640 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d5f6980 0x56467d5f8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.814+0000 7f62537fe640 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d5f6980 0x56467d5f8d70 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6adaf13af0e3a server_cookie=f9aa7af2683573e in_seq=0 out_seq=0 2024-04-21T08:51:56.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.814+0000 7f6232ffd640 10 client.26192 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:56.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.822+0000 7f6232ffd640 1 -- 172.21.15.57:0/1173014409 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56467d52dc00 con 0x56467d5f6980 2024-04-21T08:51:56.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.822+0000 7f6232ffd640 10 client.26192 handle_command_reply: tid=0 2024-04-21T08:51:56.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.822+0000 7f625c966740 1 client.26192 shutdown 2024-04-21T08:51:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:56 smithi129 ceph-mon[31272]: pgmap v198: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 2.88k op/s 2024-04-21T08:51:56.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.828+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 msgr2=0x56467d5c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.828+0000 7f625c966740 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5c24c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6244005ab0 tx=0x7f62440048f0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.831+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 shutdown_connections 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.831+0000 7f625c966740 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d5f6980 0x56467d5f8d70 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.831+0000 7f625c966740 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5c24c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.831+0000 7f625c966740 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bbea0 0x56467d5c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.831+0000 7f625c966740 1 --2- 172.21.15.57:0/1173014409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.831+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 >> 172.21.15.57:0/1173014409 conn(0x56467d5b5cb0 msgr2=0x56467d5b7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.832+0000 7f6232ffd640 0 client.26192 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:56.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.836+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 shutdown_connections 2024-04-21T08:51:56.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.837+0000 7f625c966740 1 -- 172.21.15.57:0/1173014409 wait complete. 2024-04-21T08:51:56.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.838+0000 7f625c966740 20 client.26192 trim_cache size 0 max 16384 2024-04-21T08:51:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:56 smithi057 ceph-mon[27968]: pgmap v198: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 2.88k op/s 2024-04-21T08:51:56.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.882+0000 7f625c966740 1 Processor -- start 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.883+0000 7f625c966740 1 -- start start 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.883+0000 7f625c966740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bb580 0x56467d5c4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.883+0000 7f625c966740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bbea0 0x56467d5c0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.883+0000 7f625c966740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f625c966740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f624000dea0 con 0x56467d5bb580 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f625c966740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6240010700 con 0x56467d5c4a80 2024-04-21T08:51:56.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f62537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bb580 0x56467d5c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f625c966740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f624000e4f0 con 0x56467d5bbea0 2024-04-21T08:51:56.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f62537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bb580 0x56467d5c4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49942/0 (socket says 172.21.15.57:49942) 2024-04-21T08:51:56.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f62537fe640 1 -- 172.21.15.57:0/2020032298 learned_addr learned my addr 172.21.15.57:0/2020032298 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:56.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f6253fff640 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.884 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:56 smithi136 ceph-mon[31791]: pgmap v198: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.5 MiB/s wr, 2.88k op/s 2024-04-21T08:51:56.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.884+0000 7f6252ffd640 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bbea0 0x56467d5c0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.886+0000 7f6252ffd640 1 -- 172.21.15.57:0/2020032298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 msgr2=0x56467d5d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.886+0000 7f6252ffd640 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5d0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.886+0000 7f6252ffd640 1 -- 172.21.15.57:0/2020032298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bb580 msgr2=0x56467d5c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.886+0000 7f6252ffd640 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bb580 0x56467d5c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.886+0000 7f6252ffd640 1 -- 172.21.15.57:0/2020032298 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56467d5f90e0 con 0x56467d5bbea0 2024-04-21T08:51:56.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.887+0000 7f6252ffd640 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bbea0 0x56467d5c0980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f624c0ce1a0 tx=0x7f624c009a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e232f787777255e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.888+0000 7f62397fa640 1 -- 172.21.15.57:0/2020032298 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c0b7070 con 0x56467d5bbea0 2024-04-21T08:51:56.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.888+0000 7f62397fa640 1 -- 172.21.15.57:0/2020032298 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c09c9f0 con 0x56467d5bbea0 2024-04-21T08:51:56.889 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.888+0000 7f62397fa640 1 -- 172.21.15.57:0/2020032298 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c0b5020 con 0x56467d5bbea0 2024-04-21T08:51:56.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 -- 172.21.15.57:0/2020032298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bbea0 msgr2=0x56467d5c0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bbea0 0x56467d5c0980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f624c0ce1a0 tx=0x7f624c009a30 comp rx=0 tx=0).stop 2024-04-21T08:51:56.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 -- 172.21.15.57:0/2020032298 shutdown_connections 2024-04-21T08:51:56.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5c4a80 0x56467d5d0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bbea0 0x56467d5c0980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 --2- 172.21.15.57:0/2020032298 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5bb580 0x56467d5c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.891+0000 7f625c966740 1 -- 172.21.15.57:0/2020032298 >> 172.21.15.57:0/2020032298 conn(0x56467d5b5cb0 msgr2=0x56467d5b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.895+0000 7f625c966740 1 -- 172.21.15.57:0/2020032298 shutdown_connections 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.895+0000 7f625c966740 1 -- 172.21.15.57:0/2020032298 wait complete. 2024-04-21T08:51:56.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.896+0000 7f625c966740 1 -- start start 2024-04-21T08:51:56.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.896+0000 7f625c966740 1 Processor -- start 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.897+0000 7f625c966740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.897+0000 7f625c966740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.897+0000 7f625c966740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f625c966740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f624000ebc0 con 0x56467d5c4a80 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f625c966740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6240010030 con 0x56467d5bbea0 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f625c966740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f624000f290 con 0x56467d5bb580 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f625c966740 10 client.0 _ll_register_callbacks cb 0x7f624402c7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f62537fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f6252ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:49438/0 (socket says 172.21.15.57:49438) 2024-04-21T08:51:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.898+0000 7f6252ffd640 1 -- 172.21.15.57:0/231600545 learned_addr learned my addr 172.21.15.57:0/231600545 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.899+0000 7f6253fff640 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.899+0000 7f6252ffd640 1 -- 172.21.15.57:0/231600545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 msgr2=0x56467d5cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.899+0000 7f6252ffd640 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.899+0000 7f6252ffd640 1 -- 172.21.15.57:0/231600545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 msgr2=0x56467d5c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.899+0000 7f6252ffd640 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.899+0000 7f6252ffd640 1 -- 172.21.15.57:0/231600545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56467d5c9930 con 0x56467d5bbea0 2024-04-21T08:51:56.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.901+0000 7f6252ffd640 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5cd1f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f624c0993c0 tx=0x7f624c00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f65e94229d284852 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.905+0000 7f6232ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.905+0000 7f6232ffd640 1 -- 172.21.15.57:0/231600545 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c0b7030 con 0x56467d5bbea0 2024-04-21T08:51:56.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.905+0000 7f6232ffd640 1 -- 172.21.15.57:0/231600545 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7f624c0a51d0 con 0x56467d5bbea0 2024-04-21T08:51:56.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.906+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f624c09c9f0 con 0x56467d5bbea0 2024-04-21T08:51:56.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.910+0000 7f6232ffd640 1 -- 172.21.15.57:0/231600545 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f624c0ab050 con 0x56467d5bbea0 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.910+0000 7f625c966740 10 client.26198 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.910+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56467d65d7a0 con 0x56467d5bbea0 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.911+0000 7f6232ffd640 1 -- 172.21.15.57:0/231600545 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c09c9f0 con 0x56467d5bbea0 2024-04-21T08:51:56.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.911+0000 7f6232ffd640 1 -- 172.21.15.57:0/231600545 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f624c09be40 con 0x56467d5bbea0 2024-04-21T08:51:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.913+0000 7f625c966740 10 client.26198 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.913+0000 7f625c966740 10 client.26198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.913+0000 7f625c966740 20 client.26198 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.913+0000 7f625c966740 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d65de80 0x56467d6602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.913+0000 7f625c966740 4 client.26198 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:51:56.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.913+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56467d527a70 con 0x56467d65de80 2024-04-21T08:51:56.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.915+0000 7f62537fe640 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d65de80 0x56467d6602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:56.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.927+0000 7f62537fe640 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d65de80 0x56467d6602a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f19044dba6f403de server_cookie=9d00efb05cef071c in_seq=0 out_seq=0 2024-04-21T08:51:56.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.928+0000 7f6232ffd640 10 client.26198 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:56.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.933+0000 7f6232ffd640 1 -- 172.21.15.57:0/231600545 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x56467d527a70 con 0x56467d65de80 2024-04-21T08:51:56.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.933+0000 7f6232ffd640 10 client.26198 handle_command_reply: tid=0 2024-04-21T08:51:56.935 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.934+0000 7f625c966740 1 client.26198 shutdown 2024-04-21T08:51:56.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.939+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 msgr2=0x56467d5cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.939+0000 7f625c966740 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5cd1f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f624c0993c0 tx=0x7f624c00cf90 comp rx=0 tx=0).stop 2024-04-21T08:51:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.941+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 shutdown_connections 2024-04-21T08:51:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.941+0000 7f625c966740 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56467d65de80 0x56467d6602a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.941+0000 7f625c966740 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56467d5c4a80 0x56467d5c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.941+0000 7f625c966740 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56467d5bbea0 0x56467d5cd1f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.941+0000 7f625c966740 1 --2- 172.21.15.57:0/231600545 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56467d5bb580 0x56467d5cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.941+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 >> 172.21.15.57:0/231600545 conn(0x56467d5b5cb0 msgr2=0x56467d5c1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.942+0000 7f6232ffd640 0 client.26198 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:56.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.943+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 shutdown_connections 2024-04-21T08:51:56.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.945+0000 7f625c966740 1 -- 172.21.15.57:0/231600545 wait complete. 2024-04-21T08:51:56.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.947+0000 7f625c966740 20 client.26198 trim_cache size 0 max 16384 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (5716 inodes in the stack)", 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: "8b767c17-e785-4e87-bdf4-567a8488f353": { 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "8b767c17-e785-4e87-bdf4-567a8488f353", 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:51:56.948 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:51:56.949 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:51:56.949 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:51:56.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.950+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f623c0a3f20 msgr2=0x7f623c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.950+0000 7f6259c93640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f623c0a3f20 0x7f623c0a63e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6244006010 tx=0x7f624403d000 comp rx=0 tx=0).stop 2024-04-21T08:51:56.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.950+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 msgr2=0x7f6254111db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:56.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.950+0000 7f6259c93640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f6254111db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f624c00ebc0 tx=0x7f624c00c6a0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.951+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 shutdown_connections 2024-04-21T08:51:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.951+0000 7f6259c93640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f623c0a3f20 0x7f623c0a63e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.951+0000 7f6259c93640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f625411b730 0x7f62541122f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.951+0000 7f6259c93640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62541106d0 0x7f6254111db0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.951+0000 7f6259c93640 1 --2- 172.21.15.57:0/1338230353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f625410e7b0 0x7f6254118d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:56.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.951+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 >> 172.21.15.57:0/1338230353 conn(0x7f625406d070 msgr2=0x7f625411a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.953+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 shutdown_connections 2024-04-21T08:51:56.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:51:56.953+0000 7f6259c93640 1 -- 172.21.15.57:0/1338230353 wait complete. 2024-04-21T08:51:56.973 INFO:tasks.cephfs.filesystem:scrub status for tag:8b767c17-e785-4e87-bdf4-567a8488f353 - {'path': '/', 'tag': '8b767c17-e785-4e87-bdf4-567a8488f353', 'options': 'recursive,force'} 2024-04-21T08:51:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.586+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/2917719727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34110740 msgr2=0x7fbf34108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.586+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/2917719727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34110740 0x7fbf34108f50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009cd0 tx=0x7fbf2c02c1b0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/2917719727 shutdown_connections 2024-04-21T08:51:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/2917719727 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf3411b730 0x7fbf3411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/2917719727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110b20 0x7fbf34109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/2917719727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34110740 0x7fbf34108f50 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/2917719727 >> 172.21.15.57:0/2917719727 conn(0x7fbf3406d020 msgr2=0x7fbf3406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:57.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/2917719727 shutdown_connections 2024-04-21T08:51:57.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.589+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/2917719727 wait complete. 2024-04-21T08:51:57.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 Processor -- start 2024-04-21T08:51:57.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 -- start start 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 0x7fbf3411ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf3411b730 0x7fbf34114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34115420 0x7fbf341158d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbf34111810 con 0x7fbf34115420 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf34111690 con 0x7fbf34110740 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.590+0000 7fbf3a6fd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbf34111990 con 0x7fbf3411b730 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 0x7fbf3411ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 0x7fbf3411ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:49454/0 (socket says 172.21.15.57:49454) 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 -- 172.21.15.57:0/751182945 learned_addr learned my addr 172.21.15.57:0/751182945 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf337fe640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf3411b730 0x7fbf34114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34115420 0x7fbf341158d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 -- 172.21.15.57:0/751182945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf3411b730 msgr2=0x7fbf34114ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf3411b730 0x7fbf34114ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 -- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34115420 msgr2=0x7fbf341158d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34115420 0x7fbf341158d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.591+0000 7fbf33fff640 1 -- 172.21.15.57:0/751182945 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf341bc530 con 0x7fbf34110740 2024-04-21T08:51:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.592+0000 7fbf33fff640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 0x7fbf3411ae20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c037040 tx=0x7fbf2c035e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.593+0000 7fbf317fa640 1 -- 172.21.15.57:0/751182945 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c039040 con 0x7fbf34110740 2024-04-21T08:51:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.593+0000 7fbf317fa640 1 -- 172.21.15.57:0/751182945 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf2c044070 con 0x7fbf34110740 2024-04-21T08:51:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.594+0000 7fbf317fa640 1 -- 172.21.15.57:0/751182945 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c032820 con 0x7fbf34110740 2024-04-21T08:51:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.595+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf341bc7c0 con 0x7fbf34110740 2024-04-21T08:51:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.595+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf341bcc00 con 0x7fbf34110740 2024-04-21T08:51:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.599+0000 7fbf3d3c6740 1 Processor -- start 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.599+0000 7fbf3d3c6740 1 -- start start 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 0x557a5f663620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f663fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f66d0e0 0x557a5f66f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557a5f5c88d0 con 0x557a5f6648d0 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a5f565e80 con 0x557a5f66d0e0 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557a5f5c4890 con 0x557a5f663b60 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf317fa640 1 -- 172.21.15.57:0/751182945 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbf2c0329c0 con 0x7fbf34110740 2024-04-21T08:51:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f663fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf38c9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 0x557a5f663620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf38c9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 0x557a5f663620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49980/0 (socket says 172.21.15.57:49980) 2024-04-21T08:51:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf38c9a640 1 -- 172.21.15.57:0/860726940 learned_addr learned my addr 172.21.15.57:0/860726940 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.600+0000 7fbf337fe640 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f66d0e0 0x557a5f66f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf38c9a640 1 -- 172.21.15.57:0/860726940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 msgr2=0x557a5f663fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf317fa640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbf1c0a3ff0 0x7fbf1c0a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f663fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf38c9a640 1 -- 172.21.15.57:0/860726940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f66d0e0 msgr2=0x557a5f66f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f66d0e0 0x557a5f66f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf317fa640 1 -- 172.21.15.57:0/751182945 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbf2c0f2e00 con 0x7fbf34110740 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.601+0000 7fbf38c9a640 1 -- 172.21.15.57:0/860726940 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a5f587320 con 0x557a5f6648d0 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.602+0000 7fbf33fff640 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f663fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.602+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbf1c0a3ff0 0x7fbf1c0a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.602+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 0x557a5f663620 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009c20 tx=0x7fbf24011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ac4272fdce755d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.603+0000 7fbf1a7fc640 1 -- 172.21.15.57:0/860726940 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2401b850 con 0x557a5f6648d0 2024-04-21T08:51:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.603+0000 7fbf1a7fc640 1 -- 172.21.15.57:0/860726940 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf240193b0 con 0x557a5f6648d0 2024-04-21T08:51:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.603+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbf1c0a3ff0 0x7fbf1c0a64b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf24020000 tx=0x7fbf2400ecc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.604+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/860726940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 msgr2=0x557a5f663620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:57.604 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.604+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 0x557a5f663620 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009c20 tx=0x7fbf24011810 comp rx=0 tx=0).stop 2024-04-21T08:51:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.609+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/860726940 shutdown_connections 2024-04-21T08:51:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.609+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f66d0e0 0x557a5f66f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.609+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f663fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.609+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/860726940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f6648d0 0x557a5f663620 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.609+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/860726940 >> 172.21.15.57:0/860726940 conn(0x557a5f62ac90 msgr2=0x557a5f65e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:57.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.612+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/860726940 shutdown_connections 2024-04-21T08:51:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.614+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/860726940 wait complete. 2024-04-21T08:51:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.614+0000 7fbf3d3c6740 1 -- start start 2024-04-21T08:51:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.614+0000 7fbf3d3c6740 1 Processor -- start 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f66c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f669d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 0x557a5f66a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557a5f66fe60 con 0x557a5f663b60 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a5f550eb0 con 0x557a5f6648d0 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557a5f66ffe0 con 0x557a5f66d0e0 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf3d3c6740 10 client.0 _ll_register_callbacks cb 0x557a5f57db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf38c9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 0x557a5f66a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf38c9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 0x557a5f66a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:56916/0 (socket says 172.21.15.57:56916) 2024-04-21T08:51:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf38c9a640 1 -- 172.21.15.57:0/3240523811 learned_addr learned my addr 172.21.15.57:0/3240523811 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.615+0000 7fbf337fe640 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f669d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf38c9a640 1 -- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 msgr2=0x557a5f669d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f669d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf38c9a640 1 -- 172.21.15.57:0/3240523811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 msgr2=0x557a5f66c980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:51:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f66c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf38c9a640 1 -- 172.21.15.57:0/3240523811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a5f2db210 con 0x557a5f66d0e0 2024-04-21T08:51:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 0x557a5f66a250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf24019d90 tx=0x7fbf24017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d529a368ecab627e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf137fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:51:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf137fe640 1 -- 172.21.15.57:0/3240523811 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2401bc70 con 0x557a5f66d0e0 2024-04-21T08:51:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.616+0000 7fbf137fe640 1 -- 172.21.15.57:0/3240523811 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbf24019560 con 0x557a5f66d0e0 2024-04-21T08:51:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.617+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf240193b0 con 0x557a5f66d0e0 2024-04-21T08:51:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.617+0000 7fbf137fe640 1 -- 172.21.15.57:0/3240523811 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbf24019760 con 0x557a5f66d0e0 2024-04-21T08:51:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.620+0000 7fbf3d3c6740 10 client.26086 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.620+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557a5f3a7620 con 0x557a5f66d0e0 2024-04-21T08:51:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.625+0000 7fbf137fe640 1 -- 172.21.15.57:0/3240523811 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf240193b0 con 0x557a5f66d0e0 2024-04-21T08:51:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.625+0000 7fbf137fe640 1 -- 172.21.15.57:0/3240523811 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbf24005040 con 0x557a5f66d0e0 2024-04-21T08:51:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.625+0000 7fbf3d3c6740 10 client.26086 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.625+0000 7fbf3d3c6740 10 client.26086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.625+0000 7fbf3d3c6740 20 client.26086 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.626+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f67cfb0 0x557a5f67f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.626+0000 7fbf3d3c6740 4 client.26086 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:51:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.626+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a5f5dc880 con 0x557a5f67cfb0 2024-04-21T08:51:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.627+0000 7fbf33fff640 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f67cfb0 0x557a5f67f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.630+0000 7fbf33fff640 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f67cfb0 0x557a5f67f3a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4087da962c5dbf3 server_cookie=6aece02f9f32d227 in_seq=0 out_seq=0 2024-04-21T08:51:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.631+0000 7fbf137fe640 10 client.26086 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.632+0000 7fbf137fe640 1 -- 172.21.15.57:0/3240523811 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557a5f5dc880 con 0x557a5f67cfb0 2024-04-21T08:51:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.632+0000 7fbf137fe640 10 client.26086 handle_command_reply: tid=0 2024-04-21T08:51:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.632+0000 7fbf3d3c6740 1 client.26086 shutdown 2024-04-21T08:51:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.635+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 msgr2=0x557a5f66a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.635+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 0x557a5f66a250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbf24019d90 tx=0x7fbf24017000 comp rx=0 tx=0).stop 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 shutdown_connections 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f67cfb0 0x557a5f67f3a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f66d0e0 0x557a5f66a250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f669d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/3240523811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f66c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 >> 172.21.15.57:0/3240523811 conn(0x557a5f62ac90 msgr2=0x557a5f6605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.637+0000 7fbf137fe640 0 client.26086 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.638+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 shutdown_connections 2024-04-21T08:51:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.638+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/3240523811 wait complete. 2024-04-21T08:51:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.638+0000 7fbf3d3c6740 20 client.26086 trim_cache size 0 max 16384 2024-04-21T08:51:57.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 Processor -- start 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 -- start start 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f62ac90 0x557a5f663120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f668e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f6648d0 0x557a5f65d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbf341c49a0 con 0x557a5f663b60 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf341c7200 con 0x557a5f62ac90 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf3d3c6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbf341c4ff0 con 0x557a5f6648d0 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f668e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f668e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49984/0 (socket says 172.21.15.57:49984) 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 -- 172.21.15.57:0/4201600450 learned_addr learned my addr 172.21.15.57:0/4201600450 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f6648d0 0x557a5f65d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf33fff640 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f62ac90 0x557a5f663120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 -- 172.21.15.57:0/4201600450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f6648d0 msgr2=0x557a5f65d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f6648d0 0x557a5f65d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 -- 172.21.15.57:0/4201600450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f62ac90 msgr2=0x557a5f663120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f62ac90 0x557a5f663120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.664+0000 7fbf337fe640 1 -- 172.21.15.57:0/4201600450 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a5f67f9d0 con 0x557a5f663b60 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.665+0000 7fbf337fe640 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f668e70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbf341166e0 tx=0x7fbf280090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719509ec992aa09d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.665+0000 7fbf19ffb640 1 -- 172.21.15.57:0/4201600450 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf28014c00 con 0x557a5f663b60 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.665+0000 7fbf19ffb640 1 -- 172.21.15.57:0/4201600450 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf28014da0 con 0x557a5f663b60 2024-04-21T08:51:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.665+0000 7fbf19ffb640 1 -- 172.21.15.57:0/4201600450 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf280167b0 con 0x557a5f663b60 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/4201600450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 msgr2=0x557a5f668e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f668e70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbf341166e0 tx=0x7fbf280090e0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/4201600450 shutdown_connections 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f6648d0 0x557a5f65d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f663b60 0x557a5f668e70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/4201600450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f62ac90 0x557a5f663120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/4201600450 >> 172.21.15.57:0/4201600450 conn(0x557a5f66a3b0 msgr2=0x557a5f65d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/4201600450 shutdown_connections 2024-04-21T08:51:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.666+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/4201600450 wait complete. 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 -- start start 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 Processor -- start 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 0x557a5f6754b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f6665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 0x557a5f6754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 0x557a5f6754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49998/0 (socket says 172.21.15.57:49998) 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf33fff640 1 -- 172.21.15.57:0/997975331 learned_addr learned my addr 172.21.15.57:0/997975331 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f666b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbf341c56c0 con 0x557a5f62ac90 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf341c6b30 con 0x557a5f6648d0 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.667+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbf341c5d90 con 0x557a5f663b60 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf3d3c6740 10 client.0 _ll_register_callbacks cb 0x7fbf24009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf337fe640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f6665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f666b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 -- 172.21.15.57:0/997975331 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 msgr2=0x557a5f6665c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f6665c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 -- 172.21.15.57:0/997975331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 msgr2=0x557a5f6754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 0x557a5f6754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a5f6670f0 con 0x557a5f6648d0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf337fe640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f6665c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf33fff640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 0x557a5f6754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf38c9a640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f666b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf24000c00 tx=0x7fbf24009e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4f5eb22266b24a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.668+0000 7fbf137fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf137fe640 1 -- 172.21.15.57:0/997975331 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf24019dc0 con 0x557a5f6648d0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf28014da0 con 0x557a5f6648d0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf137fe640 1 -- 172.21.15.57:0/997975331 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9387+0+0 (secure 0 0 0) 0x7fbf24019900 con 0x557a5f6648d0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf137fe640 1 -- 172.21.15.57:0/997975331 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbf24052050 con 0x557a5f6648d0 2024-04-21T08:51:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 10 client.26213 fetch_fsmap learned FSMap version 21 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557a5f6e3220 con 0x557a5f6648d0 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf137fe640 1 -- 172.21.15.57:0/997975331 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf28014da0 con 0x557a5f6648d0 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf137fe640 1 -- 172.21.15.57:0/997975331 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbf24045450 con 0x557a5f6648d0 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 10 client.26213 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 20 client.26213 populate_metadata read hostname 'smithi057' 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f66d0e0 0x557a5f6e58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 4 client.26213 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.669+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a5f5683d0 con 0x557a5f66d0e0 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.670+0000 7fbf33fff640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f66d0e0 0x557a5f6e58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.670+0000 7fbf33fff640 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f66d0e0 0x557a5f6e58e0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a2e5c77eab8d141 server_cookie=ddf4b98d63b47f90 in_seq=0 out_seq=0 2024-04-21T08:51:57.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:51:57.670+0000 7fbf137fe640 10 client.26213 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:51:58.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:51:58 smithi129 ceph-mon[31272]: pgmap v199: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 3.33k op/s 2024-04-21T08:51:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:51:58 smithi057 ceph-mon[27968]: pgmap v199: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 3.33k op/s 2024-04-21T08:51:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:51:58 smithi136 ceph-mon[31791]: pgmap v199: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 3.33k op/s 2024-04-21T08:52:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:00 smithi129 ceph-mon[31272]: pgmap v200: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.3 MiB/s wr, 2.49k op/s 2024-04-21T08:52:00.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:00 smithi057 ceph-mon[27968]: pgmap v200: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.3 MiB/s wr, 2.49k op/s 2024-04-21T08:52:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:00 smithi136 ceph-mon[31791]: pgmap v200: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.3 MiB/s wr, 2.49k op/s 2024-04-21T08:52:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:02 smithi129 ceph-mon[31272]: pgmap v201: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 1.56k op/s 2024-04-21T08:52:02.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:02 smithi057 ceph-mon[27968]: pgmap v201: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 1.56k op/s 2024-04-21T08:52:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:02 smithi136 ceph-mon[31791]: pgmap v201: 129 pgs: 129 active+clean; 1.2 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.1 MiB/s wr, 1.56k op/s 2024-04-21T08:52:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:04 smithi129 ceph-mon[31272]: pgmap v202: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 2.76k op/s 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.833+0000 7fbf137fe640 1 -- 172.21.15.57:0/997975331 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a5f5683d0 con 0x557a5f66d0e0 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.833+0000 7fbf137fe640 10 client.26213 handle_command_reply: tid=0 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.833+0000 7fbf3d3c6740 1 client.26213 shutdown 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 msgr2=0x557a5f666b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f666b00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbf24000c00 tx=0x7fbf24009e40 comp rx=0 tx=0).stop 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 shutdown_connections 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557a5f66d0e0 0x557a5f6e58e0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a5f6648d0 0x557a5f666b00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557a5f663b60 0x557a5f6665c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 --2- 172.21.15.57:0/997975331 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557a5f62ac90 0x557a5f6754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 >> 172.21.15.57:0/997975331 conn(0x557a5f66a3b0 msgr2=0x557a5f65d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf137fe640 0 client.26213 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:04.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 shutdown_connections 2024-04-21T08:52:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 1 -- 172.21.15.57:0/997975331 wait complete. 2024-04-21T08:52:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3d3c6740 20 client.26213 trim_cache size 0 max 16384 2024-04-21T08:52:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:52:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:52:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:52:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbf1c0a3ff0 msgr2=0x7fbf1c0a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbf1c0a3ff0 0x7fbf1c0a64b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf24020000 tx=0x7fbf2400ecc0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 msgr2=0x7fbf3411ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.834+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 0x7fbf3411ae20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c037040 tx=0x7fbf2c035e00 comp rx=0 tx=0).stop 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 shutdown_connections 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbf1c0a3ff0 0x7fbf1c0a64b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf34115420 0x7fbf341158d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf3411b730 0x7fbf34114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 --2- 172.21.15.57:0/751182945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf34110740 0x7fbf3411ae20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 >> 172.21.15.57:0/751182945 conn(0x7fbf3406d020 msgr2=0x7fbf3411bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 shutdown_connections 2024-04-21T08:52:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:04.835+0000 7fbf3a6fd640 1 -- 172.21.15.57:0/751182945 wait complete. 2024-04-21T08:52:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:04 smithi057 ceph-mon[27968]: pgmap v202: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 2.76k op/s 2024-04-21T08:52:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:04 smithi136 ceph-mon[31791]: pgmap v202: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 7.0 MiB/s wr, 2.76k op/s 2024-04-21T08:52:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:06 smithi129 ceph-mon[31272]: pgmap v203: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 1.98k op/s 2024-04-21T08:52:06.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:06 smithi057 ceph-mon[27968]: pgmap v203: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 1.98k op/s 2024-04-21T08:52:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:06 smithi136 ceph-mon[31791]: pgmap v203: 129 pgs: 129 active+clean; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.4 MiB/s wr, 1.98k op/s 2024-04-21T08:52:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:07 smithi129 ceph-mon[31272]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:52:07.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:07 smithi057 ceph-mon[27968]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:52:07.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:07 smithi136 ceph-mon[31791]: osdmap e95: 12 total, 12 up, 12 in 2024-04-21T08:52:08.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:08 smithi057 ceph-mon[27968]: pgmap v205: 129 pgs: 10 active+clean+snaptrim, 91 active+clean, 28 active+clean+snaptrim_wait; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 2.36k op/s 2024-04-21T08:52:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:08 smithi136 ceph-mon[31791]: pgmap v205: 129 pgs: 10 active+clean+snaptrim, 91 active+clean, 28 active+clean+snaptrim_wait; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 2.36k op/s 2024-04-21T08:52:09.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:08 smithi129 ceph-mon[31272]: pgmap v205: 129 pgs: 10 active+clean+snaptrim, 91 active+clean, 28 active+clean+snaptrim_wait; 1.2 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 2.36k op/s 2024-04-21T08:52:09.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:09 smithi057 ceph-mon[27968]: pgmap v206: 129 pgs: 16 active+clean+snaptrim, 67 active+clean, 46 active+clean+snaptrim_wait; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 2.41k op/s 2024-04-21T08:52:09.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:09 smithi136 ceph-mon[31791]: pgmap v206: 129 pgs: 16 active+clean+snaptrim, 67 active+clean, 46 active+clean+snaptrim_wait; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 2.41k op/s 2024-04-21T08:52:10.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:09 smithi129 ceph-mon[31272]: pgmap v206: 129 pgs: 16 active+clean+snaptrim, 67 active+clean, 46 active+clean+snaptrim_wait; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 2.41k op/s 2024-04-21T08:52:12.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:12 smithi057 ceph-mon[27968]: pgmap v207: 129 pgs: 16 active+clean+snaptrim, 67 active+clean, 46 active+clean+snaptrim_wait; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 2.41k op/s 2024-04-21T08:52:12.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:12 smithi136 ceph-mon[31791]: pgmap v207: 129 pgs: 16 active+clean+snaptrim, 67 active+clean, 46 active+clean+snaptrim_wait; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 2.41k op/s 2024-04-21T08:52:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:12 smithi129 ceph-mon[31272]: pgmap v207: 129 pgs: 16 active+clean+snaptrim, 67 active+clean, 46 active+clean+snaptrim_wait; 1.2 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.2 MiB/s wr, 2.41k op/s 2024-04-21T08:52:14.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:14 smithi136 ceph-mon[31791]: pgmap v208: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:14 smithi129 ceph-mon[31272]: pgmap v208: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:14 smithi057 ceph-mon[27968]: pgmap v208: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:16.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:16 smithi136 ceph-mon[31791]: pgmap v209: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:16.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:16 smithi129 ceph-mon[31272]: pgmap v209: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:16 smithi057 ceph-mon[27968]: pgmap v209: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.2 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 1.26k op/s 2024-04-21T08:52:17.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.977+0000 7f0820ac0640 1 -- 172.21.15.57:0/1200675967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 msgr2=0x7f081c105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:17.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.977+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1200675967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 0x7f081c105990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f081400acc0 tx=0x7f0814034720 comp rx=0 tx=0).stop 2024-04-21T08:52:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.979+0000 7f0820ac0640 1 -- 172.21.15.57:0/1200675967 shutdown_connections 2024-04-21T08:52:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.979+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1200675967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c10d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.979+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1200675967 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c100810 0x7f081c100c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.979+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1200675967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 0x7f081c105990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.979+0000 7f0820ac0640 1 -- 172.21.15.57:0/1200675967 >> 172.21.15.57:0/1200675967 conn(0x7f081c076160 msgr2=0x7f081c076570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:17.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.979+0000 7f0820ac0640 1 -- 172.21.15.57:0/1200675967 shutdown_connections 2024-04-21T08:52:17.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.982+0000 7f0820ac0640 1 -- 172.21.15.57:0/1200675967 wait complete. 2024-04-21T08:52:17.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.984+0000 7f0820ac0640 1 Processor -- start 2024-04-21T08:52:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 -- start start 2024-04-21T08:52:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c100810 0x7f081c1a7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c1a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 0x7f081c1abee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:17.987 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1100c0 con 0x7f081c100810 2024-04-21T08:52:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f081c10ff40 con 0x7f081c1011d0 2024-04-21T08:52:17.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.986+0000 7f0820ac0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f081c110240 con 0x7f081c1055b0 2024-04-21T08:52:17.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.987+0000 7f0819d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c1a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.987+0000 7f0819d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c1a7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55168/0 (socket says 172.21.15.57:55168) 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.987+0000 7f0819d9b640 1 -- 172.21.15.57:0/1304877271 learned_addr learned my addr 172.21.15.57:0/1304877271 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.987+0000 7f081a59c640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c100810 0x7f081c1a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.988+0000 7f0819d9b640 1 -- 172.21.15.57:0/1304877271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 msgr2=0x7f081c1abee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.988+0000 7f0819d9b640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 0x7f081c1abee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.988+0000 7f0819d9b640 1 -- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c100810 msgr2=0x7f081c1a7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.988+0000 7f0819d9b640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c100810 0x7f081c1a7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.988+0000 7f0819d9b640 1 -- 172.21.15.57:0/1304877271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081c1ac5e0 con 0x7f081c1011d0 2024-04-21T08:52:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.989+0000 7f0819d9b640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c1a7bb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f080c007890 tx=0x7f080c007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.995+0000 7f080b7fe640 1 -- 172.21.15.57:0/1304877271 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c019020 con 0x7f081c1011d0 2024-04-21T08:52:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.996+0000 7f080b7fe640 1 -- 172.21.15.57:0/1304877271 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f080c015470 con 0x7f081c1011d0 2024-04-21T08:52:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.996+0000 7f080b7fe640 1 -- 172.21.15.57:0/1304877271 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c014630 con 0x7f081c1011d0 2024-04-21T08:52:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.999+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081c1ac8d0 con 0x7f081c1011d0 2024-04-21T08:52:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:17.999+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f081c06b970 con 0x7f081c1011d0 2024-04-21T08:52:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.007+0000 7f0823781740 1 Processor -- start 2024-04-21T08:52:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.008+0000 7f0823781740 1 -- start start 2024-04-21T08:52:18.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f0823781740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd7e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f0823781740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd7efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f0823781740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd8a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f0823781740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x560e0dce38d0 con 0x560e0dd880e0 2024-04-21T08:52:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f0823781740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e0dc80e80 con 0x560e0dd7f8d0 2024-04-21T08:52:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f0823781740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560e0dcdf890 con 0x560e0dd7eb60 2024-04-21T08:52:18.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f080b7fe640 1 -- 172.21.15.57:0/1304877271 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f080c01c020 con 0x7f081c1011d0 2024-04-21T08:52:18.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f081ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd7e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f081ad9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd7e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55170/0 (socket says 172.21.15.57:55170) 2024-04-21T08:52:18.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.011+0000 7f081ad9d640 1 -- 172.21.15.57:0/2284454835 learned_addr learned my addr 172.21.15.57:0/2284454835 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:18.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f080b7fe640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08040a3fa0 0x7f08040a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f080b7fe640 1 -- 172.21.15.57:0/1304877271 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f080c0d0810 con 0x7f081c1011d0 2024-04-21T08:52:18.013 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f081ad9d640 1 -- 172.21.15.57:0/2284454835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 msgr2=0x560e0dd7efe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:52:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f081ad9d640 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd7efe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f081ad9d640 1 -- 172.21.15.57:0/2284454835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 msgr2=0x560e0dd8a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f081ad9d640 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd8a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.012+0000 7f081ad9d640 1 -- 172.21.15.57:0/2284454835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e0dca2320 con 0x560e0dd7f8d0 2024-04-21T08:52:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.013+0000 7f081ad9d640 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd7e620 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08100096f0 tx=0x7f0810009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a48fe1df3508f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:18.014 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.013+0000 7f081ad9d640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08040a3fa0 0x7f08040a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.017+0000 7f0808ff9640 1 -- 172.21.15.57:0/2284454835 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0810015020 con 0x560e0dd7f8d0 2024-04-21T08:52:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.017+0000 7f0808ff9640 1 -- 172.21.15.57:0/2284454835 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081000be30 con 0x560e0dd7f8d0 2024-04-21T08:52:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.018+0000 7f0823781740 1 -- 172.21.15.57:0/2284454835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 msgr2=0x560e0dd7e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.018+0000 7f0823781740 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd7e620 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08100096f0 tx=0x7f0810009bb0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.020+0000 7f0823781740 1 -- 172.21.15.57:0/2284454835 shutdown_connections 2024-04-21T08:52:18.020 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.020+0000 7f0823781740 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd8a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.020+0000 7f0823781740 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd7efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.020+0000 7f0823781740 1 --2- 172.21.15.57:0/2284454835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd7e620 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.021 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.020+0000 7f0823781740 1 -- 172.21.15.57:0/2284454835 >> 172.21.15.57:0/2284454835 conn(0x560e0dd45c90 msgr2=0x560e0dd79850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:18.023 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.022+0000 7f081ad9d640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08040a3fa0 0x7f08040a6460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0810016000 tx=0x7f0810019780 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:18.027 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.025+0000 7f0823781740 1 -- 172.21.15.57:0/2284454835 shutdown_connections 2024-04-21T08:52:18.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.027+0000 7f0823781740 1 -- 172.21.15.57:0/2284454835 wait complete. 2024-04-21T08:52:18.028 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.028+0000 7f0823781740 1 -- start start 2024-04-21T08:52:18.029 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.028+0000 7f0823781740 1 Processor -- start 2024-04-21T08:52:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.029+0000 7f0823781740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd87900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f0823781740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd84c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f0823781740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd851d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f0823781740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x560e0dd8ae60 con 0x560e0dd880e0 2024-04-21T08:52:18.030 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f0823781740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e0dc6beb0 con 0x560e0dd7f8d0 2024-04-21T08:52:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f0823781740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560e0dd8afe0 con 0x560e0dd7eb60 2024-04-21T08:52:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f0823781740 10 client.0 _ll_register_callbacks cb 0x560e0dc98b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.030+0000 7f081a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd87900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.031+0000 7f0819d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd84c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.031+0000 7f081a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd87900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43514/0 (socket says 172.21.15.57:43514) 2024-04-21T08:52:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.031+0000 7f081a59c640 1 -- 172.21.15.57:0/801785586 learned_addr learned my addr 172.21.15.57:0/801785586 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.034+0000 7f0819d9b640 1 -- 172.21.15.57:0/801785586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 msgr2=0x560e0dd87900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:18.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.034+0000 7f0819d9b640 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd87900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.034+0000 7f0819d9b640 1 -- 172.21.15.57:0/801785586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 msgr2=0x560e0dd851d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.034+0000 7f0819d9b640 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd851d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.034+0000 7f0819d9b640 1 -- 172.21.15.57:0/801785586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e0d9f6210 con 0x560e0dd7f8d0 2024-04-21T08:52:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.035+0000 7f0819d9b640 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd84c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f080c0cdaf0 tx=0x7f080c09be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd53ec6eac42fb13 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:18.035 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.035+0000 7f0801ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:52:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.037+0000 7f0801ffb640 1 -- 172.21.15.57:0/801785586 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c0b0e60 con 0x560e0dd7f8d0 2024-04-21T08:52:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.038+0000 7f0801ffb640 1 -- 172.21.15.57:0/801785586 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f080c0cdd70 con 0x560e0dd7f8d0 2024-04-21T08:52:18.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.038+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f081000be30 con 0x560e0dd7f8d0 2024-04-21T08:52:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.041+0000 7f0801ffb640 1 -- 172.21.15.57:0/801785586 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f080c0cdf70 con 0x560e0dd7f8d0 2024-04-21T08:52:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.041+0000 7f0823781740 10 client.26222 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.041+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e0dac2620 con 0x560e0dd7f8d0 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0801ffb640 1 -- 172.21.15.57:0/801785586 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081000be30 con 0x560e0dd7f8d0 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0801ffb640 1 -- 172.21.15.57:0/801785586 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f080c09b8e0 con 0x560e0dd7f8d0 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0823781740 10 client.26222 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0823781740 10 client.26222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0823781740 20 client.26222 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0823781740 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0dd984e0 0x560e0dd9a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0823781740 4 client.26222 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:18.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.046+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e0dcf7880 con 0x560e0dd984e0 2024-04-21T08:52:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.047+0000 7f081a59c640 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0dd984e0 0x560e0dd9a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.048+0000 7f081a59c640 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0dd984e0 0x560e0dd9a930 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a75a6cd3a5d6aa3 server_cookie=108f45f79a3832ae in_seq=0 out_seq=0 2024-04-21T08:52:18.050 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.050+0000 7f0801ffb640 10 client.26222 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.056+0000 7f0801ffb640 1 -- 172.21.15.57:0/801785586 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e0dcf7880 con 0x560e0dd984e0 2024-04-21T08:52:18.056 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.056+0000 7f0801ffb640 10 client.26222 handle_command_reply: tid=0 2024-04-21T08:52:18.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.058+0000 7f0823781740 1 client.26222 shutdown 2024-04-21T08:52:18.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.064+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 msgr2=0x560e0dd84c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:18.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.064+0000 7f0823781740 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd84c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f080c0cdaf0 tx=0x7f080c09be10 comp rx=0 tx=0).stop 2024-04-21T08:52:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.066+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 shutdown_connections 2024-04-21T08:52:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.066+0000 7f0823781740 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0dd984e0 0x560e0dd9a930 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.066+0000 7f0823781740 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd851d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.066+0000 7f0823781740 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd84c90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.066+0000 7f0823781740 1 --2- 172.21.15.57:0/801785586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd87900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.066+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 >> 172.21.15.57:0/801785586 conn(0x560e0dd45c90 msgr2=0x560e0dd7b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.067+0000 7f0801ffb640 0 client.26222 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.068+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 shutdown_connections 2024-04-21T08:52:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.070+0000 7f0823781740 1 -- 172.21.15.57:0/801785586 wait complete. 2024-04-21T08:52:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.070+0000 7f0823781740 20 client.26222 trim_cache size 0 max 16384 2024-04-21T08:52:18.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.103+0000 7f0823781740 1 Processor -- start 2024-04-21T08:52:18.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.103+0000 7f0823781740 1 -- start start 2024-04-21T08:52:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.104+0000 7f0823781740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd93db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.104+0000 7f0823781740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd7f8d0 0x560e0dd83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.104+0000 7f081a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd93db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.104+0000 7f081a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd93db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43516/0 (socket says 172.21.15.57:43516) 2024-04-21T08:52:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.105+0000 7f081a59c640 1 -- 172.21.15.57:0/468139280 learned_addr learned my addr 172.21.15.57:0/468139280 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.105+0000 7f0819d9b640 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd7f8d0 0x560e0dd83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.105+0000 7f0823781740 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd880e0 0x560e0dd78750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.105+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f081c073c70 con 0x560e0dd7f8d0 2024-04-21T08:52:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.105+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1c7500 con 0x560e0dd880e0 2024-04-21T08:52:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.105+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1c52f0 con 0x560e0dd7eb60 2024-04-21T08:52:18.107 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.106+0000 7f081ad9d640 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd880e0 0x560e0dd78750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.107+0000 7f0819d9b640 1 -- 172.21.15.57:0/468139280 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 msgr2=0x560e0dd93db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.107+0000 7f0819d9b640 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd93db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.107+0000 7f0819d9b640 1 -- 172.21.15.57:0/468139280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd880e0 msgr2=0x560e0dd78750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.107+0000 7f0819d9b640 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd880e0 0x560e0dd78750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.107+0000 7f0819d9b640 1 -- 172.21.15.57:0/468139280 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e0dd75120 con 0x560e0dd7f8d0 2024-04-21T08:52:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.108+0000 7f0819d9b640 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd7f8d0 0x560e0dd83e10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f080c098820 tx=0x7f080c0a08f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11edca6ddd7ae16 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.118+0000 7f0803fff640 1 -- 172.21.15.57:0/468139280 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080c0a5030 con 0x560e0dd7f8d0 2024-04-21T08:52:18.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.118+0000 7f0803fff640 1 -- 172.21.15.57:0/468139280 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f080c0b5070 con 0x560e0dd7f8d0 2024-04-21T08:52:18.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.119+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd7f8d0 msgr2=0x560e0dd83e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:18.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.119+0000 7f0823781740 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd7f8d0 0x560e0dd83e10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f080c098820 tx=0x7f080c0a08f0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.121+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 shutdown_connections 2024-04-21T08:52:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.121+0000 7f0823781740 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd880e0 0x560e0dd78750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.121+0000 7f0823781740 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd7f8d0 0x560e0dd83e10 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.121+0000 7f0823781740 1 --2- 172.21.15.57:0/468139280 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd93db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.121+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 >> 172.21.15.57:0/468139280 conn(0x560e0dd45c90 msgr2=0x560e0dd78040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:18.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.126+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 shutdown_connections 2024-04-21T08:52:18.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.126+0000 7f0823781740 1 -- 172.21.15.57:0/468139280 wait complete. 2024-04-21T08:52:18.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.126+0000 7f0823781740 1 -- start start 2024-04-21T08:52:18.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.127+0000 7f0823781740 1 Processor -- start 2024-04-21T08:52:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.127+0000 7f0823781740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd907e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.127+0000 7f0823781740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd82200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.127+0000 7f0823781740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd82940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f0823781740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1c59c0 con 0x560e0dd880e0 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f0823781740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1c6e30 con 0x560e0dd7f8d0 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd907e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd907e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43532/0 (socket says 172.21.15.57:43532) 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 -- 172.21.15.57:0/3594204055 learned_addr learned my addr 172.21.15.57:0/3594204055 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f081c1c6090 con 0x560e0dd7eb60 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f0823781740 10 client.0 _ll_register_callbacks cb 0x7f080c015b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:18.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 -- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 msgr2=0x560e0dd82200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd82200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 -- 172.21.15.57:0/3594204055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 msgr2=0x560e0dd82940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.128+0000 7f081a59c640 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd82940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.129+0000 7f081a59c640 1 -- 172.21.15.57:0/3594204055 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e0dd81020 con 0x560e0dd7eb60 2024-04-21T08:52:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.129+0000 7f081a59c640 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd907e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f081400acc0 tx=0x7f081403c5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b95cd90617f1db0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:18.132 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.131+0000 7f0801ffb640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:52:18.132 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.131+0000 7f0801ffb640 1 -- 172.21.15.57:0/3594204055 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081404a070 con 0x560e0dd7eb60 2024-04-21T08:52:18.132 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.131+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f080c0b5070 con 0x560e0dd7eb60 2024-04-21T08:52:18.132 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.131+0000 7f0801ffb640 1 -- 172.21.15.57:0/3594204055 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f081403c750 con 0x560e0dd7eb60 2024-04-21T08:52:18.133 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.132+0000 7f0801ffb640 1 -- 172.21.15.57:0/3594204055 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0814056050 con 0x560e0dd7eb60 2024-04-21T08:52:18.134 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.134+0000 7f0823781740 10 client.26104 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:18.134 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.134+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e0ddfde90 con 0x560e0dd7eb60 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0801ffb640 1 -- 172.21.15.57:0/3594204055 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f081403c9e0 con 0x560e0dd7eb60 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0801ffb640 1 -- 172.21.15.57:0/3594204055 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f08140463e0 con 0x560e0dd7eb60 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0823781740 10 client.26104 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0823781740 10 client.26104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0823781740 20 client.26104 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0823781740 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0ddfe670 0x560e0de00a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0823781740 4 client.26104 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.139+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f080c0b5070 con 0x560e0ddfe670 2024-04-21T08:52:18.146 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.142+0000 7f0819d9b640 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0ddfe670 0x560e0de00a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.156+0000 7f0819d9b640 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0ddfe670 0x560e0de00a90 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b70b73b3c58756fb server_cookie=1d4026e3c8724990 in_seq=0 out_seq=0 2024-04-21T08:52:18.160 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:18.158+0000 7f0801ffb640 10 client.26104 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:18.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:18 smithi136 ceph-mon[31791]: pgmap v210: 129 pgs: 19 active+clean+snaptrim, 59 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 1.30k op/s 2024-04-21T08:52:18.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:18 smithi129 ceph-mon[31272]: pgmap v210: 129 pgs: 19 active+clean+snaptrim, 59 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 1.30k op/s 2024-04-21T08:52:18.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:18 smithi057 ceph-mon[27968]: pgmap v210: 129 pgs: 19 active+clean+snaptrim, 59 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 1.30k op/s 2024-04-21T08:52:20.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:20 smithi136 ceph-mon[31791]: pgmap v211: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 940 op/s 2024-04-21T08:52:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:20 smithi129 ceph-mon[31272]: pgmap v211: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 940 op/s 2024-04-21T08:52:20.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:20 smithi057 ceph-mon[27968]: pgmap v211: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 940 op/s 2024-04-21T08:52:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0801ffb640 1 -- 172.21.15.57:0/3594204055 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f080c0b5070 con 0x560e0ddfe670 2024-04-21T08:52:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0801ffb640 10 client.26104 handle_command_reply: tid=0 2024-04-21T08:52:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0823781740 1 client.26104 shutdown 2024-04-21T08:52:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 msgr2=0x560e0dd907e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:20.943 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0823781740 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd907e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f081400acc0 tx=0x7f081403c5a0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 shutdown_connections 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0823781740 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0ddfe670 0x560e0de00a90 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.941+0000 7f0823781740 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0dd880e0 0x560e0dd82940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0823781740 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0dd7f8d0 0x560e0dd82200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0823781740 1 --2- 172.21.15.57:0/3594204055 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0dd7eb60 0x560e0dd907e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 >> 172.21.15.57:0/3594204055 conn(0x560e0dd45c90 msgr2=0x560e0dd788d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0801ffb640 0 client.26104 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 shutdown_connections 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0823781740 1 -- 172.21.15.57:0/3594204055 wait complete. 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.942+0000 7f0823781740 20 client.26104 trim_cache size 0 max 16384 2024-04-21T08:52:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:52:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:52:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:52:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:52:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08040a3fa0 msgr2=0x7f08040a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08040a3fa0 0x7f08040a6460 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0810016000 tx=0x7f0810019780 comp rx=0 tx=0).stop 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 msgr2=0x7f081c1a7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c1a7bb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f080c007890 tx=0x7f080c007d50 comp rx=0 tx=0).stop 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 shutdown_connections 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08040a3fa0 0x7f08040a6460 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f081c1055b0 0x7f081c1abee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f081c1011d0 0x7f081c1a7bb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 --2- 172.21.15.57:0/1304877271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f081c100810 0x7f081c1a7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 >> 172.21.15.57:0/1304877271 conn(0x7f081c076160 msgr2=0x7f081c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 shutdown_connections 2024-04-21T08:52:20.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:20.943+0000 7f0820ac0640 1 -- 172.21.15.57:0/1304877271 wait complete. 2024-04-21T08:52:22.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:22 smithi136 ceph-mon[31791]: pgmap v212: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 576 op/s 2024-04-21T08:52:22.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:22 smithi129 ceph-mon[31272]: pgmap v212: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 576 op/s 2024-04-21T08:52:22.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:22 smithi057 ceph-mon[27968]: pgmap v212: 129 pgs: 18 active+clean+snaptrim, 60 active+clean, 51 active+clean+snaptrim_wait; 1.2 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 576 op/s 2024-04-21T08:52:24.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:24 smithi129 ceph-mon[31272]: pgmap v213: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 1.37k op/s 2024-04-21T08:52:24.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:24 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:24 smithi057 ceph-mon[27968]: pgmap v213: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 1.37k op/s 2024-04-21T08:52:24.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:24 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:24 smithi136 ceph-mon[31791]: pgmap v213: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.7 MiB/s wr, 1.37k op/s 2024-04-21T08:52:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:24 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:26.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:26 smithi129 ceph-mon[31272]: pgmap v214: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.1 MiB/s wr, 1.13k op/s 2024-04-21T08:52:26.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:26 smithi057 ceph-mon[27968]: pgmap v214: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.1 MiB/s wr, 1.13k op/s 2024-04-21T08:52:26.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:26 smithi136 ceph-mon[31791]: pgmap v214: 129 pgs: 15 active+clean+snaptrim, 78 active+clean, 36 active+clean+snaptrim_wait; 1.2 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.1 MiB/s wr, 1.13k op/s 2024-04-21T08:52:26.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:26.974 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:27.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.169+0000 7fb97715b640 1 -- 172.21.15.57:0/2154877897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb96809ace0 msgr2=0x7fb968004730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.169+0000 7fb97715b640 1 --2- 172.21.15.57:0/2154877897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb96809ace0 0x7fb968004730 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009900 tx=0x7fb96c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 -- 172.21.15.57:0/2154877897 shutdown_connections 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 --2- 172.21.15.57:0/2154877897 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb9680986d0 0x7fb968098b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 --2- 172.21.15.57:0/2154877897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb968004c70 0x7fb9680050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 --2- 172.21.15.57:0/2154877897 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb96809ace0 0x7fb968004730 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 -- 172.21.15.57:0/2154877897 >> 172.21.15.57:0/2154877897 conn(0x7fb96800a730 msgr2=0x7fb96800ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 -- 172.21.15.57:0/2154877897 shutdown_connections 2024-04-21T08:52:27.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.170+0000 7fb97715b640 1 -- 172.21.15.57:0/2154877897 wait complete. 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb97715b640 1 Processor -- start 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb97715b640 1 -- start start 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb97715b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 0x7fb9681368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb97715b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9680986d0 0x7fb968136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb976159640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 0x7fb9681368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb976159640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 0x7fb9681368e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:44976/0 (socket says 172.21.15.57:44976) 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb975958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9680986d0 0x7fb968136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb975958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9680986d0 0x7fb968136e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40232/0 (socket says 172.21.15.57:40232) 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb976159640 1 -- 172.21.15.57:0/2881884579 learned_addr learned my addr 172.21.15.57:0/2881884579 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.171+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb96809ace0 0x7fb968131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb9680ac4a0 con 0x7fb96809ace0 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9680ac320 con 0x7fb9680986d0 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb9680ac620 con 0x7fb968004c70 2024-04-21T08:52:27.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb97695a640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb96809ace0 0x7fb968131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb976159640 1 -- 172.21.15.57:0/2881884579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9680986d0 msgr2=0x7fb968136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb976159640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9680986d0 0x7fb968136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb976159640 1 -- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb96809ace0 msgr2=0x7fb968131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb976159640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb96809ace0 0x7fb968131a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.172+0000 7fb976159640 1 -- 172.21.15.57:0/2881884579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb968132300 con 0x7fb968004c70 2024-04-21T08:52:27.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.173+0000 7fb976159640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 0x7fb9681368e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb96c02ebb0 tx=0x7fb96c02ebe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.176 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.173+0000 7fb9677fe640 1 -- 172.21.15.57:0/2881884579 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c041020 con 0x7fb968004c70 2024-04-21T08:52:27.176 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.173+0000 7fb9677fe640 1 -- 172.21.15.57:0/2881884579 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb96c02ee00 con 0x7fb968004c70 2024-04-21T08:52:27.176 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.173+0000 7fb9677fe640 1 -- 172.21.15.57:0/2881884579 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c0346a0 con 0x7fb968004c70 2024-04-21T08:52:27.176 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.173+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9681325f0 con 0x7fb968004c70 2024-04-21T08:52:27.176 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.173+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96813e600 con 0x7fb968004c70 2024-04-21T08:52:27.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.177+0000 7fb9677fe640 1 -- 172.21.15.57:0/2881884579 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb96c044020 con 0x7fb968004c70 2024-04-21T08:52:27.178 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.177+0000 7fb97afb1740 1 Processor -- start 2024-04-21T08:52:27.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.178+0000 7fb97afb1740 1 -- start start 2024-04-21T08:52:27.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.178+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad18af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.178+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad19030 0x55aeaad216d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb97695a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad18af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb976159640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad19030 0x55aeaad216d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb976159640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad19030 0x55aeaad216d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46092/0 (socket says 172.21.15.57:46092) 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad24000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb97afb1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55aeaac7d1e0 con 0x55aeaad19030 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb97afb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55aeaac1a390 con 0x55aeaad18710 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb97afb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55aeaac791a0 con 0x55aeaad21c10 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb976159640 1 -- 172.21.15.57:0/1824979132 learned_addr learned my addr 172.21.15.57:0/1824979132 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb9677fe640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb95c0a3de0 0x7fb95c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.179+0000 7fb975958640 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad24000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb9677fe640 1 -- 172.21.15.57:0/2881884579 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7fb96c0f2430 con 0x7fb968004c70 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb975958640 1 -- 172.21.15.57:0/1824979132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 msgr2=0x55aeaad18af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb975958640 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad18af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb975958640 1 -- 172.21.15.57:0/1824979132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad19030 msgr2=0x55aeaad216d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb975958640 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad19030 0x55aeaad216d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb975958640 1 -- 172.21.15.57:0/1824979132 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeaaa597d0 con 0x55aeaad21c10 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.180+0000 7fb97695a640 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad18af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.181+0000 7fb975958640 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad24000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb968133240 tx=0x7fb960011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b5de137936436eb server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.181+0000 7fb97695a640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb95c0a3de0 0x7fb95c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.181+0000 7fb964bfe640 1 -- 172.21.15.57:0/1824979132 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb960009d70 con 0x55aeaad21c10 2024-04-21T08:52:27.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.181+0000 7fb964bfe640 1 -- 172.21.15.57:0/1824979132 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb960005020 con 0x55aeaad21c10 2024-04-21T08:52:27.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 -- 172.21.15.57:0/1824979132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 msgr2=0x55aeaad24000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad24000 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb968133240 tx=0x7fb960011810 comp rx=0 tx=0).stop 2024-04-21T08:52:27.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 -- 172.21.15.57:0/1824979132 shutdown_connections 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad24000 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad19030 0x55aeaad216d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1824979132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad18af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.183+0000 7fb97afb1740 1 -- 172.21.15.57:0/1824979132 >> 172.21.15.57:0/1824979132 conn(0x55aeaad12e40 msgr2=0x55aeaad13220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.189+0000 7fb97afb1740 1 -- 172.21.15.57:0/1824979132 shutdown_connections 2024-04-21T08:52:27.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.190+0000 7fb97695a640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb95c0a3de0 0x7fb95c0a62a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb9700676b0 tx=0x7fb970067e20 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.191+0000 7fb97afb1740 1 -- 172.21.15.57:0/1824979132 wait complete. 2024-04-21T08:52:27.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.191+0000 7fb97afb1740 1 -- start start 2024-04-21T08:52:27.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.191+0000 7fb97afb1740 1 Processor -- start 2024-04-21T08:52:27.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.192+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad1ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.192+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad1f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.192+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.192+0000 7fb97afb1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55aeaad24a50 con 0x55aeaad18710 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97afb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55aeaac362f0 con 0x55aeaad19030 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45000/0 (socket says 172.21.15.57:45000) 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 -- 172.21.15.57:0/1074687031 learned_addr learned my addr 172.21.15.57:0/1074687031 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55aeaad24bd0 con 0x55aeaad21c10 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97afb1740 10 client.0 _ll_register_callbacks cb 0x55aeaac43580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb975958640 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad1f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 -- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 msgr2=0x55aeaad1f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad1f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 -- 172.21.15.57:0/1074687031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 msgr2=0x55aeaad1ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad1ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.193+0000 7fb97695a640 1 -- 172.21.15.57:0/1074687031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeaa98f3f0 con 0x55aeaad21c10 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.194+0000 7fb97695a640 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1f630 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb970067ac0 tx=0x7fb97006c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f34f100bca2a5f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.194+0000 7fb959ffb640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.194+0000 7fb959ffb640 1 -- 172.21.15.57:0/1074687031 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970079b60 con 0x55aeaad21c10 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.194+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aeaac8ac00 con 0x55aeaad21c10 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.195+0000 7fb959ffb640 1 -- 172.21.15.57:0/1074687031 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7fb970079d00 con 0x55aeaad21c10 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.195+0000 7fb959ffb640 1 -- 172.21.15.57:0/1074687031 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb970095050 con 0x55aeaad21c10 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.195+0000 7fb97afb1740 10 client.26116 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:27.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.195+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55aeaa94f0f0 con 0x55aeaad21c10 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb959ffb640 1 -- 172.21.15.57:0/1074687031 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55aeaac8ac00 con 0x55aeaad21c10 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb959ffb640 1 -- 172.21.15.57:0/1074687031 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb970086560 con 0x55aeaad21c10 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb97afb1740 10 client.26116 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb97afb1740 10 client.26116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb97afb1740 20 client.26116 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:27.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad31ff0 0x55aeaad344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb97afb1740 4 client.26116 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:27.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.197+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aeaac84a70 con 0x55aeaad31ff0 2024-04-21T08:52:27.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.199+0000 7fb976159640 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad31ff0 0x55aeaad344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.200+0000 7fb976159640 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad31ff0 0x55aeaad344b0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88e61416cecba9f8 server_cookie=b8f4140ddcd3c0f4 in_seq=0 out_seq=0 2024-04-21T08:52:27.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.200+0000 7fb959ffb640 10 client.26116 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.203+0000 7fb959ffb640 1 -- 172.21.15.57:0/1074687031 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aeaac84a70 con 0x55aeaad31ff0 2024-04-21T08:52:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.204+0000 7fb959ffb640 10 client.26116 handle_command_reply: tid=0 2024-04-21T08:52:27.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.204+0000 7fb97afb1740 1 client.26116 shutdown 2024-04-21T08:52:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.205+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 msgr2=0x55aeaad1f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.205+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1f630 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb970067ac0 tx=0x7fb97006c040 comp rx=0 tx=0).stop 2024-04-21T08:52:27.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 shutdown_connections 2024-04-21T08:52:27.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad31ff0 0x55aeaad344b0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1f630 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad1f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1074687031 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad1ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.207 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 >> 172.21.15.57:0/1074687031 conn(0x55aeaad12e40 msgr2=0x55aeaad14300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb959ffb640 0 client.26116 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.206+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 shutdown_connections 2024-04-21T08:52:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.207+0000 7fb97afb1740 1 -- 172.21.15.57:0/1074687031 wait complete. 2024-04-21T08:52:27.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.207+0000 7fb97afb1740 20 client.26116 trim_cache size 0 max 16384 2024-04-21T08:52:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.241+0000 7fb97afb1740 1 Processor -- start 2024-04-21T08:52:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.242+0000 7fb97afb1740 1 -- start start 2024-04-21T08:52:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad2d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 0x55aeaad39dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad21c10 0x55aeaad3a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97afb1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb968169830 con 0x55aeaad21c10 2024-04-21T08:52:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97afb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb96816bff0 con 0x55aeaad18710 2024-04-21T08:52:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97afb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb968169e80 con 0x55aeaad19030 2024-04-21T08:52:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 0x55aeaad39dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 0x55aeaad39dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45006/0 (socket says 172.21.15.57:45006) 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 -- 172.21.15.57:0/1062380511 learned_addr learned my addr 172.21.15.57:0/1062380511 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb97695a640 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad21c10 0x55aeaad3a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb976159640 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad2d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 -- 172.21.15.57:0/1062380511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 msgr2=0x55aeaad2d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad2d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 -- 172.21.15.57:0/1062380511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad21c10 msgr2=0x55aeaad3a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad21c10 0x55aeaad3a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.243+0000 7fb975958640 1 -- 172.21.15.57:0/1062380511 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeaad18e60 con 0x55aeaad19030 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.244+0000 7fb97695a640 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad21c10 0x55aeaad3a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:27.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.244+0000 7fb975958640 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 0x55aeaad39dc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb960014040 tx=0x7fb9600046a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a01a27abaa00003 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.244+0000 7fb95bfff640 1 -- 172.21.15.57:0/1062380511 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb960004030 con 0x55aeaad19030 2024-04-21T08:52:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.245+0000 7fb95bfff640 1 -- 172.21.15.57:0/1062380511 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9600035f0 con 0x55aeaad19030 2024-04-21T08:52:27.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.245+0000 7fb95bfff640 1 -- 172.21.15.57:0/1062380511 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9600226a0 con 0x55aeaad19030 2024-04-21T08:52:27.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 -- 172.21.15.57:0/1062380511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 msgr2=0x55aeaad39dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 0x55aeaad39dc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb960014040 tx=0x7fb9600046a0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 -- 172.21.15.57:0/1062380511 shutdown_connections 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad21c10 0x55aeaad3a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad19030 0x55aeaad39dc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 --2- 172.21.15.57:0/1062380511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad18710 0x55aeaad2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 -- 172.21.15.57:0/1062380511 >> 172.21.15.57:0/1062380511 conn(0x55aeaad12e40 msgr2=0x55aeaad13220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 -- 172.21.15.57:0/1062380511 shutdown_connections 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 -- 172.21.15.57:0/1062380511 wait complete. 2024-04-21T08:52:27.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 -- start start 2024-04-21T08:52:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.247+0000 7fb97afb1740 1 Processor -- start 2024-04-21T08:52:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad29f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad2a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb96816a4d0 con 0x55aeaad18710 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb96816b920 con 0x55aeaad19030 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb96816ab80 con 0x55aeaad21c10 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97afb1740 10 client.0 _ll_register_callbacks cb 0x7fb970069ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb976159640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad29f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb975958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad2a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb975958640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad2a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40250/0 (socket says 172.21.15.57:40250) 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb975958640 1 -- 172.21.15.57:0/40032944 learned_addr learned my addr 172.21.15.57:0/40032944 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.248+0000 7fb97695a640 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb976159640 1 -- 172.21.15.57:0/40032944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 msgr2=0x55aeaad1ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb976159640 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb976159640 1 -- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 msgr2=0x55aeaad2a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb976159640 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad2a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb976159640 1 -- 172.21.15.57:0/40032944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeaad1b3a0 con 0x55aeaad18710 2024-04-21T08:52:27.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb976159640 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad29f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x55aeaad181e0 tx=0x7fb96c02bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73817c12a63358 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.249+0000 7fb959ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.252+0000 7fb959ffb640 1 -- 172.21.15.57:0/40032944 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c0efe60 con 0x55aeaad18710 2024-04-21T08:52:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.252+0000 7fb959ffb640 1 -- 172.21.15.57:0/40032944 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7fb96c0f2150 con 0x55aeaad18710 2024-04-21T08:52:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.252+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9600035f0 con 0x55aeaad18710 2024-04-21T08:52:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb959ffb640 1 -- 172.21.15.57:0/40032944 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb96c0bdbb0 con 0x55aeaad18710 2024-04-21T08:52:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 10 client.16764 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:27.253 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55aeaad905f0 con 0x55aeaad18710 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb959ffb640 1 -- 172.21.15.57:0/40032944 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9600035f0 con 0x55aeaad18710 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb959ffb640 1 -- 172.21.15.57:0/40032944 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb96c0ba650 con 0x55aeaad18710 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 10 client.16764 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 20 client.16764 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad90cf0 0x55aeaad93190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 4 client.16764 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.253+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aeaac00590 con 0x55aeaad90cf0 2024-04-21T08:52:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.254+0000 7fb975958640 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad90cf0 0x55aeaad93190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.255+0000 7fb975958640 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad90cf0 0x55aeaad93190 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aad1fdec55a62846 server_cookie=e8ff90ffa116697d in_seq=0 out_seq=0 2024-04-21T08:52:27.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.255+0000 7fb959ffb640 10 client.16764 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.260+0000 7fb959ffb640 1 -- 172.21.15.57:0/40032944 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aeaac00590 con 0x55aeaad90cf0 2024-04-21T08:52:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.260+0000 7fb959ffb640 10 client.16764 handle_command_reply: tid=0 2024-04-21T08:52:27.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.260+0000 7fb97afb1740 1 client.16764 shutdown 2024-04-21T08:52:27.263 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.263+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 msgr2=0x55aeaad29f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.263 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.263+0000 7fb97afb1740 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad29f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x55aeaad181e0 tx=0x7fb96c02bc50 comp rx=0 tx=0).stop 2024-04-21T08:52:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 shutdown_connections 2024-04-21T08:52:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb97afb1740 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55aeaad90cf0 0x55aeaad93190 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb97afb1740 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55aeaad21c10 0x55aeaad1ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.269 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb97afb1740 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55aeaad19030 0x55aeaad2a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb97afb1740 1 --2- 172.21.15.57:0/40032944 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55aeaad18710 0x55aeaad29f00 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 >> 172.21.15.57:0/40032944 conn(0x55aeaad12e40 msgr2=0x55aeaad19e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.270 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.269+0000 7fb959ffb640 0 client.16764 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:27.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.272+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 shutdown_connections 2024-04-21T08:52:27.276 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.276+0000 7fb97afb1740 1 -- 172.21.15.57:0/40032944 wait complete. 2024-04-21T08:52:27.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.276+0000 7fb97afb1740 20 client.16764 trim_cache size 0 max 16384 2024-04-21T08:52:27.280 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:52:27.280 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:52:27.280 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:52:27.280 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.280+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb95c0a3de0 msgr2=0x7fb95c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.280+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb95c0a3de0 0x7fb95c0a62a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb9700676b0 tx=0x7fb970067e20 comp rx=0 tx=0).stop 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.280+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 msgr2=0x7fb9681368e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.280+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 0x7fb9681368e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb96c02ebb0 tx=0x7fb96c02ebe0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.282+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 shutdown_connections 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.282+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb95c0a3de0 0x7fb95c0a62a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.283 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.282+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb96809ace0 0x7fb968131a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.282+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9680986d0 0x7fb968136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.282+0000 7fb97715b640 1 --2- 172.21.15.57:0/2881884579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb968004c70 0x7fb9681368e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.282+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 >> 172.21.15.57:0/2881884579 conn(0x7fb96800a730 msgr2=0x7fb968092050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.283+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 shutdown_connections 2024-04-21T08:52:27.284 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.283+0000 7fb97715b640 1 -- 172.21.15.57:0/2881884579 wait complete. 2024-04-21T08:52:27.303 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:52:27.303 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:52:27.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.479+0000 7ff0ff64e640 1 -- 172.21.15.57:0/2712951306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8101b60 msgr2=0x7ff0f810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.479+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/2712951306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8101b60 0x7ff0f810dc90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009cd0 tx=0x7ff0f002c120 comp rx=0 tx=0).stop 2024-04-21T08:52:27.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.480+0000 7ff0ff64e640 1 -- 172.21.15.57:0/2712951306 shutdown_connections 2024-04-21T08:52:27.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.480+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/2712951306 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8101b60 0x7ff0f810dc90 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.480+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/2712951306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f81011a0 0x7ff0f8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.480+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/2712951306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f8105e30 0x7ff0f8106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.480+0000 7ff0ff64e640 1 -- 172.21.15.57:0/2712951306 >> 172.21.15.57:0/2712951306 conn(0x7ff0f8077e30 msgr2=0x7ff0f8078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.483+0000 7ff0ff64e640 1 -- 172.21.15.57:0/2712951306 shutdown_connections 2024-04-21T08:52:27.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.485+0000 7ff0ff64e640 1 -- 172.21.15.57:0/2712951306 wait complete. 2024-04-21T08:52:27.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.486+0000 7ff0ff64e640 1 Processor -- start 2024-04-21T08:52:27.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 -- start start 2024-04-21T08:52:27.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f81011a0 0x7ff0f810e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f8101b60 0x7ff0f810eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 0x7ff0f810f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f8112300 con 0x7ff0f8105e30 2024-04-21T08:52:27.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f8112180 con 0x7ff0f81011a0 2024-04-21T08:52:27.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.487+0000 7ff0ff64e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f8112480 con 0x7ff0f8101b60 2024-04-21T08:52:27.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.488+0000 7ff0fdbeb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 0x7ff0f810f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.488+0000 7ff0fdbeb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 0x7ff0f810f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46140/0 (socket says 172.21.15.57:46140) 2024-04-21T08:52:27.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.488+0000 7ff0fdbeb640 1 -- 172.21.15.57:0/3012383149 learned_addr learned my addr 172.21.15.57:0/3012383149 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.488+0000 7ff0fd3ea640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f81011a0 0x7ff0f810e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.489+0000 7ff0fdbeb640 1 -- 172.21.15.57:0/3012383149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f8101b60 msgr2=0x7ff0f810eb90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:52:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.489+0000 7ff0fdbeb640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f8101b60 0x7ff0f810eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.489+0000 7ff0fdbeb640 1 -- 172.21.15.57:0/3012383149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f81011a0 msgr2=0x7ff0f810e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.489+0000 7ff0fdbeb640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f81011a0 0x7ff0f810e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.489+0000 7ff0fdbeb640 1 -- 172.21.15.57:0/3012383149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f8072aa0 con 0x7ff0f8105e30 2024-04-21T08:52:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.490+0000 7ff0fdbeb640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 0x7ff0f810f0d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0037040 tx=0x7ff0f0035ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.490+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0039040 con 0x7ff0f8105e30 2024-04-21T08:52:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.490+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f0044070 con 0x7ff0f8105e30 2024-04-21T08:52:27.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.491+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0004330 con 0x7ff0f8105e30 2024-04-21T08:52:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.491+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f8072d30 con 0x7ff0f8105e30 2024-04-21T08:52:27.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.491+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f8073220 con 0x7ff0f8105e30 2024-04-21T08:52:27.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.493+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f8102570 con 0x7ff0f8105e30 2024-04-21T08:52:27.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.497+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff0f00044f0 con 0x7ff0f8105e30 2024-04-21T08:52:27.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.498+0000 7ff0ee7fc640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0e40a3f20 0x7ff0e40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:27.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.498+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7ff0f00f3350 con 0x7ff0f8105e30 2024-04-21T08:52:27.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.499+0000 7ff0fd3ea640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0e40a3f20 0x7ff0e40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:27.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.502+0000 7ff0fd3ea640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0e40a3f20 0x7ff0e40a63e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4006fd0 tx=0x7ff0f4008040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:27.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.507+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0f00bf690 con 0x7ff0f8105e30 2024-04-21T08:52:27.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.963+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0f8073da0 con 0x7ff0f8105e30 2024-04-21T08:52:27.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.964+0000 7ff0ee7fc640 1 -- 172.21.15.57:0/3012383149 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7ff0f00bef90 con 0x7ff0f8105e30 2024-04-21T08:52:27.965 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:52:27.966 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:52:27.966 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:52:27.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.968+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0e40a3f20 msgr2=0x7ff0e40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0e40a3f20 0x7ff0e40a63e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4006fd0 tx=0x7ff0f4008040 comp rx=0 tx=0).stop 2024-04-21T08:52:27.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 msgr2=0x7ff0f810f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:27.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 0x7ff0f810f0d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0037040 tx=0x7ff0f0035ce0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.970 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 shutdown_connections 2024-04-21T08:52:27.970 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0e40a3f20 0x7ff0e40a63e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f8105e30 0x7ff0f810f0d0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f8101b60 0x7ff0f810eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 --2- 172.21.15.57:0/3012383149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f81011a0 0x7ff0f810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:27.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.969+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 >> 172.21.15.57:0/3012383149 conn(0x7ff0f8077e30 msgr2=0x7ff0f80fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:27.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.971+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 shutdown_connections 2024-04-21T08:52:27.971 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:27.971+0000 7ff0ff64e640 1 -- 172.21.15.57:0/3012383149 wait complete. 2024-04-21T08:52:27.991 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:52:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.172+0000 7fc775f15640 1 -- 172.21.15.57:0/2608103120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 msgr2=0x7fc77010ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.172+0000 7fc775f15640 1 --2- 172.21.15.57:0/2608103120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc77010ed60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc768009be0 tx=0x7fc76802bf80 comp rx=0 tx=0).stop 2024-04-21T08:52:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.174+0000 7fc775f15640 1 -- 172.21.15.57:0/2608103120 shutdown_connections 2024-04-21T08:52:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.174+0000 7fc775f15640 1 --2- 172.21.15.57:0/2608103120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc770109910 0x7fc7701157d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.174+0000 7fc775f15640 1 --2- 172.21.15.57:0/2608103120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc770108f50 0x7fc7701093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.174+0000 7fc775f15640 1 --2- 172.21.15.57:0/2608103120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc77010ed60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.174+0000 7fc775f15640 1 -- 172.21.15.57:0/2608103120 >> 172.21.15.57:0/2608103120 conn(0x7fc77006d4c0 msgr2=0x7fc77006d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.176+0000 7fc775f15640 1 -- 172.21.15.57:0/2608103120 shutdown_connections 2024-04-21T08:52:28.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.176+0000 7fc775f15640 1 -- 172.21.15.57:0/2608103120 wait complete. 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.177+0000 7fc775f15640 1 Processor -- start 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.177+0000 7fc775f15640 1 -- start start 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc775f15640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc770108f50 0x7fc77011ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc775f15640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc770109910 0x7fc770118c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc775f15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc770119160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc775f15640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc770117550 con 0x7fc770108f50 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc775f15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701173d0 con 0x7fc77010e980 2024-04-21T08:52:28.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc775f15640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701176d0 con 0x7fc770109910 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc770119160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc76ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc770119160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40298/0 (socket says 172.21.15.57:40298) 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.178+0000 7fc76ffff640 1 -- 172.21.15.57:0/34862591 learned_addr learned my addr 172.21.15.57:0/34862591 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76ffff640 1 -- 172.21.15.57:0/34862591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc770109910 msgr2=0x7fc770118c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76ffff640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc770109910 0x7fc770118c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76ffff640 1 -- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc770108f50 msgr2=0x7fc77011ebb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76ffff640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc770108f50 0x7fc77011ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76ffff640 1 -- 172.21.15.57:0/34862591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc770119890 con 0x7fc77010e980 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76ffff640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc770119160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc76000e970 tx=0x7fc76000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76cff9640 1 -- 172.21.15.57:0/34862591 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76000ccc0 con 0x7fc77010e980 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc76cff9640 1 -- 172.21.15.57:0/34862591 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7600054d0 con 0x7fc77010e980 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7701bc4a0 con 0x7fc77010e980 2024-04-21T08:52:28.181 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.179+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7701bc960 con 0x7fc77010e980 2024-04-21T08:52:28.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.182+0000 7fc76cff9640 1 -- 172.21.15.57:0/34862591 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760010430 con 0x7fc77010e980 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 Processor -- start 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 -- start start 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae04d0 0x55ae5bae08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae0e80 0x55ae5bae9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5baebe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ae5ba451f0 con 0x55ae5bae9a00 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae5b9e24b0 con 0x55ae5bae04d0 2024-04-21T08:52:28.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.183+0000 7fc778be8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ae5ba411b0 con 0x55ae5bae0e80 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.185+0000 7fc76effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5baebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.185+0000 7fc76effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5baebe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46150/0 (socket says 172.21.15.57:46150) 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.185+0000 7fc76effd640 1 -- 172.21.15.57:0/1153195287 learned_addr learned my addr 172.21.15.57:0/1153195287 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.185+0000 7fc76effd640 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae04d0 0x55ae5bae08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.185+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae0e80 0x55ae5bae9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76cff9640 1 -- 172.21.15.57:0/34862591 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc760010690 con 0x7fc77010e980 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76f7fe640 1 -- 172.21.15.57:0/1153195287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae04d0 msgr2=0x55ae5bae08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae04d0 0x55ae5bae08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76f7fe640 1 -- 172.21.15.57:0/1153195287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 msgr2=0x55ae5baebe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5baebe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76f7fe640 1 -- 172.21.15.57:0/1153195287 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5b9d41b0 con 0x55ae5bae0e80 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae0e80 0x55ae5bae9430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc768009b30 tx=0x7fc768009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ec052e30e2b252 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.186+0000 7fc755ffb640 1 -- 172.21.15.57:0/1153195287 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768042020 con 0x55ae5bae0e80 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.187+0000 7fc755ffb640 1 -- 172.21.15.57:0/1153195287 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76802c920 con 0x55ae5bae0e80 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.188+0000 7fc778be8740 1 -- 172.21.15.57:0/1153195287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae0e80 msgr2=0x55ae5bae9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.188+0000 7fc778be8740 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae0e80 0x55ae5bae9430 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc768009b30 tx=0x7fc768009920 comp rx=0 tx=0).stop 2024-04-21T08:52:28.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.188+0000 7fc76cff9640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc7580a3c40 0x7fc7580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.189+0000 7fc778be8740 1 -- 172.21.15.57:0/1153195287 shutdown_connections 2024-04-21T08:52:28.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.189+0000 7fc778be8740 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5baebe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.189+0000 7fc76cff9640 1 -- 172.21.15.57:0/34862591 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7fc7600d10c0 con 0x7fc77010e980 2024-04-21T08:52:28.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.189+0000 7fc778be8740 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae0e80 0x55ae5bae9430 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.189+0000 7fc778be8740 1 --2- 172.21.15.57:0/1153195287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae04d0 0x55ae5bae08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.189+0000 7fc778be8740 1 -- 172.21.15.57:0/1153195287 >> 172.21.15.57:0/1153195287 conn(0x55ae5badaba0 msgr2=0x55ae5badaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:28.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.192+0000 7fc76effd640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc7580a3c40 0x7fc7580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.194+0000 7fc778be8740 1 -- 172.21.15.57:0/1153195287 shutdown_connections 2024-04-21T08:52:28.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.194+0000 7fc76effd640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc7580a3c40 0x7fc7580a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc764009fd0 tx=0x7fc764009e00 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:28.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.195+0000 7fc778be8740 1 -- 172.21.15.57:0/1153195287 wait complete. 2024-04-21T08:52:28.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.195+0000 7fc778be8740 1 -- start start 2024-04-21T08:52:28.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.195+0000 7fc778be8740 1 Processor -- start 2024-04-21T08:52:28.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5bae9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 0x55ae5bae64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae9a00 0x55ae5bae6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ae5baec9b0 con 0x55ae5bae0e80 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae5b9fe500 con 0x55ae5bae9a00 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ae5baecb30 con 0x55ae5bae04d0 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc778be8740 10 client.0 _ll_register_callbacks cb 0x55ae5b9ccb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc76effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 0x55ae5bae64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc76effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 0x55ae5bae64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46160/0 (socket says 172.21.15.57:46160) 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc76effd640 1 -- 172.21.15.57:0/259014115 learned_addr learned my addr 172.21.15.57:0/259014115 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:28.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.196+0000 7fc76ffff640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae9a00 0x55ae5bae6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.197+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5bae9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.197+0000 7fc76effd640 1 -- 172.21.15.57:0/259014115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 msgr2=0x55ae5bae9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.197+0000 7fc76effd640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5bae9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.197+0000 7fc76effd640 1 -- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae9a00 msgr2=0x55ae5bae6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.197+0000 7fc76effd640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae9a00 0x55ae5bae6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.197+0000 7fc76effd640 1 -- 172.21.15.57:0/259014115 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae5b7573f0 con 0x55ae5bae0e80 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.198+0000 7fc76effd640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 0x55ae5bae64f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc764001c90 tx=0x7fc764000fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df57d82e83209e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.198+0000 7fc74effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.199+0000 7fc74effd640 1 -- 172.21.15.57:0/259014115 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764012040 con 0x55ae5bae0e80 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.199+0000 7fc74effd640 1 -- 172.21.15.57:0/259014115 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7fc7640190d0 con 0x55ae5bae0e80 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.199+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc76802c920 con 0x55ae5bae0e80 2024-04-21T08:52:28.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.199+0000 7fc74effd640 1 -- 172.21.15.57:0/259014115 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc764017440 con 0x55ae5bae0e80 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.199+0000 7fc778be8740 10 client.16776 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.199+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ae5b8217d0 con 0x55ae5bae0e80 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.200+0000 7fc74effd640 1 -- 172.21.15.57:0/259014115 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76802c920 con 0x55ae5bae0e80 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.200+0000 7fc74effd640 1 -- 172.21.15.57:0/259014115 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc76402d470 con 0x55ae5bae0e80 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.200+0000 7fc778be8740 10 client.16776 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.200+0000 7fc778be8740 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.200+0000 7fc778be8740 20 client.16776 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.200+0000 7fc778be8740 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5baf9b20 0x55ae5bafbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.201+0000 7fc778be8740 4 client.16776 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.201+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae5b9e4930 con 0x55ae5baf9b20 2024-04-21T08:52:28.204 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.202+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5baf9b20 0x55ae5bafbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.204+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5baf9b20 0x55ae5bafbf10 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41703ddf5a1fe05 server_cookie=f0b24dfe51779019 in_seq=0 out_seq=0 2024-04-21T08:52:28.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.204+0000 7fc74effd640 10 client.16776 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:28.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.206+0000 7fc74effd640 1 -- 172.21.15.57:0/259014115 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae5b9e4930 con 0x55ae5baf9b20 2024-04-21T08:52:28.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.206+0000 7fc74effd640 10 client.16776 handle_command_reply: tid=0 2024-04-21T08:52:28.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.206+0000 7fc778be8740 1 client.16776 shutdown 2024-04-21T08:52:28.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.207+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 msgr2=0x55ae5bae64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.207+0000 7fc778be8740 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 0x55ae5bae64f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc764001c90 tx=0x7fc764000fd0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.208+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 shutdown_connections 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.208+0000 7fc778be8740 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5baf9b20 0x55ae5bafbf10 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.208+0000 7fc778be8740 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae9a00 0x55ae5bae6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.208+0000 7fc778be8740 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae0e80 0x55ae5bae64f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.208+0000 7fc778be8740 1 --2- 172.21.15.57:0/259014115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5bae9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.208+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 >> 172.21.15.57:0/259014115 conn(0x55ae5badaba0 msgr2=0x55ae5badc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.209+0000 7fc74effd640 0 client.16776 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.209+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 shutdown_connections 2024-04-21T08:52:28.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.209+0000 7fc778be8740 1 -- 172.21.15.57:0/259014115 wait complete. 2024-04-21T08:52:28.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.210+0000 7fc778be8740 20 client.16776 trim_cache size 0 max 16384 2024-04-21T08:52:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.240+0000 7fc778be8740 1 Processor -- start 2024-04-21T08:52:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.240+0000 7fc778be8740 1 -- start start 2024-04-21T08:52:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.240+0000 7fc778be8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5baf5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc778be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5bae1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc76f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5baf5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc76f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5baf5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45070/0 (socket says 172.21.15.57:45070) 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc76effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5bae1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc76effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5bae1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40328/0 (socket says 172.21.15.57:40328) 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc778be8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5badc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc76f7fe640 1 -- 172.21.15.57:0/2075630272 learned_addr learned my addr 172.21.15.57:0/2075630272 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701c49b0 con 0x55ae5bae9a00 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701c7210 con 0x55ae5bae0e80 2024-04-21T08:52:28.243 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.241+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701c5000 con 0x55ae5bae04d0 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.242+0000 7fc76ffff640 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5badc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.242+0000 7fc76f7fe640 1 -- 172.21.15.57:0/2075630272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 msgr2=0x55ae5bae1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.242+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5bae1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.242+0000 7fc76f7fe640 1 -- 172.21.15.57:0/2075630272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 msgr2=0x55ae5badc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.242+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5badc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.242+0000 7fc76f7fe640 1 -- 172.21.15.57:0/2075630272 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5bae0c20 con 0x55ae5bae04d0 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.243+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5baf5900 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc768009be0 tx=0x7fc768000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15c7f15a9c4d6ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.243+0000 7fc7557fa640 1 -- 172.21.15.57:0/2075630272 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768042040 con 0x55ae5bae04d0 2024-04-21T08:52:28.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.243+0000 7fc7557fa640 1 -- 172.21.15.57:0/2075630272 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc768003e20 con 0x55ae5bae04d0 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 msgr2=0x55ae5baf5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5baf5900 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc768009be0 tx=0x7fc768000f40 comp rx=0 tx=0).stop 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 shutdown_connections 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae9a00 0x55ae5badc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5bae1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 --2- 172.21.15.57:0/2075630272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae04d0 0x55ae5baf5900 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.244+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 >> 172.21.15.57:0/2075630272 conn(0x55ae5badaba0 msgr2=0x55ae5badbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:28.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.245+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 shutdown_connections 2024-04-21T08:52:28.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.245+0000 7fc778be8740 1 -- 172.21.15.57:0/2075630272 wait complete. 2024-04-21T08:52:28.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.245+0000 7fc778be8740 1 -- start start 2024-04-21T08:52:28.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.245+0000 7fc778be8740 1 Processor -- start 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5baf1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 0x55ae5baf2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae2860 0x55ae5bae2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701c56d0 con 0x55ae5bae2860 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701c6b40 con 0x55ae5bae0e80 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc7701c5da0 con 0x55ae5bae9a00 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc778be8740 10 client.0 _ll_register_callbacks cb 0x7fc764007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc76effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 0x55ae5baf2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc76effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 0x55ae5baf2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45078/0 (socket says 172.21.15.57:45078) 2024-04-21T08:52:28.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.246+0000 7fc76effd640 1 -- 172.21.15.57:0/355426781 learned_addr learned my addr 172.21.15.57:0/355426781 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5baf1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76ffff640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae2860 0x55ae5bae2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76effd640 1 -- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 msgr2=0x55ae5baf1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76effd640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5baf1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76effd640 1 -- 172.21.15.57:0/355426781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae2860 msgr2=0x55ae5bae2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76effd640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae2860 0x55ae5bae2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.247+0000 7fc76effd640 1 -- 172.21.15.57:0/355426781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae5bae33d0 con 0x55ae5bae9a00 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.248+0000 7fc76effd640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 0x55ae5baf2350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc76402d8c0 tx=0x7fc764019300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8098bd47d5049c9e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.248+0000 7fc74effd640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.248+0000 7fc74effd640 1 -- 172.21.15.57:0/355426781 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76402dcc0 con 0x55ae5bae9a00 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.248+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ae5ba52c10 con 0x55ae5bae9a00 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.249+0000 7fc74effd640 1 -- 172.21.15.57:0/355426781 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7fc764012040 con 0x55ae5bae9a00 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.249+0000 7fc74effd640 1 -- 172.21.15.57:0/355426781 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc76403e050 con 0x55ae5bae9a00 2024-04-21T08:52:28.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.249+0000 7fc778be8740 10 client.26137 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.249+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ae5bb5ff40 con 0x55ae5bae9a00 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.249+0000 7fc74effd640 1 -- 172.21.15.57:0/355426781 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ae5ba52c10 con 0x55ae5bae9a00 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.249+0000 7fc74effd640 1 -- 172.21.15.57:0/355426781 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc764017b40 con 0x55ae5bae9a00 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc778be8740 10 client.26137 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc778be8740 10 client.26137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc778be8740 20 client.26137 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc778be8740 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5bb60600 0x55ae5bb62a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc778be8740 4 client.26137 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae5ba4ca80 con 0x55ae5bb60600 2024-04-21T08:52:28.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.250+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5bb60600 0x55ae5bb62a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.251+0000 7fc76f7fe640 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5bb60600 0x55ae5bb62a20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=526f06bea48c882 server_cookie=4e49efd8f71cb6a1 in_seq=0 out_seq=0 2024-04-21T08:52:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.251+0000 7fc74effd640 10 client.26137 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:28.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.252+0000 7fc74effd640 1 -- 172.21.15.57:0/355426781 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae5ba4ca80 con 0x55ae5bb60600 2024-04-21T08:52:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.252+0000 7fc74effd640 10 client.26137 handle_command_reply: tid=0 2024-04-21T08:52:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.252+0000 7fc778be8740 1 client.26137 shutdown 2024-04-21T08:52:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.254+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 msgr2=0x55ae5baf2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.254+0000 7fc778be8740 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 0x55ae5baf2350 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc76402d8c0 tx=0x7fc764019300 comp rx=0 tx=0).stop 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 shutdown_connections 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc778be8740 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae5bb60600 0x55ae5bb62a20 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc778be8740 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae5bae2860 0x55ae5bae2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc778be8740 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae5bae9a00 0x55ae5baf2350 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc778be8740 1 --2- 172.21.15.57:0/355426781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae5bae0e80 0x55ae5baf1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 >> 172.21.15.57:0/355426781 conn(0x55ae5badaba0 msgr2=0x55ae5bad9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:28.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.255+0000 7fc74effd640 0 client.26137 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.256+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 shutdown_connections 2024-04-21T08:52:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.256+0000 7fc778be8740 1 -- 172.21.15.57:0/355426781 wait complete. 2024-04-21T08:52:28.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.256+0000 7fc778be8740 20 client.26137 trim_cache size 0 max 16384 2024-04-21T08:52:28.257 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:52:28.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.257+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc7580a3c40 msgr2=0x7fc7580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.257+0000 7fc775f15640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc7580a3c40 0x7fc7580a6100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc764009fd0 tx=0x7fc764009e00 comp rx=0 tx=0).stop 2024-04-21T08:52:28.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.257+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 msgr2=0x7fc770119160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:28.258 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.257+0000 7fc775f15640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc770119160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc76000e970 tx=0x7fc76000ee30 comp rx=0 tx=0).stop 2024-04-21T08:52:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.258+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 shutdown_connections 2024-04-21T08:52:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.258+0000 7fc775f15640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc7580a3c40 0x7fc7580a6100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.258+0000 7fc775f15640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc77010e980 0x7fc770119160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.258+0000 7fc775f15640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc770109910 0x7fc770118c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.258+0000 7fc775f15640 1 --2- 172.21.15.57:0/34862591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc770108f50 0x7fc77011ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:28.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.258+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 >> 172.21.15.57:0/34862591 conn(0x7fc77006d4c0 msgr2=0x7fc770072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:28.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.260+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 shutdown_connections 2024-04-21T08:52:28.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:28.260+0000 7fc775f15640 1 -- 172.21.15.57:0/34862591 wait complete. 2024-04-21T08:52:28.283 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:28 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:52:28.283 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:28 smithi057 ceph-mon[27968]: pgmap v215: 129 pgs: 9 active+clean+snaptrim, 103 active+clean, 17 active+clean+snaptrim_wait; 1.2 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 1.29k op/s 2024-04-21T08:52:28.283 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:28 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3012383149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:28 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:52:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:28 smithi129 ceph-mon[31272]: pgmap v215: 129 pgs: 9 active+clean+snaptrim, 103 active+clean, 17 active+clean+snaptrim_wait; 1.2 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 1.29k op/s 2024-04-21T08:52:28.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:28 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3012383149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:28 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:52:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:28 smithi136 ceph-mon[31791]: pgmap v215: 129 pgs: 9 active+clean+snaptrim, 103 active+clean, 17 active+clean+snaptrim_wait; 1.2 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.4 MiB/s wr, 1.29k op/s 2024-04-21T08:52:28.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:28 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3012383149' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:52:29.285 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:52:29.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:52:29.286 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:52:29.461 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.461+0000 7f78bff52640 1 -- 172.21.15.57:0/1105744734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b8109000 msgr2=0x7f78b81093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.462 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.461+0000 7f78bff52640 1 --2- 172.21.15.57:0/1105744734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b8109000 0x7f78b81093e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009900 tx=0x7f78a802bc80 comp rx=0 tx=0).stop 2024-04-21T08:52:29.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.462+0000 7f78bff52640 1 -- 172.21.15.57:0/1105744734 shutdown_connections 2024-04-21T08:52:29.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.462+0000 7f78bff52640 1 --2- 172.21.15.57:0/1105744734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b810d920 0x7f78b81154b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.462+0000 7f78bff52640 1 --2- 172.21.15.57:0/1105744734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b81099b0 0x7f78b810d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.462+0000 7f78bff52640 1 --2- 172.21.15.57:0/1105744734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b8109000 0x7f78b81093e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.462+0000 7f78bff52640 1 -- 172.21.15.57:0/1105744734 >> 172.21.15.57:0/1105744734 conn(0x7f78b8077dc0 msgr2=0x7f78b807a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.463+0000 7f78bff52640 1 -- 172.21.15.57:0/1105744734 shutdown_connections 2024-04-21T08:52:29.464 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.463+0000 7f78bff52640 1 -- 172.21.15.57:0/1105744734 wait complete. 2024-04-21T08:52:29.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.464+0000 7f78bff52640 1 Processor -- start 2024-04-21T08:52:29.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.464+0000 7f78bff52640 1 -- start start 2024-04-21T08:52:29.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.464+0000 7f78bff52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b8109000 0x7f78b80843d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.466 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bff52640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b810d920 0x7f78b8084910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bff52640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b807d450 0x7f78b807d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bff52640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8118990 con 0x7f78b8109000 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bff52640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8118810 con 0x7f78b810d920 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bff52640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f78b8118b10 con 0x7f78b807d450 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bdcee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b8109000 0x7f78b80843d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bdcee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b8109000 0x7f78b80843d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46180/0 (socket says 172.21.15.57:46180) 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bdcee640 1 -- 172.21.15.57:0/3147994946 learned_addr learned my addr 172.21.15.57:0/3147994946 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78be4ef640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b807d450 0x7f78b807d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.465+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b810d920 0x7f78b8084910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.466+0000 7f78bd4ed640 1 -- 172.21.15.57:0/3147994946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b807d450 msgr2=0x7f78b807d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.466+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b807d450 0x7f78b807d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.466+0000 7f78bd4ed640 1 -- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b8109000 msgr2=0x7f78b80843d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.466+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b8109000 0x7f78b80843d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.466+0000 7f78bd4ed640 1 -- 172.21.15.57:0/3147994946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b807e1c0 con 0x7f78b810d920 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.467+0000 7f78be4ef640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b807d450 0x7f78b807d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.467+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b810d920 0x7f78b8084910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f78b806b4a0 tx=0x7f78b000be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.468+0000 7f78aeffd640 1 -- 172.21.15.57:0/3147994946 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0019070 con 0x7f78b810d920 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.468+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b8081ef0 con 0x7f78b810d920 2024-04-21T08:52:29.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.468+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b8082440 con 0x7f78b810d920 2024-04-21T08:52:29.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.468+0000 7f78aeffd640 1 -- 172.21.15.57:0/3147994946 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b00148f0 con 0x7f78b810d920 2024-04-21T08:52:29.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.468+0000 7f78aeffd640 1 -- 172.21.15.57:0/3147994946 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b0012830 con 0x7f78b810d920 2024-04-21T08:52:29.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.473+0000 7f78c2c28740 1 Processor -- start 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 -- start start 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f84cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f8d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f8de10 0x559021f90260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x559021ee8f00 con 0x559021f8de10 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559021e86590 con 0x559021f848e0 2024-04-21T08:52:29.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78c2c28740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x559021ee4ec0 con 0x559021f85290 2024-04-21T08:52:29.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78be4ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f84cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78be4ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f84cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40360/0 (socket says 172.21.15.57:40360) 2024-04-21T08:52:29.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.475+0000 7f78be4ef640 1 -- 172.21.15.57:0/2420619877 learned_addr learned my addr 172.21.15.57:0/2420619877 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78bdcee640 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f8d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78be4ef640 1 -- 172.21.15.57:0/2420619877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 msgr2=0x559021f8d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78be4ef640 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f8d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78be4ef640 1 -- 172.21.15.57:0/2420619877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f8de10 msgr2=0x559021f90260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78be4ef640 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f8de10 0x559021f90260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78be4ef640 1 -- 172.21.15.57:0/2420619877 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559021cc37e0 con 0x559021f848e0 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78aeffd640 1 -- 172.21.15.57:0/3147994946 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f78b001c070 con 0x7f78b810d920 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.476+0000 7f78be4ef640 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f84cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d70 tx=0x7f78b400e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c00f0771353076c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.477+0000 7f78a3fff640 1 -- 172.21.15.57:0/2420619877 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4017070 con 0x559021f848e0 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.477+0000 7f78a3fff640 1 -- 172.21.15.57:0/2420619877 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b4013920 con 0x559021f848e0 2024-04-21T08:52:29.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.477+0000 7f78a3fff640 1 -- 172.21.15.57:0/2420619877 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b4012790 con 0x559021f848e0 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.477+0000 7f78aeffd640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f78a40a3ef0 0x7f78a40a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.477+0000 7f78aeffd640 1 -- 172.21.15.57:0/3147994946 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f78b00d02b0 con 0x7f78b810d920 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 -- 172.21.15.57:0/2420619877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 msgr2=0x559021f84cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f84cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d70 tx=0x7f78b400e5a0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 -- 172.21.15.57:0/2420619877 shutdown_connections 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f8de10 0x559021f90260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f8d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2420619877 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f84cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78c2c28740 1 -- 172.21.15.57:0/2420619877 >> 172.21.15.57:0/2420619877 conn(0x559021f7efb0 msgr2=0x559021f7f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.478+0000 7f78bdcee640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f78a40a3ef0 0x7f78a40a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.479+0000 7f78c2c28740 1 -- 172.21.15.57:0/2420619877 shutdown_connections 2024-04-21T08:52:29.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.479+0000 7f78c2c28740 1 -- 172.21.15.57:0/2420619877 wait complete. 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.479+0000 7f78c2c28740 1 -- start start 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.479+0000 7f78c2c28740 1 Processor -- start 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f8ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 0x559021f8b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x559021f90d80 con 0x559021f85290 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559021ea2580 con 0x559021f848e0 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x559021f90f00 con 0x559021f8de10 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78c2c28740 10 client.0 _ll_register_callbacks cb 0x559021ea8650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78bd4ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78be4ef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 0x559021f8b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78be4ef640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 0x559021f8b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45116/0 (socket says 172.21.15.57:45116) 2024-04-21T08:52:29.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.480+0000 7f78be4ef640 1 -- 172.21.15.57:0/3304235965 learned_addr learned my addr 172.21.15.57:0/3304235965 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.482+0000 7f78be4ef640 1 -- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 msgr2=0x559021f8ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.484 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.481+0000 7f78bdcee640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f8ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.483+0000 7f78be4ef640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f8ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.483+0000 7f78be4ef640 1 -- 172.21.15.57:0/3304235965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 msgr2=0x559021f8b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.483+0000 7f78be4ef640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.484+0000 7f78be4ef640 1 -- 172.21.15.57:0/3304235965 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559021d23f70 con 0x559021f8de10 2024-04-21T08:52:29.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.484+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.484+0000 7f78bdcee640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f8ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.484+0000 7f78bdcee640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f78a40a3ef0 0x7f78a40a63b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x559021f80020 tx=0x7f78a8000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.484+0000 7f78be4ef640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 0x559021f8b8b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78b4013d80 tx=0x7f78b4013db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b673c23d79de669e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.484+0000 7f78a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.485+0000 7f78a17fa640 1 -- 172.21.15.57:0/3304235965 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b404b060 con 0x559021f8de10 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.485+0000 7f78a17fa640 1 -- 172.21.15.57:0/3304235965 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f78b40126d0 con 0x559021f8de10 2024-04-21T08:52:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.485+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78b4013920 con 0x559021f8de10 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.485+0000 7f78a17fa640 1 -- 172.21.15.57:0/3304235965 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78b4052050 con 0x559021f8de10 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.485+0000 7f78c2c28740 10 client.26146 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.485+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559021bbb170 con 0x559021f8de10 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78a17fa640 1 -- 172.21.15.57:0/3304235965 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78b4013920 con 0x559021f8de10 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78a17fa640 1 -- 172.21.15.57:0/3304235965 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f78b401c970 con 0x559021f8de10 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78c2c28740 10 client.26146 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:29.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78c2c28740 10 client.26146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:29.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78c2c28740 20 client.26146 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:29.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78c2c28740 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559021f9e780 0x559021fa0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78c2c28740 4 client.26146 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:29.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559021ef6920 con 0x559021f9e780 2024-04-21T08:52:29.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.486+0000 7f78bdcee640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559021f9e780 0x559021fa0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.490+0000 7f78bdcee640 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559021f9e780 0x559021fa0ba0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd1ab29f5323165d server_cookie=da883fede41cb572 in_seq=0 out_seq=0 2024-04-21T08:52:29.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.490+0000 7f78a17fa640 10 client.26146 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.493+0000 7f78a17fa640 1 -- 172.21.15.57:0/3304235965 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559021ef6920 con 0x559021f9e780 2024-04-21T08:52:29.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.493+0000 7f78a17fa640 10 client.26146 handle_command_reply: tid=0 2024-04-21T08:52:29.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 client.26146 shutdown 2024-04-21T08:52:29.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 msgr2=0x559021f8b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 0x559021f8b8b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f78b4013d80 tx=0x7f78b4013db0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 shutdown_connections 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559021f9e780 0x559021fa0ba0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f8de10 0x559021f8b8b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 --2- 172.21.15.57:0/3304235965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f848e0 0x559021f8ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.494+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 >> 172.21.15.57:0/3304235965 conn(0x559021f7efb0 msgr2=0x559021f80d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.495+0000 7f78a17fa640 0 client.26146 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.495+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 shutdown_connections 2024-04-21T08:52:29.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.495+0000 7f78c2c28740 1 -- 172.21.15.57:0/3304235965 wait complete. 2024-04-21T08:52:29.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.496+0000 7f78c2c28740 20 client.26146 trim_cache size 0 max 16384 2024-04-21T08:52:29.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.533+0000 7f78c2c28740 1 Processor -- start 2024-04-21T08:52:29.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 -- start start 2024-04-21T08:52:29.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 0x559021f99cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021f7de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f789c00e130 con 0x559021f85290 2024-04-21T08:52:29.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f789c010990 con 0x559021f8de10 2024-04-21T08:52:29.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.536+0000 7f78c2c28740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f789c00e780 con 0x559021f848e0 2024-04-21T08:52:29.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.540+0000 7f78be4ef640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021f7de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.540+0000 7f78bdcee640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 0x559021f99cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.540+0000 7f78bdcee640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 0x559021f99cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45120/0 (socket says 172.21.15.57:45120) 2024-04-21T08:52:29.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.541+0000 7f78bdcee640 1 -- 172.21.15.57:0/781183743 learned_addr learned my addr 172.21.15.57:0/781183743 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.541+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.542+0000 7f78bdcee640 1 -- 172.21.15.57:0/781183743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 msgr2=0x559021f7de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.542+0000 7f78bdcee640 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021f7de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.542+0000 7f78bdcee640 1 -- 172.21.15.57:0/781183743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 msgr2=0x559021f8b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.542+0000 7f78bdcee640 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.542+0000 7f78bdcee640 1 -- 172.21.15.57:0/781183743 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559021f89de0 con 0x559021f848e0 2024-04-21T08:52:29.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.543+0000 7f78bdcee640 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 0x559021f99cd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78a802c1c0 tx=0x7f78a8041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec4577d258691ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.543+0000 7f78a37fe640 1 -- 172.21.15.57:0/781183743 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a80044b0 con 0x559021f848e0 2024-04-21T08:52:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.544+0000 7f78a37fe640 1 -- 172.21.15.57:0/781183743 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a805a070 con 0x559021f848e0 2024-04-21T08:52:29.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.544+0000 7f78a37fe640 1 -- 172.21.15.57:0/781183743 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8005040 con 0x559021f848e0 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 -- 172.21.15.57:0/781183743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 msgr2=0x559021f99cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 0x559021f99cd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78a802c1c0 tx=0x7f78a8041000 comp rx=0 tx=0).stop 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 -- 172.21.15.57:0/781183743 shutdown_connections 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021f7de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f85290 0x559021f8b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 --2- 172.21.15.57:0/781183743 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f848e0 0x559021f99cd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 -- 172.21.15.57:0/781183743 >> 172.21.15.57:0/781183743 conn(0x559021f7efb0 msgr2=0x559021f7d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 -- 172.21.15.57:0/781183743 shutdown_connections 2024-04-21T08:52:29.546 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.545+0000 7f78c2c28740 1 -- 172.21.15.57:0/781183743 wait complete. 2024-04-21T08:52:29.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.546+0000 7f78c2c28740 1 -- start start 2024-04-21T08:52:29.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.546+0000 7f78c2c28740 1 Processor -- start 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f848e0 0x559021f961e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f96720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021fffe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f789c00ee50 con 0x559021f848e0 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0102c0 con 0x559021f8de10 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f789c00f520 con 0x559021f85290 2024-04-21T08:52:29.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.549+0000 7f78c2c28740 10 client.0 _ll_register_callbacks cb 0x7f78a8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78bdcee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f848e0 0x559021f961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78bdcee640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f848e0 0x559021f961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46200/0 (socket says 172.21.15.57:46200) 2024-04-21T08:52:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78bdcee640 1 -- 172.21.15.57:0/2720827021 learned_addr learned my addr 172.21.15.57:0/2720827021 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f96720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78be4ef640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021fffe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78bd4ed640 1 -- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 msgr2=0x559021fffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.550+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021fffe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.551+0000 7f78bd4ed640 1 -- 172.21.15.57:0/2720827021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f848e0 msgr2=0x559021f961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.551+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f848e0 0x559021f961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.551+0000 7f78bd4ed640 1 -- 172.21.15.57:0/2720827021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559022000670 con 0x559021f85290 2024-04-21T08:52:29.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.552+0000 7f78bd4ed640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f96720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78b00cd6a0 tx=0x7f78b009b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a8b94597944016f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.555+0000 7f78a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:52:29.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.558+0000 7f78a17fa640 1 -- 172.21.15.57:0/2720827021 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b00ba060 con 0x559021f85290 2024-04-21T08:52:29.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.558+0000 7f78a17fa640 1 -- 172.21.15.57:0/2720827021 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f78b00a46d0 con 0x559021f85290 2024-04-21T08:52:29.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.558+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a805a070 con 0x559021f85290 2024-04-21T08:52:29.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.559+0000 7f78a17fa640 1 -- 172.21.15.57:0/2720827021 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f78b00a9050 con 0x559021f85290 2024-04-21T08:52:29.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.560+0000 7f78c2c28740 10 client.26152 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:29.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.560+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559021f881d0 con 0x559021f85290 2024-04-21T08:52:29.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78a17fa640 1 -- 172.21.15.57:0/2720827021 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a805a070 con 0x559021f85290 2024-04-21T08:52:29.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78a17fa640 1 -- 172.21.15.57:0/2720827021 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f78b00cdce0 con 0x559021f85290 2024-04-21T08:52:29.561 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78c2c28740 10 client.26152 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:29.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78c2c28740 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:29.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78c2c28740 20 client.26152 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:29.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559022016250 0x559022018670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78c2c28740 4 client.26152 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:52:29.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.561+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559021ef0790 con 0x559022016250 2024-04-21T08:52:29.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.562+0000 7f78bdcee640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559022016250 0x559022018670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.563+0000 7f78bdcee640 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559022016250 0x559022018670 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42c0f38e08695c1e server_cookie=844a3ed33b9e753b in_seq=0 out_seq=0 2024-04-21T08:52:29.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.563+0000 7f78a17fa640 10 client.26152 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.565+0000 7f78a17fa640 1 -- 172.21.15.57:0/2720827021 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559021ef0790 con 0x559022016250 2024-04-21T08:52:29.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.565+0000 7f78a17fa640 10 client.26152 handle_command_reply: tid=0 2024-04-21T08:52:29.566 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.565+0000 7f78c2c28740 1 client.26152 shutdown 2024-04-21T08:52:29.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.566+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 msgr2=0x559021f96720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.567 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.566+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f96720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78b00cd6a0 tx=0x7f78b009b9c0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.569+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 shutdown_connections 2024-04-21T08:52:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.569+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559022016250 0x559022018670 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.569+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559021f8de10 0x559021fffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.569+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559021f85290 0x559021f96720 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.569+0000 7f78c2c28740 1 --2- 172.21.15.57:0/2720827021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559021f848e0 0x559021f961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.569 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.569+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 >> 172.21.15.57:0/2720827021 conn(0x559021f7efb0 msgr2=0x559021f7e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.571+0000 7f78a17fa640 0 client.26152 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.576+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 shutdown_connections 2024-04-21T08:52:29.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.578+0000 7f78c2c28740 1 -- 172.21.15.57:0/2720827021 wait complete. 2024-04-21T08:52:29.580 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.579+0000 7f78c2c28740 20 client.26152 trim_cache size 0 max 16384 2024-04-21T08:52:29.582 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:52:29.582 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:52:29.582 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "0b874346-6e34-4cca-8a18-946870a8591d", 2024-04-21T08:52:29.582 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:52:29.582 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f78a40a3ef0 msgr2=0x7f78a40a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f78a40a3ef0 0x7f78a40a63b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x559021f80020 tx=0x7f78a8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b810d920 msgr2=0x7f78b8084910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b810d920 0x7f78b8084910 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f78b806b4a0 tx=0x7f78b000be00 comp rx=0 tx=0).stop 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 shutdown_connections 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f78a40a3ef0 0x7f78a40a63b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78b807d450 0x7f78b807d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78b810d920 0x7f78b8084910 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 --2- 172.21.15.57:0/3147994946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f78b8109000 0x7f78b80843d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 >> 172.21.15.57:0/3147994946 conn(0x7f78b8077dc0 msgr2=0x7f78b810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.582+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 shutdown_connections 2024-04-21T08:52:29.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.583+0000 7f78bff52640 1 -- 172.21.15.57:0/3147994946 wait complete. 2024-04-21T08:52:29.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:29.602 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:52:29.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.779+0000 7f3269814640 1 -- 172.21.15.57:0/1558492823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 msgr2=0x7f326411db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.779+0000 7f3269814640 1 --2- 172.21.15.57:0/1558492823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 0x7f326411db20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3254009cd0 tx=0x7f325402c0e0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.780+0000 7f3269814640 1 -- 172.21.15.57:0/1558492823 shutdown_connections 2024-04-21T08:52:29.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.780+0000 7f3269814640 1 --2- 172.21.15.57:0/1558492823 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 0x7f326411db20 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.780+0000 7f3269814640 1 --2- 172.21.15.57:0/1558492823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f326410e7b0 0x7f326410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.780+0000 7f3269814640 1 --2- 172.21.15.57:0/1558492823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32641106d0 0x7f326410e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.780+0000 7f3269814640 1 -- 172.21.15.57:0/1558492823 >> 172.21.15.57:0/1558492823 conn(0x7f326406d070 msgr2=0x7f326406d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.781+0000 7f3269814640 1 -- 172.21.15.57:0/1558492823 shutdown_connections 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.781+0000 7f3269814640 1 -- 172.21.15.57:0/1558492823 wait complete. 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.782+0000 7f3269814640 1 Processor -- start 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.782+0000 7f3269814640 1 -- start start 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3269814640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326410e7b0 0x7f32641a7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3269814640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 0x7f32641a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3269814640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 0x7f32641abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3269814640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3264120f70 con 0x7f326411b730 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3269814640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3264120df0 con 0x7f32641106d0 2024-04-21T08:52:29.785 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3269814640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f32641210f0 con 0x7f326410e7b0 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f32627fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 0x7f32641a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f32627fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 0x7f32641a79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40390/0 (socket says 172.21.15.57:40390) 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f32627fc640 1 -- 172.21.15.57:0/1849228217 learned_addr learned my addr 172.21.15.57:0/1849228217 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326410e7b0 0x7f32641a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.783+0000 7f32637fe640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 0x7f32641abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.784+0000 7f32627fc640 1 -- 172.21.15.57:0/1849228217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326410e7b0 msgr2=0x7f32641a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.784+0000 7f32627fc640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326410e7b0 0x7f32641a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.784+0000 7f32627fc640 1 -- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 msgr2=0x7f32641abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.784+0000 7f32627fc640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 0x7f32641abd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.784+0000 7f32627fc640 1 -- 172.21.15.57:0/1849228217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32641ac460 con 0x7f32641106d0 2024-04-21T08:52:29.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.784+0000 7f32627fc640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 0x7f32641a79b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f324c00d950 tx=0x7f324c00dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.786+0000 7f3268812640 1 -- 172.21.15.57:0/1849228217 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c017070 con 0x7f32641106d0 2024-04-21T08:52:29.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.786+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32641ac750 con 0x7f32641106d0 2024-04-21T08:52:29.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.786+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f326407a1b0 con 0x7f32641106d0 2024-04-21T08:52:29.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.786+0000 7f3268812640 1 -- 172.21.15.57:0/1849228217 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f324c00de80 con 0x7f32641106d0 2024-04-21T08:52:29.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.786+0000 7f3268812640 1 -- 172.21.15.57:0/1849228217 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c00f620 con 0x7f32641106d0 2024-04-21T08:52:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.790+0000 7f3268812640 1 -- 172.21.15.57:0/1849228217 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f324c009c30 con 0x7f32641106d0 2024-04-21T08:52:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f3268812640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32500a3f20 0x7f32500a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 Processor -- start 2024-04-21T08:52:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 -- start start 2024-04-21T08:52:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f3268812640 1 -- 172.21.15.57:0/1849228217 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f324c0d05d0 con 0x7f32641106d0 2024-04-21T08:52:29.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48c3580 0x55b7d48c3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48cc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b7d48281e0 con 0x55b7d48c3580 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b7d47c5390 con 0x55b7d48c3ea0 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f326c409740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b7d48241a0 con 0x55b7d48cca80 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f32627fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f32627fc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40406/0 (socket says 172.21.15.57:40406) 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.792+0000 7f32627fc640 1 -- 172.21.15.57:0/1923101503 learned_addr learned my addr 172.21.15.57:0/1923101503 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.793+0000 7f32637fe640 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.793+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32500a3f20 0x7f32500a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.793+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48c3580 0x55b7d48c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.794+0000 7f3262ffd640 1 -- 172.21.15.57:0/1923101503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 msgr2=0x55b7d48cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.794+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.794+0000 7f3262ffd640 1 -- 172.21.15.57:0/1923101503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 msgr2=0x55b7d48cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.794+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.794+0000 7f3262ffd640 1 -- 172.21.15.57:0/1923101503 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7d46047d0 con 0x55b7d48c3580 2024-04-21T08:52:29.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.794+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32500a3f20 0x7f32500a63e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f325c005e10 tx=0x7f325c00ab10 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.795+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48c3580 0x55b7d48c3960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f325c0058c0 tx=0x7f325c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8674b5fc96df25f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.796+0000 7f3259ffb640 1 -- 172.21.15.57:0/1923101503 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c00fe50 con 0x55b7d48c3580 2024-04-21T08:52:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.796+0000 7f3259ffb640 1 -- 172.21.15.57:0/1923101503 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325c023020 con 0x55b7d48c3580 2024-04-21T08:52:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.797+0000 7f326c409740 1 -- 172.21.15.57:0/1923101503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48c3580 msgr2=0x55b7d48c3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.797+0000 7f326c409740 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48c3580 0x55b7d48c3960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f325c0058c0 tx=0x7f325c01d000 comp rx=0 tx=0).stop 2024-04-21T08:52:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.799+0000 7f326c409740 1 -- 172.21.15.57:0/1923101503 shutdown_connections 2024-04-21T08:52:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.799+0000 7f326c409740 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.799+0000 7f326c409740 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.799+0000 7f326c409740 1 --2- 172.21.15.57:0/1923101503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48c3580 0x55b7d48c3960 secure :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f325c0058c0 tx=0x7f325c01d000 comp rx=0 tx=0).stop 2024-04-21T08:52:29.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.799+0000 7f326c409740 1 -- 172.21.15.57:0/1923101503 >> 172.21.15.57:0/1923101503 conn(0x55b7d48bdcb0 msgr2=0x55b7d48be090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.800+0000 7f326c409740 1 -- 172.21.15.57:0/1923101503 shutdown_connections 2024-04-21T08:52:29.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.801+0000 7f326c409740 1 -- 172.21.15.57:0/1923101503 wait complete. 2024-04-21T08:52:29.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.801+0000 7f326c409740 1 -- start start 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.801+0000 7f326c409740 1 Processor -- start 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48d8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b7d48cf860 con 0x55b7d48ca4e0 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b7d47e12f0 con 0x55b7d48c3ea0 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b7d48cf9e0 con 0x55b7d48cca80 2024-04-21T08:52:29.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f326c409740 10 client.0 _ll_register_callbacks cb 0x55b7d47ee580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f32637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48d8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f32637fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48d8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46256/0 (socket says 172.21.15.57:46256) 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.802+0000 7f32637fe640 1 -- 172.21.15.57:0/594545911 learned_addr learned my addr 172.21.15.57:0/594545911 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.803+0000 7f32637fe640 1 -- 172.21.15.57:0/594545911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 msgr2=0x55b7d48c9fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.803+0000 7f32637fe640 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48c9fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.803+0000 7f32637fe640 1 -- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 msgr2=0x55b7d48c9a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.803+0000 7f32637fe640 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c9a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.803+0000 7f32637fe640 1 -- 172.21.15.57:0/594545911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7d453a3f0 con 0x55b7d48ca4e0 2024-04-21T08:52:29.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.804+0000 7f32637fe640 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48d8d50 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3254009cd0 tx=0x7f3254009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5186ac41fb14eb4d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.804+0000 7f3242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:29.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.805+0000 7f3242ffd640 1 -- 172.21.15.57:0/594545911 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254044070 con 0x55b7d48ca4e0 2024-04-21T08:52:29.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.806+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f325c023020 con 0x55b7d48ca4e0 2024-04-21T08:52:29.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.806+0000 7f3242ffd640 1 -- 172.21.15.57:0/594545911 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f3254003970 con 0x55b7d48ca4e0 2024-04-21T08:52:29.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.806+0000 7f3242ffd640 1 -- 172.21.15.57:0/594545911 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3254050050 con 0x55b7d48ca4e0 2024-04-21T08:52:29.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.807+0000 7f326c409740 10 client.16803 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:29.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.807+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b7d44fa0f0 con 0x55b7d48ca4e0 2024-04-21T08:52:29.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.808+0000 7f3242ffd640 1 -- 172.21.15.57:0/594545911 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325c023020 con 0x55b7d48ca4e0 2024-04-21T08:52:29.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.808+0000 7f3242ffd640 1 -- 172.21.15.57:0/594545911 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f325403f440 con 0x55b7d48ca4e0 2024-04-21T08:52:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.811+0000 7f326c409740 10 client.16803 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.811+0000 7f326c409740 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.811+0000 7f326c409740 20 client.16803 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.811+0000 7f326c409740 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48edf10 0x55b7d48f0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.811+0000 7f326c409740 4 client.16803 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:29.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.811+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7d4835c00 con 0x55b7d48edf10 2024-04-21T08:52:29.814 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.813+0000 7f3262ffd640 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48edf10 0x55b7d48f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.815+0000 7f3262ffd640 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48edf10 0x55b7d48f0390 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c0df2147efaea2 server_cookie=847265b9d321a785 in_seq=0 out_seq=0 2024-04-21T08:52:29.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.815+0000 7f3242ffd640 10 client.16803 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.821+0000 7f3242ffd640 1 -- 172.21.15.57:0/594545911 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7d4835c00 con 0x55b7d48edf10 2024-04-21T08:52:29.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.821+0000 7f3242ffd640 10 client.16803 handle_command_reply: tid=0 2024-04-21T08:52:29.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.821+0000 7f326c409740 1 client.16803 shutdown 2024-04-21T08:52:29.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.822+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 msgr2=0x55b7d48d8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.822+0000 7f326c409740 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48d8d50 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3254009cd0 tx=0x7f3254009960 comp rx=0 tx=0).stop 2024-04-21T08:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 shutdown_connections 2024-04-21T08:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48edf10 0x55b7d48f0390 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48d8d50 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48cca80 0x55b7d48c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 --2- 172.21.15.57:0/594545911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 >> 172.21.15.57:0/594545911 conn(0x55b7d48bdcb0 msgr2=0x55b7d48bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f3242ffd640 0 client.16803 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 shutdown_connections 2024-04-21T08:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.823+0000 7f326c409740 1 -- 172.21.15.57:0/594545911 wait complete. 2024-04-21T08:52:29.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.824+0000 7f326c409740 20 client.16803 trim_cache size 0 max 16384 2024-04-21T08:52:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.861+0000 7f326c409740 1 Processor -- start 2024-04-21T08:52:29.862 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.861+0000 7f326c409740 1 -- start start 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.861+0000 7f326c409740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 0x55b7d48eff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f326c409740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f3262ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 0x55b7d48eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f3262ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 0x55b7d48eff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:46262/0 (socket says 172.21.15.57:46262) 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f326c409740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48ca4e0 0x55b7d48f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f326c409740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f324400e190 con 0x55b7d48bdcb0 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f326c409740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32440109f0 con 0x55b7d48c3ea0 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f326c409740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f324400e7e0 con 0x55b7d48ca4e0 2024-04-21T08:52:29.863 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.862+0000 7f3262ffd640 1 -- 172.21.15.57:0/1901144145 learned_addr learned my addr 172.21.15.57:0/1901144145 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.863+0000 7f32637fe640 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48ca4e0 0x55b7d48f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.863+0000 7f32627fc640 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.863+0000 7f3262ffd640 1 -- 172.21.15.57:0/1901144145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48ca4e0 msgr2=0x55b7d48f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.864+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48ca4e0 0x55b7d48f61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.864+0000 7f3262ffd640 1 -- 172.21.15.57:0/1901144145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 msgr2=0x55b7d48f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.864+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.864+0000 7f3262ffd640 1 -- 172.21.15.57:0/1901144145 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7d48c7fb0 con 0x55b7d48bdcb0 2024-04-21T08:52:29.865 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.864+0000 7f3262ffd640 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 0x55b7d48eff20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f325c025040 tx=0x7f325c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3fca5a04fef6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.865+0000 7f32597fa640 1 -- 172.21.15.57:0/1901144145 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c033070 con 0x55b7d48bdcb0 2024-04-21T08:52:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.865+0000 7f32597fa640 1 -- 172.21.15.57:0/1901144145 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325c023040 con 0x55b7d48bdcb0 2024-04-21T08:52:29.866 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.865+0000 7f32597fa640 1 -- 172.21.15.57:0/1901144145 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c01f9f0 con 0x55b7d48bdcb0 2024-04-21T08:52:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.866+0000 7f326c409740 1 -- 172.21.15.57:0/1901144145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 msgr2=0x55b7d48eff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.867 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.866+0000 7f326c409740 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 0x55b7d48eff20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f325c025040 tx=0x7f325c020000 comp rx=0 tx=0).stop 2024-04-21T08:52:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.868+0000 7f326c409740 1 -- 172.21.15.57:0/1901144145 shutdown_connections 2024-04-21T08:52:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.868+0000 7f326c409740 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48ca4e0 0x55b7d48f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.868+0000 7f326c409740 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48c3ea0 0x55b7d48f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.868+0000 7f326c409740 1 --2- 172.21.15.57:0/1901144145 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48bdcb0 0x55b7d48eff20 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.868+0000 7f326c409740 1 -- 172.21.15.57:0/1901144145 >> 172.21.15.57:0/1901144145 conn(0x55b7d48ca100 msgr2=0x55b7d48c8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.869+0000 7f326c409740 1 -- 172.21.15.57:0/1901144145 shutdown_connections 2024-04-21T08:52:29.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.869+0000 7f326c409740 1 -- 172.21.15.57:0/1901144145 wait complete. 2024-04-21T08:52:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.870+0000 7f326c409740 1 -- start start 2024-04-21T08:52:29.871 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.870+0000 7f326c409740 1 Processor -- start 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48bdcb0 0x55b7d48c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f324400eeb0 con 0x55b7d48ca4e0 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3244010320 con 0x55b7d48bdcb0 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f324400f580 con 0x55b7d48c3ea0 2024-04-21T08:52:29.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.871+0000 7f326c409740 10 client.0 _ll_register_callbacks cb 0x7f3254037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.872+0000 7f32627fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.872+0000 7f32627fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:45176/0 (socket says 172.21.15.57:45176) 2024-04-21T08:52:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.872+0000 7f32627fc640 1 -- 172.21.15.57:0/3988665359 learned_addr learned my addr 172.21.15.57:0/3988665359 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:29.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.872+0000 7f32637fe640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.872+0000 7f3262ffd640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48bdcb0 0x55b7d48c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3262ffd640 1 -- 172.21.15.57:0/3988665359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48c3ea0 msgr2=0x55b7d48c6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3262ffd640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3262ffd640 1 -- 172.21.15.57:0/3988665359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 msgr2=0x55b7d48c7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3262ffd640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48c7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3262ffd640 1 -- 172.21.15.57:0/3988665359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7d49534d0 con 0x55b7d48bdcb0 2024-04-21T08:52:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3262ffd640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48bdcb0 0x55b7d48c65e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55b7d48c95f0 tx=0x7f325c01dea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7658fca5bf03bba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:52:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3242ffd640 1 -- 172.21.15.57:0/3988665359 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c03c030 con 0x55b7d48bdcb0 2024-04-21T08:52:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f3242ffd640 1 -- 172.21.15.57:0/3988665359 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f325c011240 con 0x55b7d48bdcb0 2024-04-21T08:52:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.873+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f325c023040 con 0x55b7d48bdcb0 2024-04-21T08:52:29.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.874+0000 7f3242ffd640 1 -- 172.21.15.57:0/3988665359 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f325c046050 con 0x55b7d48bdcb0 2024-04-21T08:52:29.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.875+0000 7f326c409740 10 client.26288 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:29.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.875+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b7d4954720 con 0x55b7d48bdcb0 2024-04-21T08:52:29.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.875+0000 7f3242ffd640 1 -- 172.21.15.57:0/3988665359 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f325c023040 con 0x55b7d48bdcb0 2024-04-21T08:52:29.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.875+0000 7f3242ffd640 1 -- 172.21.15.57:0/3988665359 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f325c01e050 con 0x55b7d48bdcb0 2024-04-21T08:52:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.876+0000 7f326c409740 10 client.26288 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.876+0000 7f326c409740 10 client.26288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.876+0000 7f326c409740 20 client.26288 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.876+0000 7f326c409740 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48cca80 0x55b7d4956c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.876+0000 7f326c409740 4 client.26288 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:52:29.877 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.876+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7d482fa70 con 0x55b7d48cca80 2024-04-21T08:52:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.877+0000 7f32627fc640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48cca80 0x55b7d4956c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.877+0000 7f3242ffd640 10 client.26288 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.878 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.878+0000 7f32627fc640 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48cca80 0x55b7d4956c50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ec90b0ec6cb1ff server_cookie=1f8213ea2bcb2342 in_seq=0 out_seq=0 2024-04-21T08:52:29.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.890+0000 7f3242ffd640 1 -- 172.21.15.57:0/3988665359 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x55b7d482fa70 con 0x55b7d48cca80 2024-04-21T08:52:29.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.890+0000 7f3242ffd640 10 client.26288 handle_command_reply: tid=0 2024-04-21T08:52:29.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.890+0000 7f326c409740 1 client.26288 shutdown 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.891+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48bdcb0 msgr2=0x55b7d48c65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.891+0000 7f326c409740 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48bdcb0 0x55b7d48c65e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55b7d48c95f0 tx=0x7f325c01dea0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 shutdown_connections 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b7d48cca80 0x55b7d4956c50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b7d48ca4e0 0x55b7d48c7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b7d48c3ea0 0x55b7d48c6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 --2- 172.21.15.57:0/3988665359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b7d48bdcb0 0x55b7d48c65e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.892 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 >> 172.21.15.57:0/3988665359 conn(0x55b7d48ca100 msgr2=0x55b7d48bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f3242ffd640 0 client.26288 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:29.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 shutdown_connections 2024-04-21T08:52:29.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.892+0000 7f326c409740 1 -- 172.21.15.57:0/3988665359 wait complete. 2024-04-21T08:52:29.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.893+0000 7f326c409740 20 client.26288 trim_cache size 0 max 16384 2024-04-21T08:52:29.894 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:52:29.894 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (1827 inodes in the stack)", 2024-04-21T08:52:29.894 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:52:29.894 INFO:teuthology.orchestra.run.smithi057.stdout: "0b874346-6e34-4cca-8a18-946870a8591d": { 2024-04-21T08:52:29.895 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:52:29.895 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "0b874346-6e34-4cca-8a18-946870a8591d", 2024-04-21T08:52:29.895 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:52:29.895 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:52:29.895 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:52:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:52:29.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.894+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32500a3f20 msgr2=0x7f32500a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.894+0000 7f3269814640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32500a3f20 0x7f32500a63e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f325c005e10 tx=0x7f325c00ab10 comp rx=0 tx=0).stop 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 msgr2=0x7f32641a79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 0x7f32641a79b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f324c00d950 tx=0x7f324c00dd50 comp rx=0 tx=0).stop 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 shutdown_connections 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f32500a3f20 0x7f32500a63e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f326411b730 0x7f32641abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32641106d0 0x7f32641a79b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 --2- 172.21.15.57:0/1849228217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326410e7b0 0x7f32641a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.895+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 >> 172.21.15.57:0/1849228217 conn(0x7f326406d070 msgr2=0x7f326411a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.896+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 shutdown_connections 2024-04-21T08:52:29.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:52:29.896+0000 7f3269814640 1 -- 172.21.15.57:0/1849228217 wait complete. 2024-04-21T08:52:29.916 INFO:tasks.cephfs.filesystem:scrub status for tag:0b874346-6e34-4cca-8a18-946870a8591d - {'path': '/', 'tag': '0b874346-6e34-4cca-8a18-946870a8591d', 'options': 'recursive,force'} 2024-04-21T08:52:30.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:30 smithi057 ceph-mon[27968]: pgmap v216: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 1.22k op/s 2024-04-21T08:52:30.436 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:30 smithi136 ceph-mon[31791]: pgmap v216: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 1.22k op/s 2024-04-21T08:52:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:30 smithi129 ceph-mon[31272]: pgmap v216: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 1.22k op/s 2024-04-21T08:52:31.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:31 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:52:31.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:31 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:31.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:31 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:52:31.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:31 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:52:31.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:31 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:31.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:31 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:52:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:31 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:52:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:31 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:52:31.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:31 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:52:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:32 smithi129 ceph-mon[31272]: pgmap v217: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 3.2 MiB/s wr, 1.07k op/s 2024-04-21T08:52:32.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:32 smithi057 ceph-mon[27968]: pgmap v217: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 3.2 MiB/s wr, 1.07k op/s 2024-04-21T08:52:32.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:32 smithi136 ceph-mon[31791]: pgmap v217: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.3 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 3.2 MiB/s wr, 1.07k op/s 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 -- 172.21.15.57:0/404431062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec10c560 msgr2=0x7f87ec10e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 --2- 172.21.15.57:0/404431062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec10c560 0x7f87ec10e9b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009ca0 tx=0x7f87e402c180 comp rx=0 tx=0).stop 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 -- 172.21.15.57:0/404431062 shutdown_connections 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 --2- 172.21.15.57:0/404431062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec10c560 0x7f87ec10e9b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 --2- 172.21.15.57:0/404431062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87ec103b40 0x7f87ec10c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 --2- 172.21.15.57:0/404431062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87ec103220 0x7f87ec103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 -- 172.21.15.57:0/404431062 >> 172.21.15.57:0/404431062 conn(0x7f87ec077fd0 msgr2=0x7f87ec0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:34.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 -- 172.21.15.57:0/404431062 shutdown_connections 2024-04-21T08:52:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 -- 172.21.15.57:0/404431062 wait complete. 2024-04-21T08:52:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 Processor -- start 2024-04-21T08:52:34.049 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.048+0000 7f87f0c81640 1 -- start start 2024-04-21T08:52:34.050 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87f0c81640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87ec103220 0x7f87ec100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87f0c81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 0x7f87ec1013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87f0c81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87ec10c560 0x7f87ec1018f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87f0c81640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec112180 con 0x7f87ec103b40 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87f0c81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec112000 con 0x7f87ec10c560 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87f0c81640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec112300 con 0x7f87ec103220 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 0x7f87ec1013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 0x7f87ec1013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58668/0 (socket says 172.21.15.57:58668) 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 -- 172.21.15.57:0/327671948 learned_addr learned my addr 172.21.15.57:0/327671948 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87ea59c640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87ec103220 0x7f87ec100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87ead9d640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87ec10c560 0x7f87ec1018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 -- 172.21.15.57:0/327671948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87ec103220 msgr2=0x7f87ec100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87ec103220 0x7f87ec100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 -- 172.21.15.57:0/327671948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87ec10c560 msgr2=0x7f87ec1018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87ec10c560 0x7f87ec1018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 -- 172.21.15.57:0/327671948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ec1a6b20 con 0x7f87ec103b40 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.049+0000 7f87e3fff640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 0x7f87ec1013b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f87d400e9f0 tx=0x7f87d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.050+0000 7f87e37fe640 1 -- 172.21.15.57:0/327671948 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d400cd80 con 0x7f87ec103b40 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.050+0000 7f87e37fe640 1 -- 172.21.15.57:0/327671948 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d40108e0 con 0x7f87ec103b40 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.050+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ec1a6e10 con 0x7f87ec103b40 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.050+0000 7f87e37fe640 1 -- 172.21.15.57:0/327671948 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4018980 con 0x7f87ec103b40 2024-04-21T08:52:34.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.050+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ec1a7320 con 0x7f87ec103b40 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.052+0000 7f87f3873740 1 Processor -- start 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 -- start start 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 0x556cb07e9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07f30e0 0x556cb07f54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556cb074e8d0 con 0x556cb07ea8d0 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556cb06ebe80 con 0x556cb07e9b60 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87f3873740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556cb074a890 con 0x556cb07f30e0 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87e37fe640 1 -- 172.21.15.57:0/327671948 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f87d4018b20 con 0x7f87ec103b40 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87ea59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 0x556cb07e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87ea59c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07e9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40316/0 (socket says 172.21.15.57:40316) 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 0x556cb07e9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58670/0 (socket says 172.21.15.57:58670) 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87ea59c640 1 -- 172.21.15.57:0/297949815 learned_addr learned my addr 172.21.15.57:0/297949815 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.053+0000 7f87e3fff640 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07f30e0 0x556cb07f54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ead9d640 1 -- 172.21.15.57:0/297949815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07f30e0 msgr2=0x556cb07f54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ead9d640 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07f30e0 0x556cb07f54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ead9d640 1 -- 172.21.15.57:0/297949815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 msgr2=0x556cb07e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ead9d640 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ead9d640 1 -- 172.21.15.57:0/297949815 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb070d320 con 0x556cb07ea8d0 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ea59c640 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ead9d640 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 0x556cb07e9620 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f87e402c790 tx=0x7f87e4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4344087e42043a04 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87e0ff9640 1 -- 172.21.15.57:0/297949815 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4043070 con 0x556cb07ea8d0 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87e37fe640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f87d80a3e40 0x7f87d80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87e0ff9640 1 -- 172.21.15.57:0/297949815 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e4003b70 con 0x556cb07ea8d0 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87e0ff9640 1 -- 172.21.15.57:0/297949815 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4039040 con 0x556cb07ea8d0 2024-04-21T08:52:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87ea59c640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f87d80a3e40 0x7f87d80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.054+0000 7f87e37fe640 1 -- 172.21.15.57:0/327671948 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f87d40d0e20 con 0x7f87ec103b40 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 -- 172.21.15.57:0/297949815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 msgr2=0x556cb07e9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 0x556cb07e9620 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f87e402c790 tx=0x7f87e4009900 comp rx=0 tx=0).stop 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87ea59c640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f87d80a3e40 0x7f87d80a6300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x556cb07eb030 tx=0x7f87dc005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 -- 172.21.15.57:0/297949815 shutdown_connections 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07f30e0 0x556cb07f54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 --2- 172.21.15.57:0/297949815 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07ea8d0 0x556cb07e9620 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 -- 172.21.15.57:0/297949815 >> 172.21.15.57:0/297949815 conn(0x556cb07b0c90 msgr2=0x556cb07e4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 -- 172.21.15.57:0/297949815 shutdown_connections 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 -- 172.21.15.57:0/297949815 wait complete. 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 -- start start 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.055+0000 7f87f3873740 1 Processor -- start 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 0x556cb07f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07efe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07f30e0 0x556cb07f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556cb07f5db0 con 0x556cb07e9b60 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556cb06d6eb0 con 0x556cb07f30e0 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x556cb07f5f30 con 0x556cb07ea8d0 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87f3873740 10 client.0 _ll_register_callbacks cb 0x556cb0703b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 0x556cb07f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 0x556cb07f2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58682/0 (socket says 172.21.15.57:58682) 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 -- 172.21.15.57:0/441136577 learned_addr learned my addr 172.21.15.57:0/441136577 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87e3fff640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ead9d640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07f30e0 0x556cb07f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 -- 172.21.15.57:0/441136577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 msgr2=0x556cb07efe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07efe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 -- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07f30e0 msgr2=0x556cb07f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07f30e0 0x556cb07f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 -- 172.21.15.57:0/441136577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb0461210 con 0x556cb07e9b60 2024-04-21T08:52:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ead9d640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07f30e0 0x556cb07f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87ea59c640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 0x556cb07f2a80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00fa50 tx=0x7f87dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68025bc3c5a543f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.056+0000 7f87d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87d1ffb640 1 -- 172.21.15.57:0/441136577 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc02d060 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87d1ffb640 1 -- 172.21.15.57:0/441136577 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f87dc018030 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e4003b70 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87d1ffb640 1 -- 172.21.15.57:0/441136577 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f87dc034050 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87f3873740 10 client.16821 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556cb052d620 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.057+0000 7f87d1ffb640 1 -- 172.21.15.57:0/441136577 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87dc01dad0 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87d1ffb640 1 -- 172.21.15.57:0/441136577 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f87dc0287f0 con 0x556cb07e9b60 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87f3873740 10 client.16821 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87f3873740 10 client.16821 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87f3873740 20 client.16821 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87f3873740 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb08030f0 0x556cb08054e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87f3873740 4 client.16821 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f87e4003b70 con 0x556cb08030f0 2024-04-21T08:52:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87e3fff640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb08030f0 0x556cb08054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87d1ffb640 10 client.16821 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.058+0000 7f87e3fff640 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb08030f0 0x556cb08054e0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a282f8feff492a99 server_cookie=890f71de05bddc4e in_seq=0 out_seq=0 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87d1ffb640 1 -- 172.21.15.57:0/441136577 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f87e4003b70 con 0x556cb08030f0 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87d1ffb640 10 client.16821 handle_command_reply: tid=0 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 client.16821 shutdown 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 msgr2=0x556cb07f2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 0x556cb07f2a80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00fa50 tx=0x7f87dc004730 comp rx=0 tx=0).stop 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 shutdown_connections 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb08030f0 0x556cb08054e0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07f30e0 0x556cb07f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07efe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87d1ffb640 0 client.16821 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 --2- 172.21.15.57:0/441136577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07e9b60 0x556cb07f2a80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 >> 172.21.15.57:0/441136577 conn(0x556cb07b0c90 msgr2=0x556cb07e65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 shutdown_connections 2024-04-21T08:52:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 1 -- 172.21.15.57:0/441136577 wait complete. 2024-04-21T08:52:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.060+0000 7f87f3873740 20 client.16821 trim_cache size 0 max 16384 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.076+0000 7f87f3873740 1 Processor -- start 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.076+0000 7f87f3873740 1 -- start start 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87f3873740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87f3873740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07f2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87f3873740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87f3873740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1af610 con 0x556cb07f30e0 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87f3873740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b1e70 con 0x556cb07e9b60 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87f3873740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1afc60 con 0x556cb07ea8d0 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.077 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07f2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58688/0 (socket says 172.21.15.57:58688) 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 -- 172.21.15.57:0/1803984762 learned_addr learned my addr 172.21.15.57:0/1803984762 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87e3fff640 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ea59c640 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 -- 172.21.15.57:0/1803984762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 msgr2=0x556cb07f2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07f2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 -- 172.21.15.57:0/1803984762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 msgr2=0x556cb07fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07fecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 -- 172.21.15.57:0/1803984762 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb07e8e30 con 0x556cb07f30e0 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87e3fff640 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07f2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87ead9d640 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07f2a80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f87e402c1c0 tx=0x7f87e4003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1e2001df181030 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.077+0000 7f87d3fff640 1 -- 172.21.15.57:0/1803984762 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e404c020 con 0x556cb07f30e0 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87d3fff640 1 -- 172.21.15.57:0/1803984762 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e40044f0 con 0x556cb07f30e0 2024-04-21T08:52:34.078 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87d3fff640 1 -- 172.21.15.57:0/1803984762 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4039070 con 0x556cb07f30e0 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 -- 172.21.15.57:0/1803984762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 msgr2=0x556cb07f2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07f2a80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f87e402c1c0 tx=0x7f87e4003b20 comp rx=0 tx=0).stop 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 -- 172.21.15.57:0/1803984762 shutdown_connections 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07f2a80 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07ea8d0 0x556cb07f2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 --2- 172.21.15.57:0/1803984762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07e9b60 0x556cb07fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 -- 172.21.15.57:0/1803984762 >> 172.21.15.57:0/1803984762 conn(0x556cb07b0c90 msgr2=0x556cb07eb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 -- 172.21.15.57:0/1803984762 shutdown_connections 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.078+0000 7f87f3873740 1 -- 172.21.15.57:0/1803984762 wait complete. 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 -- start start 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 Processor -- start 2024-04-21T08:52:34.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07e9b60 0x556cb07fb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07ea8d0 0x556cb07ec850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b0330 con 0x556cb07f30e0 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b17a0 con 0x556cb07ea8d0 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f87ec1b0a00 con 0x556cb07e9b60 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87f3873740 10 client.0 _ll_register_callbacks cb 0x7f87dc00f910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ea59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07e9b60 0x556cb07fb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87e3fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07ea8d0 0x556cb07ec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ea59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07e9b60 0x556cb07fb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:58086/0 (socket says 172.21.15.57:58086) 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07ecd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:58690/0 (socket says 172.21.15.57:58690) 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ea59c640 1 -- 172.21.15.57:0/3800803642 learned_addr learned my addr 172.21.15.57:0/3800803642 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 -- 172.21.15.57:0/3800803642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07e9b60 msgr2=0x556cb07fb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07e9b60 0x556cb07fb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 -- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07ea8d0 msgr2=0x556cb07ec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07ea8d0 0x556cb07ec850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.079+0000 7f87ead9d640 1 -- 172.21.15.57:0/3800803642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb07ed570 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87ead9d640 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07ecd90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f87e402c6c0 tx=0x7f87e4009970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88530789e510a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87d1ffb640 1 -- 172.21.15.57:0/3800803642 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e4055070 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87d1ffb640 1 -- 172.21.15.57:0/3800803642 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9563+0+0 (secure 0 0 0) 0x7f87e4004240 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87e40044f0 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87d1ffb640 1 -- 172.21.15.57:0/3800803642 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f87e4039070 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87f3873740 10 client.16827 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x556cb0869630 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87d1ffb640 1 -- 172.21.15.57:0/3800803642 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87e40044f0 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.080+0000 7f87d1ffb640 1 -- 172.21.15.57:0/3800803642 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f87e405a6c0 con 0x556cb07f30e0 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87f3873740 10 client.16827 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87f3873740 10 client.16827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:34.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87f3873740 20 client.16827 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87f3873740 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb0869850 0x556cb086bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87f3873740 4 client.16827 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556cb0762880 con 0x556cb0869850 2024-04-21T08:52:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87ea59c640 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb0869850 0x556cb086bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87ea59c640 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb0869850 0x556cb086bc40 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf108f5654261764 server_cookie=ae5dccb3e6e00814 in_seq=0 out_seq=0 2024-04-21T08:52:34.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:34.081+0000 7f87d1ffb640 10 client.16827 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:34 smithi129 ceph-mon[31272]: pgmap v218: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 1.17k op/s 2024-04-21T08:52:34.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:34 smithi057 ceph-mon[27968]: pgmap v218: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 1.17k op/s 2024-04-21T08:52:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:34 smithi136 ceph-mon[31791]: pgmap v218: 129 pgs: 3 active+clean+snaptrim, 119 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.0 MiB/s wr, 1.17k op/s 2024-04-21T08:52:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:36 smithi129 ceph-mon[31272]: pgmap v219: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 390 op/s 2024-04-21T08:52:36.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:36 smithi129 ceph-mon[31272]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:52:36.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:36 smithi057 ceph-mon[27968]: pgmap v219: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 390 op/s 2024-04-21T08:52:36.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:36 smithi057 ceph-mon[27968]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:52:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:36 smithi136 ceph-mon[31791]: pgmap v219: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 390 op/s 2024-04-21T08:52:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:36 smithi136 ceph-mon[31791]: osdmap e96: 12 total, 12 up, 12 in 2024-04-21T08:52:37.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:37 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:37.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:37 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:37.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:37 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:52:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:38 smithi129 ceph-mon[31272]: pgmap v221: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 902 op/s 2024-04-21T08:52:38.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:38 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:38.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:38 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:38.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:38 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:52:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:38 smithi057 ceph-mon[27968]: pgmap v221: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 902 op/s 2024-04-21T08:52:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:38 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:38 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:38 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:52:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:38 smithi136 ceph-mon[31791]: pgmap v221: 129 pgs: 2 active+clean+snaptrim, 120 active+clean, 7 active+clean+snaptrim_wait; 1.3 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 902 op/s 2024-04-21T08:52:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:38 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:52:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:38 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:52:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:38 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:52:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:40 smithi129 ceph-mon[31272]: pgmap v222: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 1.21k op/s 2024-04-21T08:52:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:40 smithi129 ceph-mon[31272]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:52:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:40 smithi057 ceph-mon[27968]: pgmap v222: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 1.21k op/s 2024-04-21T08:52:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:40 smithi057 ceph-mon[27968]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:52:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:40 smithi136 ceph-mon[31791]: pgmap v222: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 1.21k op/s 2024-04-21T08:52:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:40 smithi136 ceph-mon[31791]: osdmap e97: 12 total, 12 up, 12 in 2024-04-21T08:52:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87d1ffb640 1 -- 172.21.15.57:0/3800803642 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556cb0762880 con 0x556cb0869850 2024-04-21T08:52:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87d1ffb640 10 client.16827 handle_command_reply: tid=0 2024-04-21T08:52:42.481 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 client.16827 shutdown 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 msgr2=0x556cb07ecd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07ecd90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f87e402c6c0 tx=0x7f87e4009970 comp rx=0 tx=0).stop 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 shutdown_connections 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x556cb0869850 0x556cb086bc40 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556cb07f30e0 0x556cb07ecd90 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556cb07ea8d0 0x556cb07ec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 --2- 172.21.15.57:0/3800803642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x556cb07e9b60 0x556cb07fb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 >> 172.21.15.57:0/3800803642 conn(0x556cb07b0c90 msgr2=0x556cb07ee5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87d1ffb640 0 client.16827 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 shutdown_connections 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 1 -- 172.21.15.57:0/3800803642 wait complete. 2024-04-21T08:52:42.482 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.481+0000 7f87f3873740 20 client.16827 trim_cache size 0 max 16384 2024-04-21T08:52:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:52:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:52:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:52:42.483 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f87d80a3e40 msgr2=0x7f87d80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f87d80a3e40 0x7f87d80a6300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x556cb07eb030 tx=0x7f87dc005f90 comp rx=0 tx=0).stop 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 msgr2=0x7f87ec1013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 0x7f87ec1013b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f87d400e9f0 tx=0x7f87d400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 shutdown_connections 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f87d80a3e40 0x7f87d80a6300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87ec10c560 0x7f87ec1018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87ec103b40 0x7f87ec1013b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 --2- 172.21.15.57:0/327671948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87ec103220 0x7f87ec100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 >> 172.21.15.57:0/327671948 conn(0x7f87ec077fd0 msgr2=0x7f87ec0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 shutdown_connections 2024-04-21T08:52:42.485 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:42.482+0000 7f87f0c81640 1 -- 172.21.15.57:0/327671948 wait complete. 2024-04-21T08:52:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:42 smithi129 ceph-mon[31272]: pgmap v224: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 2.0 MiB/s wr, 1.36k op/s 2024-04-21T08:52:42.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:42 smithi057 ceph-mon[27968]: pgmap v224: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 2.0 MiB/s wr, 1.36k op/s 2024-04-21T08:52:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:42 smithi136 ceph-mon[31791]: pgmap v224: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 2.0 MiB/s wr, 1.36k op/s 2024-04-21T08:52:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:44 smithi129 ceph-mon[31272]: pgmap v225: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 2.42k op/s 2024-04-21T08:52:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:44 smithi057 ceph-mon[27968]: pgmap v225: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 2.42k op/s 2024-04-21T08:52:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:44 smithi136 ceph-mon[31791]: pgmap v225: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 6.2 MiB/s wr, 2.42k op/s 2024-04-21T08:52:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:46 smithi129 ceph-mon[31272]: pgmap v226: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 5.7 MiB/s wr, 2.14k op/s 2024-04-21T08:52:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:46 smithi057 ceph-mon[27968]: pgmap v226: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 5.7 MiB/s wr, 2.14k op/s 2024-04-21T08:52:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:46 smithi136 ceph-mon[31791]: pgmap v226: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 5.7 MiB/s wr, 2.14k op/s 2024-04-21T08:52:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:48 smithi129 ceph-mon[31272]: pgmap v227: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 2.22k op/s 2024-04-21T08:52:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:48 smithi057 ceph-mon[27968]: pgmap v227: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 2.22k op/s 2024-04-21T08:52:48.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:48 smithi136 ceph-mon[31791]: pgmap v227: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 2.22k op/s 2024-04-21T08:52:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:50 smithi129 ceph-mon[31272]: pgmap v228: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 2.82k op/s 2024-04-21T08:52:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:50 smithi129 ceph-mon[31272]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:52:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:50 smithi057 ceph-mon[27968]: pgmap v228: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 2.82k op/s 2024-04-21T08:52:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:50 smithi057 ceph-mon[27968]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:52:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:50 smithi136 ceph-mon[31791]: pgmap v228: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 2.82k op/s 2024-04-21T08:52:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:50 smithi136 ceph-mon[31791]: osdmap e98: 12 total, 12 up, 12 in 2024-04-21T08:52:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:52 smithi129 ceph-mon[31272]: pgmap v230: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 2.82k op/s 2024-04-21T08:52:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:52 smithi057 ceph-mon[27968]: pgmap v230: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 2.82k op/s 2024-04-21T08:52:52.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:52 smithi136 ceph-mon[31791]: pgmap v230: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 2.82k op/s 2024-04-21T08:52:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:54 smithi129 ceph-mon[31272]: pgmap v231: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 2.67k op/s 2024-04-21T08:52:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:54 smithi057 ceph-mon[27968]: pgmap v231: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 2.67k op/s 2024-04-21T08:52:54.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:54 smithi136 ceph-mon[31791]: pgmap v231: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 2.67k op/s 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/413481875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 msgr2=0x7f0a1810ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/413481875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a1810ee70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009ca0 tx=0x7f0a1002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/413481875 shutdown_connections 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/413481875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a1810ee70 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/413481875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a18102b80 0x7f0a18106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/413481875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a181021d0 0x7f0a181025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/413481875 >> 172.21.15.57:0/413481875 conn(0x7f0a180793e0 msgr2=0x7f0a180797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/413481875 shutdown_connections 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.578+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/413481875 wait complete. 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 Processor -- start 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 -- start start 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a181021d0 0x7f0a18198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a18198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a1819cef0 0x7f0a1819d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181121f0 con 0x7f0a181072a0 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18112070 con 0x7f0a181021d0 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1ee8c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18112370 con 0x7f0a1819cef0 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a18198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a18198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:48986/0 (socket says 172.21.15.57:48986) 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a17fff640 1 -- 172.21.15.57:0/3919433036 learned_addr learned my addr 172.21.15.57:0/3919433036 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.579+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a181021d0 0x7f0a18198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a1d429640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a1819cef0 0x7f0a1819d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a17fff640 1 -- 172.21.15.57:0/3919433036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a1819cef0 msgr2=0x7f0a1819d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a17fff640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a1819cef0 0x7f0a1819d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a17fff640 1 -- 172.21.15.57:0/3919433036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a181021d0 msgr2=0x7f0a18198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a17fff640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a181021d0 0x7f0a18198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a17fff640 1 -- 172.21.15.57:0/3919433036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1819d9e0 con 0x7f0a181072a0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a17fff640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a18198b70 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800e9c0 tx=0x7f0a0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a15ffb640 1 -- 172.21.15.57:0/3919433036 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800cd80 con 0x7f0a181072a0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1819dcd0 con 0x7f0a181072a0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a15ffb640 1 -- 172.21.15.57:0/3919433036 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a080054d0 con 0x7f0a181072a0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a15ffb640 1 -- 172.21.15.57:0/3919433036 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08010430 con 0x7f0a181072a0 2024-04-21T08:52:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.580+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a181a54c0 con 0x7f0a181072a0 2024-04-21T08:52:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.582+0000 7f0a15ffb640 1 -- 172.21.15.57:0/3919433036 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0a08004fd0 con 0x7f0a181072a0 2024-04-21T08:52:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 Processor -- start 2024-04-21T08:52:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 -- start start 2024-04-21T08:52:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6fa8d0 0x55765e6f9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e6f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55765e65e8d0 con 0x55765e6fa8d0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a15ffb640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f0a040a3dc0 0x7f0a040a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55765e5fbe80 con 0x55765e6f9b60 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a21a82740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55765e65a890 con 0x55765e7030e0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a1d429640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6fa8d0 0x55765e6f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a1d429640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6fa8d0 0x55765e6f9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49002/0 (socket says 172.21.15.57:49002) 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a1d429640 1 -- 172.21.15.57:0/3140390198 learned_addr learned my addr 172.21.15.57:0/3140390198 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e6f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a1d429640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f0a040a3dc0 0x7f0a040a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a17fff640 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.583+0000 7f0a15ffb640 1 -- 172.21.15.57:0/3919433036 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9431+0+0 (secure 0 0 0) 0x7f0a080d0fb0 con 0x7f0a181072a0 2024-04-21T08:52:55.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1cc28640 1 -- 172.21.15.57:0/3140390198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 msgr2=0x55765e7054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1cc28640 1 -- 172.21.15.57:0/3140390198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6fa8d0 msgr2=0x55765e6f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6fa8d0 0x55765e6f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1cc28640 1 -- 172.21.15.57:0/3140390198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55765e61d320 con 0x55765e6f9b60 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1d429640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f0a040a3dc0 0x7f0a040a6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a10005fe0 tx=0x7f0a100095c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e6f9fe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000ab40 tx=0x7f0a0000f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d761345b0002d2c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a0effd640 1 -- 172.21.15.57:0/3140390198 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00004020 con 0x55765e6f9b60 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a0effd640 1 -- 172.21.15.57:0/3140390198 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a000041c0 con 0x55765e6f9b60 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.584+0000 7f0a0effd640 1 -- 172.21.15.57:0/3140390198 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00018730 con 0x55765e6f9b60 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 -- 172.21.15.57:0/3140390198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 msgr2=0x55765e6f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e6f9fe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000ab40 tx=0x7f0a0000f3c0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 -- 172.21.15.57:0/3140390198 shutdown_connections 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.590 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e6f9fe0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3140390198 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6fa8d0 0x55765e6f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 -- 172.21.15.57:0/3140390198 >> 172.21.15.57:0/3140390198 conn(0x55765e6c0c90 msgr2=0x55765e6f4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 -- 172.21.15.57:0/3140390198 shutdown_connections 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 -- 172.21.15.57:0/3140390198 wait complete. 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 -- start start 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.585+0000 7f0a21a82740 1 Processor -- start 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e702a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6fa8d0 0x55765e6ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55765e705e10 con 0x55765e6f9b60 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55765e5e6eb0 con 0x55765e6fa8d0 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55765e705f90 con 0x55765e7030e0 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 10 client.0 _ll_register_callbacks cb 0x55765e613b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e702a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e702a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49018/0 (socket says 172.21.15.57:49018) 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2862749493 learned_addr learned my addr 172.21.15.57:0/2862749493 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a17fff640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6fa8d0 0x55765e6ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1d429640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2862749493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 msgr2=0x55765e7002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6fa8d0 msgr2=0x55765e6ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6fa8d0 0x55765e6ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2862749493 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55765e371210 con 0x55765e6f9b60 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e702a10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000a520 tx=0x7f0a0000a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd659d3afee8813 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f09f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f09f7fff640 1 -- 172.21.15.57:0/2862749493 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00027020 con 0x55765e6f9b60 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.586+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a000041c0 con 0x55765e6f9b60 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f09f7fff640 1 -- 172.21.15.57:0/2862749493 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9431+0+0 (secure 0 0 0) 0x7f0a00004370 con 0x55765e6f9b60 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f09f7fff640 1 -- 172.21.15.57:0/2862749493 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0a00032050 con 0x55765e6f9b60 2024-04-21T08:52:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 10 client.16839 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55765e43d620 con 0x55765e6f9b60 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f09f7fff640 1 -- 172.21.15.57:0/2862749493 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a000041c0 con 0x55765e6f9b60 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f09f7fff640 1 -- 172.21.15.57:0/2862749493 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0a00004a60 con 0x55765e6f9b60 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 10 client.16839 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 20 client.16839 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e713030 0x55765e715420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 4 client.16839 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55765e672880 con 0x55765e713030 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.587+0000 7f0a17fff640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e713030 0x55765e715420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.588+0000 7f0a17fff640 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e713030 0x55765e715420 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea15cbd2519d22f3 server_cookie=ae56b70e44bfd410 in_seq=0 out_seq=0 2024-04-21T08:52:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.588+0000 7f09f7fff640 10 client.16839 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f09f7fff640 1 -- 172.21.15.57:0/2862749493 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55765e672880 con 0x55765e713030 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f09f7fff640 10 client.16839 handle_command_reply: tid=0 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 client.16839 shutdown 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 msgr2=0x55765e702a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e702a10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000a520 tx=0x7f0a0000a800 comp rx=0 tx=0).stop 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 shutdown_connections 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e713030 0x55765e715420 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e7030e0 0x55765e7002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f09f7fff640 0 client.16839 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6fa8d0 0x55765e6ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.595 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2862749493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e702a10 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 >> 172.21.15.57:0/2862749493 conn(0x55765e6c0c90 msgr2=0x55765e6f65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 shutdown_connections 2024-04-21T08:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 1 -- 172.21.15.57:0/2862749493 wait complete. 2024-04-21T08:52:55.596 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.590+0000 7f0a21a82740 20 client.16839 trim_cache size 0 max 16384 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 Processor -- start 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 -- start start 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e70ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e7024d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 0x55765e702a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181ad2e0 con 0x55765e7030e0 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181afb40 con 0x55765e6f9b60 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a21a82740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181ad930 con 0x55765e6fa8d0 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 0x55765e702a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 0x55765e702a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49020/0 (socket says 172.21.15.57:49020) 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 -- 172.21.15.57:0/3769085182 learned_addr learned my addr 172.21.15.57:0/3769085182 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e70ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a17fff640 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e7024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 -- 172.21.15.57:0/3769085182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 msgr2=0x55765e7024d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e7024d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 -- 172.21.15.57:0/3769085182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 msgr2=0x55765e70ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e70ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 -- 172.21.15.57:0/3769085182 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55765e6f8e30 con 0x55765e7030e0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e70ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.606+0000 7f0a1d429640 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 0x55765e702a10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009ca0 tx=0x7f0a10041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0ec2da1ffcefba server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a0e7fc640 1 -- 172.21.15.57:0/3769085182 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10039070 con 0x55765e7030e0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a0e7fc640 1 -- 172.21.15.57:0/3769085182 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a10055070 con 0x55765e7030e0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a0e7fc640 1 -- 172.21.15.57:0/3769085182 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a100044d0 con 0x55765e7030e0 2024-04-21T08:52:55.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 -- 172.21.15.57:0/3769085182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 msgr2=0x55765e702a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 0x55765e702a10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009ca0 tx=0x7f0a10041000 comp rx=0 tx=0).stop 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 -- 172.21.15.57:0/3769085182 shutdown_connections 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e7030e0 0x55765e702a10 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e7024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 --2- 172.21.15.57:0/3769085182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e6f9b60 0x55765e70ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 -- 172.21.15.57:0/3769085182 >> 172.21.15.57:0/3769085182 conn(0x55765e6c0c90 msgr2=0x55765e6f4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 -- 172.21.15.57:0/3769085182 shutdown_connections 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.607+0000 7f0a21a82740 1 -- 172.21.15.57:0/3769085182 wait complete. 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 -- start start 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 Processor -- start 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e70b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e70b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e7030e0 0x55765e6fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181ae000 con 0x55765e6f9b60 2024-04-21T08:52:55.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181af470 con 0x55765e7030e0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0a181ae6d0 con 0x55765e6fa8d0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a21a82740 10 client.0 _ll_register_callbacks cb 0x7f0a1806b7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e70b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e70b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49030/0 (socket says 172.21.15.57:49030) 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2129349481 learned_addr learned my addr 172.21.15.57:0/2129349481 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1d429640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e7030e0 0x55765e6fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a17fff640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e70b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2129349481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 msgr2=0x55765e70b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e70b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e7030e0 msgr2=0x55765e6fd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e7030e0 0x55765e6fd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.608+0000 7f0a1cc28640 1 -- 172.21.15.57:0/2129349481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55765e6fd8d0 con 0x55765e6f9b60 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a1cc28640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e70b220 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000f8c0 tx=0x7f0a0000aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d201b64f94e5260 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:52:55.613 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f09f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f09f7fff640 1 -- 172.21.15.57:0/2129349481 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0003b060 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f09f7fff640 1 -- 172.21.15.57:0/2129349481 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9431+0+0 (secure 0 0 0) 0x7f0a00004130 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a10055070 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f09f7fff640 1 -- 172.21.15.57:0/2129349481 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0a00042050 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 10 client.16845 fetch_fsmap learned FSMap version 21 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55765e704130 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f09f7fff640 1 -- 172.21.15.57:0/2129349481 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a10055070 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f09f7fff640 1 -- 172.21.15.57:0/2129349481 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f0a00004660 con 0x55765e6f9b60 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 10 client.16845 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 20 client.16845 populate_metadata read hostname 'smithi057' 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e779530 0x55765e77b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:52:55.614 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 4 client.16845 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:52:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.609+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55765e5fe3d0 con 0x55765e779530 2024-04-21T08:52:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.610+0000 7f0a17fff640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e779530 0x55765e77b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:52:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.610+0000 7f09f7fff640 10 client.16845 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:52:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:52:55.610+0000 7f0a17fff640 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e779530 0x55765e77b920 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fceb110c65b028ae server_cookie=2c1a7af54c939a5b in_seq=0 out_seq=0 2024-04-21T08:52:56.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:56 smithi129 ceph-mon[31272]: pgmap v232: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 2.62k op/s 2024-04-21T08:52:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:56 smithi057 ceph-mon[27968]: pgmap v232: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 2.62k op/s 2024-04-21T08:52:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:56 smithi136 ceph-mon[31791]: pgmap v232: 129 pgs: 129 active+clean; 1.3 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 2.62k op/s 2024-04-21T08:52:58.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:52:58 smithi129 ceph-mon[31272]: pgmap v233: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 1.98k op/s 2024-04-21T08:52:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:52:58 smithi057 ceph-mon[27968]: pgmap v233: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 1.98k op/s 2024-04-21T08:52:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:52:58 smithi136 ceph-mon[31791]: pgmap v233: 129 pgs: 129 active+clean; 1.3 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.4 MiB/s wr, 1.98k op/s 2024-04-21T08:52:59.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:52:59.918 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:53:00.140 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.139+0000 7f727fbe2640 1 -- 172.21.15.57:0/2615979426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 msgr2=0x7f727810ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:00.140 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.139+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2615979426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 0x7f727810ec10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7274009890 tx=0x7f727402bbb0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.146+0000 7f727fbe2640 1 -- 172.21.15.57:0/2615979426 shutdown_connections 2024-04-21T08:53:00.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.146+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2615979426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727811b730 0x7f727811db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.146+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2615979426 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 0x7f727810ec10 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.146+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2615979426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f727810e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.146 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.146+0000 7f727fbe2640 1 -- 172.21.15.57:0/2615979426 >> 172.21.15.57:0/2615979426 conn(0x7f727806d070 msgr2=0x7f727806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:00.151 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.152+0000 7f727fbe2640 1 -- 172.21.15.57:0/2615979426 shutdown_connections 2024-04-21T08:53:00.155 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.155+0000 7f727fbe2640 1 -- 172.21.15.57:0/2615979426 wait complete. 2024-04-21T08:53:00.156 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.156+0000 7f727fbe2640 1 Processor -- start 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 -- start start 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 0x7f727811ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f7278116f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727811b730 0x7f7278117460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7278113750 con 0x7f727810e7b0 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72781135d0 con 0x7f72781106d0 2024-04-21T08:53:00.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.157+0000 7f727fbe2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f72781138d0 con 0x7f727811b730 2024-04-21T08:53:00.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.158+0000 7f727d17d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f7278116f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.159+0000 7f727d17d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f7278116f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:47370/0 (socket says 172.21.15.57:47370) 2024-04-21T08:53:00.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.159+0000 7f727d17d640 1 -- 172.21.15.57:0/2269144077 learned_addr learned my addr 172.21.15.57:0/2269144077 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:00.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.159+0000 7f727e17f640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727811b730 0x7f7278117460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.161+0000 7f727d97e640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 0x7f727811ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.162+0000 7f727d17d640 1 -- 172.21.15.57:0/2269144077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727811b730 msgr2=0x7f7278117460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:00.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.162+0000 7f727d17d640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727811b730 0x7f7278117460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.162+0000 7f727d17d640 1 -- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 msgr2=0x7f727811ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:00.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.162+0000 7f727d17d640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 0x7f727811ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.162 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.162+0000 7f727d17d640 1 -- 172.21.15.57:0/2269144077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7278117b90 con 0x7f72781106d0 2024-04-21T08:53:00.164 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.165+0000 7f727d17d640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f7278116f20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f727402c140 tx=0x7f7274004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:00.168 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.168+0000 7f726effd640 1 -- 172.21.15.57:0/2269144077 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274039040 con 0x7f72781106d0 2024-04-21T08:53:00.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.168+0000 7f726effd640 1 -- 172.21.15.57:0/2269144077 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7274042070 con 0x7f72781106d0 2024-04-21T08:53:00.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.168+0000 7f726effd640 1 -- 172.21.15.57:0/2269144077 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274004920 con 0x7f72781106d0 2024-04-21T08:53:00.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.169+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72781ab4d0 con 0x7f72781106d0 2024-04-21T08:53:00.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.169+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72781ab880 con 0x7f72781106d0 2024-04-21T08:53:00.176 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.176+0000 7f72827cc740 1 Processor -- start 2024-04-21T08:53:00.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.180+0000 7f726effd640 1 -- 172.21.15.57:0/2269144077 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7274005030 con 0x7f72781106d0 2024-04-21T08:53:00.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.181+0000 7f726effd640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72640a3f20 0x7f72640a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.181+0000 7f726effd640 1 -- 172.21.15.57:0/2269144077 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9431+0+0 (secure 0 0 0) 0x7f72740f1750 con 0x7f72781106d0 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.182+0000 7f727e17f640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72640a3f20 0x7f72640a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.182+0000 7f72827cc740 1 -- start start 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.183+0000 7f72827cc740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 0x56424d71b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.183+0000 7f72827cc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d724540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.183+0000 7f72827cc740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d726e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.183+0000 7f72827cc740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56424d6801e0 con 0x56424d724a80 2024-04-21T08:53:00.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.183+0000 7f72827cc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56424d61d390 con 0x56424d71bea0 2024-04-21T08:53:00.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.183+0000 7f72827cc740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56424d67c1a0 con 0x56424d71b580 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.185+0000 7f727d97e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 0x56424d71b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.185+0000 7f727d97e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 0x56424d71b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59692/0 (socket says 172.21.15.57:59692) 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.185+0000 7f727d97e640 1 -- 172.21.15.57:0/4128332004 learned_addr learned my addr 172.21.15.57:0/4128332004 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.185+0000 7f727e17f640 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d726e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.185+0000 7f727d97e640 1 -- 172.21.15.57:0/4128332004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 msgr2=0x56424d724540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.185+0000 7f727d97e640 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d724540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.186+0000 7f727d97e640 1 -- 172.21.15.57:0/4128332004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 msgr2=0x56424d726e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:00.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.186+0000 7f727d97e640 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d726e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.186+0000 7f727d97e640 1 -- 172.21.15.57:0/4128332004 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56424d45c7d0 con 0x56424d71b580 2024-04-21T08:53:00.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.187+0000 7f727e17f640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72640a3f20 0x7f72640a63e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7270005c40 tx=0x7f727001ae40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:00.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.189+0000 7f727d97e640 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 0x56424d71b960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f726800cbf0 tx=0x7f726800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386db863f5133616 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:00.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.194+0000 7f7263fff640 1 -- 172.21.15.57:0/4128332004 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268010630 con 0x56424d71b580 2024-04-21T08:53:00.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.194+0000 7f7263fff640 1 -- 172.21.15.57:0/4128332004 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7268010c80 con 0x56424d71b580 2024-04-21T08:53:00.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.195+0000 7f72827cc740 1 -- 172.21.15.57:0/4128332004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 msgr2=0x56424d71b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:00.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.195+0000 7f72827cc740 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 0x56424d71b960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f726800cbf0 tx=0x7f726800e720 comp rx=0 tx=0).stop 2024-04-21T08:53:00.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.203+0000 7f72827cc740 1 -- 172.21.15.57:0/4128332004 shutdown_connections 2024-04-21T08:53:00.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.203+0000 7f72827cc740 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d726e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.203+0000 7f72827cc740 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d724540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.203+0000 7f72827cc740 1 --2- 172.21.15.57:0/4128332004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71b580 0x56424d71b960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.203+0000 7f72827cc740 1 -- 172.21.15.57:0/4128332004 >> 172.21.15.57:0/4128332004 conn(0x56424d715cb0 msgr2=0x56424d716090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:00.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.209+0000 7f72827cc740 1 -- 172.21.15.57:0/4128332004 shutdown_connections 2024-04-21T08:53:00.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.214+0000 7f72827cc740 1 -- 172.21.15.57:0/4128332004 wait complete. 2024-04-21T08:53:00.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.214+0000 7f72827cc740 1 -- start start 2024-04-21T08:53:00.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.214+0000 7f72827cc740 1 Processor -- start 2024-04-21T08:53:00.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d721990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d721ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d722410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x56424d7278c0 con 0x56424d724a80 2024-04-21T08:53:00.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56424d6392f0 con 0x56424d71b580 2024-04-21T08:53:00.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56424d727a40 con 0x56424d71bea0 2024-04-21T08:53:00.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.219+0000 7f72827cc740 10 client.0 _ll_register_callbacks cb 0x56424d646580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:00.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.220+0000 7f727e17f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d722410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.220+0000 7f727e17f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d722410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:49074/0 (socket says 172.21.15.57:49074) 2024-04-21T08:53:00.220 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.220+0000 7f727e17f640 1 -- 172.21.15.57:0/1500723123 learned_addr learned my addr 172.21.15.57:0/1500723123 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:00.221 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.221+0000 7f727d17d640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d721ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.223+0000 7f727d97e640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d721990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.226+0000 7f727e17f640 1 -- 172.21.15.57:0/1500723123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 msgr2=0x56424d721ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:00.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.226+0000 7f727e17f640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d721ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.226+0000 7f727e17f640 1 -- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 msgr2=0x56424d721990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:00.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.226+0000 7f727e17f640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d721990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:00.227 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.226+0000 7f727e17f640 1 -- 172.21.15.57:0/1500723123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56424d3923f0 con 0x56424d724a80 2024-04-21T08:53:00.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.230+0000 7f727e17f640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d722410 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f727001f750 tx=0x7f7270004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb37e197b6d43c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:00.234 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.232+0000 7f72617fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:53:00.234 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.232+0000 7f72617fa640 1 -- 172.21.15.57:0/1500723123 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270027040 con 0x56424d724a80 2024-04-21T08:53:00.234 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.232+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7268010c80 con 0x56424d724a80 2024-04-21T08:53:00.234 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.232+0000 7f72617fa640 1 -- 172.21.15.57:0/1500723123 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9431+0+0 (secure 0 0 0) 0x7f7270030710 con 0x56424d724a80 2024-04-21T08:53:00.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.237+0000 7f72617fa640 1 -- 172.21.15.57:0/1500723123 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7270030910 con 0x56424d724a80 2024-04-21T08:53:00.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.239+0000 7f72827cc740 10 client.16857 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:00.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.239+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56424d3520f0 con 0x56424d724a80 2024-04-21T08:53:00.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.244+0000 7f72617fa640 1 -- 172.21.15.57:0/1500723123 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7268010c80 con 0x56424d724a80 2024-04-21T08:53:00.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.244+0000 7f72617fa640 1 -- 172.21.15.57:0/1500723123 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f727001e410 con 0x56424d724a80 2024-04-21T08:53:00.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.244+0000 7f72827cc740 10 client.16857 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.244+0000 7f72827cc740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.244+0000 7f72827cc740 20 client.16857 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.245+0000 7f72827cc740 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d734f40 0x56424d737360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.245+0000 7f72827cc740 4 client.16857 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.245+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56424d68dc00 con 0x56424d734f40 2024-04-21T08:53:00.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.245+0000 7f727d97e640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d734f40 0x56424d737360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:00.256 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.257+0000 7f727d97e640 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d734f40 0x56424d737360 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38545a4772956c3f server_cookie=536d5b91937948bb in_seq=0 out_seq=0 2024-04-21T08:53:00.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:00.258+0000 7f72617fa640 10 client.16857 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:00 smithi129 ceph-mon[31272]: pgmap v234: 129 pgs: 129 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.7 MiB/s wr, 1.16k op/s 2024-04-21T08:53:00.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:00 smithi057 ceph-mon[27968]: pgmap v234: 129 pgs: 129 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.7 MiB/s wr, 1.16k op/s 2024-04-21T08:53:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:00 smithi136 ceph-mon[31791]: pgmap v234: 129 pgs: 129 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 7.7 MiB/s wr, 1.16k op/s 2024-04-21T08:53:02.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:02 smithi129 ceph-mon[31272]: pgmap v235: 129 pgs: 129 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.2 MiB/s wr, 1.09k op/s 2024-04-21T08:53:02.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:02 smithi057 ceph-mon[27968]: pgmap v235: 129 pgs: 129 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.2 MiB/s wr, 1.09k op/s 2024-04-21T08:53:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:02 smithi136 ceph-mon[31791]: pgmap v235: 129 pgs: 129 active+clean; 1.4 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 7.2 MiB/s wr, 1.09k op/s 2024-04-21T08:53:03.921 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f09f7fff640 1 -- 172.21.15.57:0/2129349481 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55765e5fe3d0 con 0x55765e779530 2024-04-21T08:53:03.921 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f09f7fff640 10 client.16845 handle_command_reply: tid=0 2024-04-21T08:53:03.921 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 client.16845 shutdown 2024-04-21T08:53:03.921 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 msgr2=0x55765e70b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e70b220 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0a0000f8c0 tx=0x7f0a0000aa50 comp rx=0 tx=0).stop 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 shutdown_connections 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55765e779530 0x55765e77b920 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55765e7030e0 0x55765e6fd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55765e6fa8d0 0x55765e70b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 --2- 172.21.15.57:0/2129349481 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55765e6f9b60 0x55765e70b220 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 >> 172.21.15.57:0/2129349481 conn(0x55765e6c0c90 msgr2=0x55765e6fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.922 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.921+0000 7f09f7fff640 0 client.16845 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:03.923 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72617fa640 1 -- 172.21.15.57:0/1500723123 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56424d68dc00 con 0x56424d734f40 2024-04-21T08:53:03.923 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72617fa640 10 client.16857 handle_command_reply: tid=0 2024-04-21T08:53:03.923 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 client.16857 shutdown 2024-04-21T08:53:03.923 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 shutdown_connections 2024-04-21T08:53:03.923 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f0a21a82740 1 -- 172.21.15.57:0/2129349481 wait complete. 2024-04-21T08:53:03.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 msgr2=0x56424d722410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d722410 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f727001f750 tx=0x7f7270004ab0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.924 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f0a21a82740 20 client.16845 trim_cache size 0 max 16384 2024-04-21T08:53:03.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 shutdown_connections 2024-04-21T08:53:03.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d734f40 0x56424d737360 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d722410 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:53:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:53:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:53:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d721ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 --2- 172.21.15.57:0/1500723123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d721990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72617fa640 0 client.16857 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 >> 172.21.15.57:0/1500723123 conn(0x56424d715cb0 msgr2=0x56424d717a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 shutdown_connections 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 1 -- 172.21.15.57:0/1500723123 wait complete. 2024-04-21T08:53:03.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.922+0000 7f72827cc740 20 client.16857 trim_cache size 0 max 16384 2024-04-21T08:53:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f0a040a3dc0 msgr2=0x7f0a040a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f0a040a3dc0 0x7f0a040a6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a10005fe0 tx=0x7f0a100095c0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 msgr2=0x7f0a18198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a18198b70 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800e9c0 tx=0x7f0a0800ee80 comp rx=0 tx=0).stop 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 shutdown_connections 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f0a040a3dc0 0x7f0a040a6280 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a1819cef0 0x7f0a1819d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a181072a0 0x7f0a18198b70 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 --2- 172.21.15.57:0/3919433036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a181021d0 0x7f0a18198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 >> 172.21.15.57:0/3919433036 conn(0x7f0a180793e0 msgr2=0x7f0a180ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 shutdown_connections 2024-04-21T08:53:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:03.923+0000 7f0a1ee8c640 1 -- 172.21.15.57:0/3919433036 wait complete. 2024-04-21T08:53:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 Processor -- start 2024-04-21T08:53:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 -- start start 2024-04-21T08:53:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d7240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d730860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d720af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f725c00dfe0 con 0x56424d724a80 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f725c010840 con 0x56424d71b580 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.941+0000 7f72827cc740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f725c00e630 con 0x56424d71bea0 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d720af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d720af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40992/0 (socket says 172.21.15.57:40992) 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 -- 172.21.15.57:0/170732639 learned_addr learned my addr 172.21.15.57:0/170732639 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727d97e640 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d7240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727d17d640 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d730860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 -- 172.21.15.57:0/170732639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 msgr2=0x56424d730860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d730860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 -- 172.21.15.57:0/170732639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 msgr2=0x56424d7240c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d7240c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 -- 172.21.15.57:0/170732639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56424d71bcd0 con 0x56424d724a80 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f727e17f640 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d720af0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f727001e9d0 tx=0x7f72700308a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f6d112ae573fe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f72637fe640 1 -- 172.21.15.57:0/170732639 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727001ea50 con 0x56424d724a80 2024-04-21T08:53:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f72637fe640 1 -- 172.21.15.57:0/170732639 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7270028040 con 0x56424d724a80 2024-04-21T08:53:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.942+0000 7f72637fe640 1 -- 172.21.15.57:0/170732639 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270042420 con 0x56424d724a80 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 -- 172.21.15.57:0/170732639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 msgr2=0x56424d720af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d720af0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f727001e9d0 tx=0x7f72700308a0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 -- 172.21.15.57:0/170732639 shutdown_connections 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d724a80 0x56424d720af0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d71bea0 0x56424d730860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 --2- 172.21.15.57:0/170732639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71b580 0x56424d7240c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 -- 172.21.15.57:0/170732639 >> 172.21.15.57:0/170732639 conn(0x56424d715cb0 msgr2=0x56424d7144e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 -- 172.21.15.57:0/170732639 shutdown_connections 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 -- 172.21.15.57:0/170732639 wait complete. 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.943+0000 7f72827cc740 1 -- start start 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 Processor -- start 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 0x56424d72d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d71e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d724a80 0x56424d71e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f725c00ed00 con 0x56424d71b580 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f725c010170 con 0x56424d71bea0 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f725c00f3d0 con 0x56424d724a80 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f72827cc740 10 client.0 _ll_register_callbacks cb 0x7f7270021e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 0x56424d72d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d17d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d71e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 0x56424d72d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:40996/0 (socket says 172.21.15.57:40996) 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 -- 172.21.15.57:0/4177280853 learned_addr learned my addr 172.21.15.57:0/4177280853 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727e17f640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d724a80 0x56424d71e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 -- 172.21.15.57:0/4177280853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d724a80 msgr2=0x56424d71e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d724a80 0x56424d71e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 -- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 msgr2=0x56424d71e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d71e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d97e640 1 -- 172.21.15.57:0/4177280853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56424d71f110 con 0x56424d71b580 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.944+0000 7f727d17d640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d71e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f727d97e640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 0x56424d72d160 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f726800f750 tx=0x7f726800f6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebc220cc7d13f2f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72617fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72617fa640 1 -- 172.21.15.57:0/4177280853 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7268004220 con 0x56424d71b580 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72617fa640 1 -- 172.21.15.57:0/4177280853 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9431+0+0 (secure 0 0 0) 0x7f72680043c0 con 0x56424d71b580 2024-04-21T08:53:03.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7270028040 con 0x56424d71b580 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72617fa640 1 -- 172.21.15.57:0/4177280853 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7268005040 con 0x56424d71b580 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 10 client.16863 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x56424d79b550 con 0x56424d71b580 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72617fa640 1 -- 172.21.15.57:0/4177280853 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7270028040 con 0x56424d71b580 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72617fa640 1 -- 172.21.15.57:0/4177280853 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7268004890 con 0x56424d71b580 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 10 client.16863 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 10 client.16863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 20 client.16863 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d79bad0 0x56424d79dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 4 client.16863 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.945+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56424d687a70 con 0x56424d79bad0 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.946+0000 7f727d17d640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d79bad0 0x56424d79dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.946+0000 7f72617fa640 10 client.16863 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.946+0000 7f727d17d640 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d79bad0 0x56424d79dec0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=357405e9d29bf4 server_cookie=b6b3f5b266c225b2 in_seq=0 out_seq=0 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72617fa640 1 -- 172.21.15.57:0/4177280853 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x56424d687a70 con 0x56424d79bad0 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72617fa640 10 client.16863 handle_command_reply: tid=0 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 client.16863 shutdown 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 msgr2=0x56424d72d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 0x56424d72d160 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f726800f750 tx=0x7f726800f6e0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 shutdown_connections 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x56424d79bad0 0x56424d79dec0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56424d724a80 0x56424d71e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56424d71bea0 0x56424d71e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72617fa640 0 client.16863 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 --2- 172.21.15.57:0/4177280853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x56424d71b580 0x56424d72d160 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 >> 172.21.15.57:0/4177280853 conn(0x56424d715cb0 msgr2=0x56424d721950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 shutdown_connections 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 1 -- 172.21.15.57:0/4177280853 wait complete. 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.948+0000 7f72827cc740 20 client.16863 trim_cache size 0 max 16384 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (15277 inodes in the stack)", 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stdout: "0b874346-6e34-4cca-8a18-946870a8591d": { 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:53:03.953 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "0b874346-6e34-4cca-8a18-946870a8591d", 2024-04-21T08:53:03.954 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:53:03.954 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:53:03.954 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:53:03.954 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72640a3f20 msgr2=0x7f72640a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72640a3f20 0x7f72640a63e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7270005c40 tx=0x7f727001ae40 comp rx=0 tx=0).stop 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 msgr2=0x7f7278116f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f7278116f20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f727402c140 tx=0x7f7274004270 comp rx=0 tx=0).stop 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 shutdown_connections 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72640a3f20 0x7f72640a63e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727811b730 0x7f7278117460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72781106d0 0x7f7278116f20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 --2- 172.21.15.57:0/2269144077 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f727810e7b0 0x7f727811ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 >> 172.21.15.57:0/2269144077 conn(0x7f727806d070 msgr2=0x7f727811c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:03.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 shutdown_connections 2024-04-21T08:53:03.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:03.949+0000 7f727fbe2640 1 -- 172.21.15.57:0/2269144077 wait complete. 2024-04-21T08:53:03.959 INFO:tasks.cephfs.filesystem:scrub status for tag:0b874346-6e34-4cca-8a18-946870a8591d - {'path': '/', 'tag': '0b874346-6e34-4cca-8a18-946870a8591d', 'options': 'recursive,force'} 2024-04-21T08:53:04.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:04 smithi129 ceph-mon[31272]: pgmap v236: 129 pgs: 129 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.0 MiB/s wr, 2.28k op/s 2024-04-21T08:53:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:04 smithi057 ceph-mon[27968]: pgmap v236: 129 pgs: 129 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.0 MiB/s wr, 2.28k op/s 2024-04-21T08:53:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:04 smithi136 ceph-mon[31791]: pgmap v236: 129 pgs: 129 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.0 MiB/s wr, 2.28k op/s 2024-04-21T08:53:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:06 smithi129 ceph-mon[31272]: pgmap v237: 129 pgs: 129 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 1.78k op/s 2024-04-21T08:53:06.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:06 smithi057 ceph-mon[27968]: pgmap v237: 129 pgs: 129 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 1.78k op/s 2024-04-21T08:53:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:06 smithi136 ceph-mon[31791]: pgmap v237: 129 pgs: 129 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.0 MiB/s wr, 1.78k op/s 2024-04-21T08:53:07.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:07 smithi129 ceph-mon[31272]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:53:07.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:07 smithi057 ceph-mon[27968]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:53:07.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:07 smithi136 ceph-mon[31791]: osdmap e99: 12 total, 12 up, 12 in 2024-04-21T08:53:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:08 smithi129 ceph-mon[31272]: pgmap v239: 129 pgs: 7 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 100 active+clean; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.1 MiB/s wr, 1.99k op/s 2024-04-21T08:53:08.857 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:08 smithi057 ceph-mon[27968]: pgmap v239: 129 pgs: 7 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 100 active+clean; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.1 MiB/s wr, 1.99k op/s 2024-04-21T08:53:08.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:08 smithi136 ceph-mon[31791]: pgmap v239: 129 pgs: 7 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 100 active+clean; 1.3 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.1 MiB/s wr, 1.99k op/s 2024-04-21T08:53:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:10 smithi129 ceph-mon[31272]: pgmap v240: 129 pgs: 16 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 2.07k op/s 2024-04-21T08:53:10.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:10 smithi057 ceph-mon[27968]: pgmap v240: 129 pgs: 16 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 2.07k op/s 2024-04-21T08:53:10.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:10 smithi136 ceph-mon[31791]: pgmap v240: 129 pgs: 16 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 2.07k op/s 2024-04-21T08:53:12.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:12 smithi129 ceph-mon[31272]: pgmap v241: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 2.10k op/s 2024-04-21T08:53:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:12 smithi057 ceph-mon[27968]: pgmap v241: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 2.10k op/s 2024-04-21T08:53:12.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:12 smithi136 ceph-mon[31791]: pgmap v241: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.5 MiB/s wr, 2.10k op/s 2024-04-21T08:53:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:14 smithi129 ceph-mon[31272]: pgmap v242: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 792 op/s 2024-04-21T08:53:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:14 smithi057 ceph-mon[27968]: pgmap v242: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 792 op/s 2024-04-21T08:53:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:14 smithi136 ceph-mon[31791]: pgmap v242: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 792 op/s 2024-04-21T08:53:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:16 smithi129 ceph-mon[31272]: pgmap v243: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 813 op/s 2024-04-21T08:53:16.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:16 smithi057 ceph-mon[27968]: pgmap v243: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 813 op/s 2024-04-21T08:53:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:16 smithi136 ceph-mon[31791]: pgmap v243: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 813 op/s 2024-04-21T08:53:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.085+0000 7f6ddc954640 1 -- 172.21.15.57:0/1447490308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd811b730 msgr2=0x7f6dd811db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.085+0000 7f6ddc954640 1 --2- 172.21.15.57:0/1447490308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd811b730 0x7f6dd811db20 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009890 tx=0x7f6dcc02bc10 comp rx=0 tx=0).stop 2024-04-21T08:53:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.087+0000 7f6ddc954640 1 -- 172.21.15.57:0/1447490308 shutdown_connections 2024-04-21T08:53:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.087+0000 7f6ddc954640 1 --2- 172.21.15.57:0/1447490308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd811b730 0x7f6dd811db20 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.087+0000 7f6ddc954640 1 --2- 172.21.15.57:0/1447490308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dd8110b20 0x7f6dd8109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.087+0000 7f6ddc954640 1 --2- 172.21.15.57:0/1447490308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dd8110740 0x7f6dd8108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.087+0000 7f6ddc954640 1 -- 172.21.15.57:0/1447490308 >> 172.21.15.57:0/1447490308 conn(0x7f6dd806d020 msgr2=0x7f6dd806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.092+0000 7f6ddc954640 1 -- 172.21.15.57:0/1447490308 shutdown_connections 2024-04-21T08:53:17.093 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.093+0000 7f6ddc954640 1 -- 172.21.15.57:0/1447490308 wait complete. 2024-04-21T08:53:17.098 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.098+0000 7f6ddc954640 1 Processor -- start 2024-04-21T08:53:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 -- start start 2024-04-21T08:53:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 0x7f6dd811a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.102 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dd811b730 0x7f6dd811af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dd8114a80 0x7f6dd8114f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81113e0 con 0x7f6dd8110740 2024-04-21T08:53:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd8111260 con 0x7f6dd8114a80 2024-04-21T08:53:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.102+0000 7f6ddc954640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd8111560 con 0x7f6dd811b730 2024-04-21T08:53:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.104+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dd8114a80 0x7f6dd8114f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.104+0000 7f6dd5d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dd811b730 0x7f6dd811af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.104+0000 7f6dd659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 0x7f6dd811a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.105+0000 7f6dd659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 0x7f6dd811a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38060/0 (socket says 172.21.15.57:38060) 2024-04-21T08:53:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.105+0000 7f6dd659c640 1 -- 172.21.15.57:0/84433126 learned_addr learned my addr 172.21.15.57:0/84433126 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.107+0000 7f6dd659c640 1 -- 172.21.15.57:0/84433126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dd811b730 msgr2=0x7f6dd811af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.107+0000 7f6dd659c640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dd811b730 0x7f6dd811af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.107+0000 7f6dd659c640 1 -- 172.21.15.57:0/84433126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dd8114a80 msgr2=0x7f6dd8114f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.107+0000 7f6dd659c640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dd8114a80 0x7f6dd8114f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.107+0000 7f6dd659c640 1 -- 172.21.15.57:0/84433126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd8115660 con 0x7f6dd8110740 2024-04-21T08:53:17.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.108+0000 7f6dd659c640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 0x7f6dd811a9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800e9c0 tx=0x7f6dc800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.110+0000 7f6dc77fe640 1 -- 172.21.15.57:0/84433126 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800ccc0 con 0x7f6dd8110740 2024-04-21T08:53:17.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.110+0000 7f6dc77fe640 1 -- 172.21.15.57:0/84433126 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dc80054d0 con 0x7f6dd8110740 2024-04-21T08:53:17.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.110+0000 7f6dc77fe640 1 -- 172.21.15.57:0/84433126 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8010430 con 0x7f6dd8110740 2024-04-21T08:53:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.113+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd81bc8c0 con 0x7f6dd8110740 2024-04-21T08:53:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.113+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd81bcc70 con 0x7f6dd8110740 2024-04-21T08:53:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.117+0000 7f6ddf616740 1 Processor -- start 2024-04-21T08:53:17.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.117+0000 7f6ddf616740 1 -- start start 2024-04-21T08:53:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.117+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69fa8d0 0x55fba69f9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.122+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69fa8d0 0x55fba69f9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.122+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69fa8d0 0x55fba69f9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47568/0 (socket says 172.21.15.57:47568) 2024-04-21T08:53:17.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.127+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69f9b60 0x55fba69f9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.127+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/2327065193 learned_addr learned my addr 172.21.15.57:0/2327065193 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:17.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.128+0000 7f6dc77fe640 1 -- 172.21.15.57:0/84433126 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6dc80046c0 con 0x7f6dd8110740 2024-04-21T08:53:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.128+0000 7f6ddf616740 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba6a030e0 0x55fba6a054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.128+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fba695e8d0 con 0x55fba69f9b60 2024-04-21T08:53:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.128+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fba68fbe80 con 0x55fba6a030e0 2024-04-21T08:53:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.128+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fba695a890 con 0x55fba69fa8d0 2024-04-21T08:53:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.129+0000 7f6dd659c640 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69f9b60 0x55fba69f9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.130+0000 7f6dd5d9b640 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba6a030e0 0x55fba6a054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.135+0000 7f6dc77fe640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6dc00a3fe0 0x7f6dc00a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.135+0000 7f6dc77fe640 1 -- 172.21.15.57:0/84433126 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7f6dc80d0ef0 con 0x7f6dd8110740 2024-04-21T08:53:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.136+0000 7f6dd659c640 1 -- 172.21.15.57:0/2327065193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69fa8d0 msgr2=0x55fba69f9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.136+0000 7f6dd659c640 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69fa8d0 0x55fba69f9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.136+0000 7f6dd659c640 1 -- 172.21.15.57:0/2327065193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba6a030e0 msgr2=0x55fba6a054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.136+0000 7f6dd659c640 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba6a030e0 0x55fba6a054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.136+0000 7f6dd659c640 1 -- 172.21.15.57:0/2327065193 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba691d320 con 0x55fba69f9b60 2024-04-21T08:53:17.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.137+0000 7f6dd5d9b640 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba6a030e0 0x55fba6a054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.137+0000 7f6dd659c640 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69f9b60 0x55fba69f9fe0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80ce1d0 tx=0x7f6dc800efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3c2277a307dcf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.138+0000 7f6dc4ff9640 1 -- 172.21.15.57:0/2327065193 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800ce60 con 0x55fba69f9b60 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.138+0000 7f6dc4ff9640 1 -- 172.21.15.57:0/2327065193 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dc80d4540 con 0x55fba69f9b60 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.138+0000 7f6dc4ff9640 1 -- 172.21.15.57:0/2327065193 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc80a5690 con 0x55fba69f9b60 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.138+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6dc00a3fe0 0x7f6dc00a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.139+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6dc00a3fe0 0x7f6dc00a64a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005780 tx=0x7f6dcc02bc10 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.139+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69f9b60 msgr2=0x55fba69f9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:17.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.139+0000 7f6ddf616740 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69f9b60 0x55fba69f9fe0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80ce1d0 tx=0x7f6dc800efb0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.141+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 shutdown_connections 2024-04-21T08:53:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.141+0000 7f6ddf616740 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba6a030e0 0x55fba6a054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.141+0000 7f6ddf616740 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69f9b60 0x55fba69f9fe0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.141+0000 7f6ddf616740 1 --2- 172.21.15.57:0/2327065193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69fa8d0 0x55fba69f9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.141+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 >> 172.21.15.57:0/2327065193 conn(0x55fba69c0c90 msgr2=0x55fba69f4850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.149+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 shutdown_connections 2024-04-21T08:53:17.149 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.150+0000 7f6ddf616740 1 -- 172.21.15.57:0/2327065193 wait complete. 2024-04-21T08:53:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.150+0000 7f6ddf616740 1 -- start start 2024-04-21T08:53:17.150 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.150+0000 7f6ddf616740 1 Processor -- start 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba6a02b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba69fff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 0x55fba6a00660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fba6a05db0 con 0x55fba6a030e0 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fba68e6eb0 con 0x55fba69fa8d0 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fba6a05f30 con 0x55fba69f9b60 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6ddf616740 10 client.0 _ll_register_callbacks cb 0x55fba6913b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6dd659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba6a02b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.154 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 0x55fba6a00660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6dd659c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba6a02b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47572/0 (socket says 172.21.15.57:47572) 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 0x55fba6a00660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38084/0 (socket says 172.21.15.57:38084) 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.152+0000 7f6dd659c640 1 -- 172.21.15.57:0/3417789705 learned_addr learned my addr 172.21.15.57:0/3417789705 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.153+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3417789705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 msgr2=0x55fba6a02b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.153+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba6a02b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.153+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 msgr2=0x55fba69fff20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.153+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba69fff20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.153+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3417789705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba6671210 con 0x55fba6a030e0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.153+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 0x55fba6a00660 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc0058c0 tx=0x7f6dcc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22aa5e9f33dc9a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.154+0000 7f6dbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.154+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3417789705 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc039040 con 0x55fba6a030e0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.154+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fba6972880 con 0x55fba6a030e0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.154+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3417789705 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7f6dcc0037f0 con 0x55fba6a030e0 2024-04-21T08:53:17.155 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.154+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3417789705 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6dcc0039f0 con 0x55fba6a030e0 2024-04-21T08:53:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.155+0000 7f6ddf616740 10 client.16875 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:17.156 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.155+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fba673d620 con 0x55fba6a030e0 2024-04-21T08:53:17.166 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.166+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3417789705 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fba6972880 con 0x55fba6a030e0 2024-04-21T08:53:17.166 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3417789705 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6dcc03c070 con 0x55fba6a030e0 2024-04-21T08:53:17.166 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6ddf616740 10 client.16875 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6ddf616740 10 client.16875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6ddf616740 20 client.16875 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a13530 0x55fba6a159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6ddf616740 4 client.16875 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.167+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fba68fe3d0 con 0x55fba6a13530 2024-04-21T08:53:17.175 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.176+0000 7f6dd659c640 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a13530 0x55fba6a159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.180+0000 7f6dbdffb640 10 client.16875 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.180+0000 7f6dd659c640 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a13530 0x55fba6a159f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b1978189dd348be server_cookie=e72489f6032c8257 in_seq=0 out_seq=0 2024-04-21T08:53:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.184+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3417789705 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fba68fe3d0 con 0x55fba6a13530 2024-04-21T08:53:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.184+0000 7f6dbdffb640 10 client.16875 handle_command_reply: tid=0 2024-04-21T08:53:17.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.184+0000 7f6ddf616740 1 client.16875 shutdown 2024-04-21T08:53:17.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.187+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 msgr2=0x55fba6a00660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.187+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 0x55fba6a00660 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc0058c0 tx=0x7f6dcc005850 comp rx=0 tx=0).stop 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.189+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 shutdown_connections 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.189+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a13530 0x55fba6a159f0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.189+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba6a030e0 0x55fba6a00660 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.189+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba69fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.189+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3417789705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba6a02b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.189+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 >> 172.21.15.57:0/3417789705 conn(0x55fba69c0c90 msgr2=0x55fba69f65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:17.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.191+0000 7f6dbdffb640 0 client.16875 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.193+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 shutdown_connections 2024-04-21T08:53:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.194+0000 7f6ddf616740 1 -- 172.21.15.57:0/3417789705 wait complete. 2024-04-21T08:53:17.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.195+0000 7f6ddf616740 20 client.16875 trim_cache size 0 max 16384 2024-04-21T08:53:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.231+0000 7f6ddf616740 1 Processor -- start 2024-04-21T08:53:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.235+0000 7f6ddf616740 1 -- start start 2024-04-21T08:53:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.235+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69c0c90 0x55fba6a0ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.235+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba69fb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.235+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba6a02650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.235+0000 7f6ddf616740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd8115f60 con 0x55fba69c0c90 2024-04-21T08:53:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.236+0000 7f6ddf616740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd807e570 con 0x55fba69fa8d0 2024-04-21T08:53:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.236+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba6a02650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.236+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba6a02650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:39826/0 (socket says 172.21.15.57:39826) 2024-04-21T08:53:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.236+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/1948439266 learned_addr learned my addr 172.21.15.57:0/1948439266 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.236+0000 7f6ddf616740 1 -- 172.21.15.57:0/1948439266 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd807c3b0 con 0x55fba69f9b60 2024-04-21T08:53:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.236+0000 7f6dd5d9b640 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba69fb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.237+0000 7f6dd659c640 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69c0c90 0x55fba6a0ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.239+0000 7f6dd659c640 1 -- 172.21.15.57:0/1948439266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 msgr2=0x55fba69fb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.239+0000 7f6dd659c640 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba69fb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.239+0000 7f6dd659c640 1 -- 172.21.15.57:0/1948439266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 msgr2=0x55fba6a02650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.239+0000 7f6dd659c640 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba6a02650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.239+0000 7f6dd659c640 1 -- 172.21.15.57:0/1948439266 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba69f8e30 con 0x55fba69c0c90 2024-04-21T08:53:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.239+0000 7f6dd659c640 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69c0c90 0x55fba6a0ecb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009230 tx=0x7f6dc80094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c66a5adbc79afc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.241+0000 7f6dbffff640 1 -- 172.21.15.57:0/1948439266 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8005680 con 0x55fba69c0c90 2024-04-21T08:53:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.241+0000 7f6dbffff640 1 -- 172.21.15.57:0/1948439266 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dc80a5690 con 0x55fba69c0c90 2024-04-21T08:53:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.241+0000 7f6dbffff640 1 -- 172.21.15.57:0/1948439266 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc80bbbe0 con 0x55fba69c0c90 2024-04-21T08:53:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.246+0000 7f6ddf616740 1 -- 172.21.15.57:0/1948439266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69c0c90 msgr2=0x55fba6a0ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.246+0000 7f6ddf616740 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69c0c90 0x55fba6a0ecb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009230 tx=0x7f6dc80094c0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.250+0000 7f6ddf616740 1 -- 172.21.15.57:0/1948439266 shutdown_connections 2024-04-21T08:53:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.250+0000 7f6ddf616740 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69fa8d0 0x55fba6a02650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.250+0000 7f6ddf616740 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69f9b60 0x55fba69fb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.250+0000 7f6ddf616740 1 --2- 172.21.15.57:0/1948439266 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69c0c90 0x55fba6a0ecb0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.250+0000 7f6ddf616740 1 -- 172.21.15.57:0/1948439266 >> 172.21.15.57:0/1948439266 conn(0x55fba6a056b0 msgr2=0x55fba69faec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:17.256 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.257+0000 7f6ddf616740 1 -- 172.21.15.57:0/1948439266 shutdown_connections 2024-04-21T08:53:17.257 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.258+0000 7f6ddf616740 1 -- 172.21.15.57:0/1948439266 wait complete. 2024-04-21T08:53:17.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.258+0000 7f6ddf616740 1 -- start start 2024-04-21T08:53:17.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.258+0000 7f6ddf616740 1 Processor -- start 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.259+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69c0c90 0x55fba69fd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.259+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69f9b60 0x55fba69fdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.259+0000 7f6ddf616740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 0x55fba6a04080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.259+0000 7f6ddf616740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd807ca30 con 0x55fba69fa8d0 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.259+0000 7f6ddf616740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd807dea0 con 0x55fba69f9b60 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.260+0000 7f6ddf616740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd807d100 con 0x55fba69c0c90 2024-04-21T08:53:17.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.260+0000 7f6ddf616740 10 client.0 _ll_register_callbacks cb 0x7f6dcc0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.260+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 0x55fba6a04080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.260+0000 7f6dd6d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 0x55fba6a04080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38090/0 (socket says 172.21.15.57:38090) 2024-04-21T08:53:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.260+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3356379827 learned_addr learned my addr 172.21.15.57:0/3356379827 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.261+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3356379827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69c0c90 msgr2=0x55fba69fd5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.261+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69c0c90 0x55fba69fd5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.261+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69f9b60 msgr2=0x55fba69fdb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:53:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.261+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69f9b60 0x55fba69fdb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.261+0000 7f6dd6d9d640 1 -- 172.21.15.57:0/3356379827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba6a05f20 con 0x55fba69fa8d0 2024-04-21T08:53:17.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.261+0000 7f6dd6d9d640 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 0x55fba6a04080 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x55fba6a00540 tx=0x7f6dcc043610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4dbbe933a3f17b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.263+0000 7f6dbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:53:17.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.263+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3356379827 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc039040 con 0x55fba69fa8d0 2024-04-21T08:53:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.263+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3356379827 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7f6dcc003da0 con 0x55fba69fa8d0 2024-04-21T08:53:17.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.263+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dc80a5690 con 0x55fba69fa8d0 2024-04-21T08:53:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.271+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3356379827 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6dcc02c5b0 con 0x55fba69fa8d0 2024-04-21T08:53:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.272+0000 7f6ddf616740 10 client.16881 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.272+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fba6a79f90 con 0x55fba69fa8d0 2024-04-21T08:53:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.273+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3356379827 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dcc0437d0 con 0x55fba69fa8d0 2024-04-21T08:53:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.273+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3356379827 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6dcc03c070 con 0x55fba69fa8d0 2024-04-21T08:53:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.274+0000 7f6ddf616740 10 client.16881 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.274+0000 7f6ddf616740 10 client.16881 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.274+0000 7f6ddf616740 20 client.16881 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:17.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.274+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a030e0 0x55fba6a7c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.274+0000 7f6ddf616740 4 client.16881 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.274+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f6dc80a5690 con 0x55fba6a030e0 2024-04-21T08:53:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.281+0000 7f6dd659c640 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a030e0 0x55fba6a7c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.299+0000 7f6dd659c640 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a030e0 0x55fba6a7c770 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3770f831d5aa109f server_cookie=5c1d8608bc20beb0 in_seq=0 out_seq=0 2024-04-21T08:53:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:17.299+0000 7f6dbdffb640 10 client.16881 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:18 smithi129 ceph-mon[31272]: pgmap v244: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 802 op/s 2024-04-21T08:53:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:18 smithi057 ceph-mon[27968]: pgmap v244: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 802 op/s 2024-04-21T08:53:18.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:18 smithi136 ceph-mon[31791]: pgmap v244: 129 pgs: 20 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 51 active+clean; 1.4 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.2 MiB/s wr, 802 op/s 2024-04-21T08:53:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:20 smithi129 ceph-mon[31272]: pgmap v245: 129 pgs: 19 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 734 op/s 2024-04-21T08:53:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:20 smithi057 ceph-mon[27968]: pgmap v245: 129 pgs: 19 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 734 op/s 2024-04-21T08:53:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:20 smithi136 ceph-mon[31791]: pgmap v245: 129 pgs: 19 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.8 MiB/s wr, 734 op/s 2024-04-21T08:53:22.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:22 smithi129 ceph-mon[31272]: pgmap v246: 129 pgs: 19 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 502 op/s 2024-04-21T08:53:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:22 smithi057 ceph-mon[27968]: pgmap v246: 129 pgs: 19 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 502 op/s 2024-04-21T08:53:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:22 smithi136 ceph-mon[31791]: pgmap v246: 129 pgs: 19 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 52 active+clean; 1.4 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.9 MiB/s wr, 502 op/s 2024-04-21T08:53:24.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:24 smithi057 ceph-mon[27968]: pgmap v247: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 1.14k op/s 2024-04-21T08:53:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.865+0000 7f6dbdffb640 1 -- 172.21.15.57:0/3356379827 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f6dc80a5690 con 0x55fba6a030e0 2024-04-21T08:53:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.865+0000 7f6dbdffb640 10 client.16881 handle_command_reply: tid=0 2024-04-21T08:53:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.865+0000 7f6ddf616740 1 client.16881 shutdown 2024-04-21T08:53:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.865+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 msgr2=0x55fba6a04080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.865+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 0x55fba6a04080 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x55fba6a00540 tx=0x7f6dcc043610 comp rx=0 tx=0).stop 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.865+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 shutdown_connections 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fba6a030e0 0x55fba6a7c770 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fba69fa8d0 0x55fba6a04080 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6dbdffb640 0 client.16881 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fba69f9b60 0x55fba69fdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 --2- 172.21.15.57:0/3356379827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fba69c0c90 0x55fba69fd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 >> 172.21.15.57:0/3356379827 conn(0x55fba6a056b0 msgr2=0x55fba69f5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 shutdown_connections 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 1 -- 172.21.15.57:0/3356379827 wait complete. 2024-04-21T08:53:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.866+0000 7f6ddf616740 20 client.16881 trim_cache size 0 max 16384 2024-04-21T08:53:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:53:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:53:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:53:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6dc00a3fe0 msgr2=0x7f6dc00a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6dc00a3fe0 0x7f6dc00a64a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005780 tx=0x7f6dcc02bc10 comp rx=0 tx=0).stop 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 msgr2=0x7f6dd811a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 0x7f6dd811a9c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800e9c0 tx=0x7f6dc800ee80 comp rx=0 tx=0).stop 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 shutdown_connections 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f6dc00a3fe0 0x7f6dc00a64a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6dd8114a80 0x7f6dd8114f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6dd811b730 0x7f6dd811af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 --2- 172.21.15.57:0/84433126 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6dd8110740 0x7f6dd811a9c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 >> 172.21.15.57:0/84433126 conn(0x7f6dd806d020 msgr2=0x7f6dd811bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 shutdown_connections 2024-04-21T08:53:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:24.867+0000 7f6ddc954640 1 -- 172.21.15.57:0/84433126 wait complete. 2024-04-21T08:53:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:24 smithi136 ceph-mon[31791]: pgmap v247: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 1.14k op/s 2024-04-21T08:53:25.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:24 smithi129 ceph-mon[31272]: pgmap v247: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.3 MiB/s wr, 1.14k op/s 2024-04-21T08:53:25.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:25 smithi136 ceph-mon[31791]: pgmap v248: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 1.12k op/s 2024-04-21T08:53:26.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:25 smithi129 ceph-mon[31272]: pgmap v248: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 1.12k op/s 2024-04-21T08:53:26.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:25 smithi057 ceph-mon[27968]: pgmap v248: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.5 MiB/s wr, 1.12k op/s 2024-04-21T08:53:26.526 INFO:tasks.workunit.client.0.smithi057.stderr:+ cd linux-6.5.11 2024-04-21T08:53:26.531 INFO:tasks.workunit.client.0.smithi057.stderr:+ make defconfig 2024-04-21T08:53:28.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:28 smithi136 ceph-mon[31791]: pgmap v249: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 1.08k op/s 2024-04-21T08:53:28.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:28 smithi129 ceph-mon[31272]: pgmap v249: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 1.08k op/s 2024-04-21T08:53:28.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:28 smithi057 ceph-mon[27968]: pgmap v249: 129 pgs: 16 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 66 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 1.08k op/s 2024-04-21T08:53:30.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:30 smithi136 ceph-mon[31791]: pgmap v250: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 1.28k op/s 2024-04-21T08:53:30.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:30 smithi129 ceph-mon[31272]: pgmap v250: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 1.28k op/s 2024-04-21T08:53:30.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:30 smithi057 ceph-mon[27968]: pgmap v250: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 1.28k op/s 2024-04-21T08:53:31.066 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/basic/fixdep 2024-04-21T08:53:31.334 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/conf.o 2024-04-21T08:53:31.672 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/confdata.o 2024-04-21T08:53:32.023 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/expr.o 2024-04-21T08:53:32.291 INFO:tasks.workunit.client.0.smithi057.stdout: LEX scripts/kconfig/lexer.lex.c 2024-04-21T08:53:32.312 INFO:tasks.workunit.client.0.smithi057.stdout: YACC scripts/kconfig/parser.tab.[ch] 2024-04-21T08:53:32.385 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:32 smithi136 ceph-mon[31791]: pgmap v251: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 1.25k op/s 2024-04-21T08:53:32.407 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/lexer.lex.o 2024-04-21T08:53:32.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:32 smithi129 ceph-mon[31272]: pgmap v251: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 1.25k op/s 2024-04-21T08:53:32.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:32 smithi057 ceph-mon[27968]: pgmap v251: 129 pgs: 10 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 91 active+clean; 1.4 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 1.25k op/s 2024-04-21T08:53:32.733 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/menu.o 2024-04-21T08:53:32.939 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/parser.tab.o 2024-04-21T08:53:33.276 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/preprocess.o 2024-04-21T08:53:33.445 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/symbol.o 2024-04-21T08:53:33.947 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kconfig/util.o 2024-04-21T08:53:33.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:53:33.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.077+0000 7fc58def2640 1 -- 172.21.15.57:0/1090518388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103a70 msgr2=0x7fc58810bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.077+0000 7fc58def2640 1 --2- 172.21.15.57:0/1090518388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103a70 0x7fc58810bf50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc578009900 tx=0x7fc57802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 -- 172.21.15.57:0/1090518388 shutdown_connections 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 --2- 172.21.15.57:0/1090518388 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58810c520 0x7fc58810e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 --2- 172.21.15.57:0/1090518388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103a70 0x7fc58810bf50 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 --2- 172.21.15.57:0/1090518388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103150 0x7fc588103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 -- 172.21.15.57:0/1090518388 >> 172.21.15.57:0/1090518388 conn(0x7fc588077c00 msgr2=0x7fc5880ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 -- 172.21.15.57:0/1090518388 shutdown_connections 2024-04-21T08:53:34.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 -- 172.21.15.57:0/1090518388 wait complete. 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 Processor -- start 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.078+0000 7fc58def2640 1 -- start start 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 0x7fc588198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103a70 0x7fc588198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58810c520 0x7fc58819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881121a0 con 0x7fc588103150 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc588112020 con 0x7fc588103a70 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc588112320 con 0x7fc58810c520 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 0x7fc588198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 0x7fc588198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41710/0 (socket says 172.21.15.57:41710) 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 -- 172.21.15.57:0/887206268 learned_addr learned my addr 172.21.15.57:0/887206268 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc587fff640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58810c520 0x7fc58819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc586ffd640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103a70 0x7fc588198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 -- 172.21.15.57:0/887206268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58810c520 msgr2=0x7fc58819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58810c520 0x7fc58819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 -- 172.21.15.57:0/887206268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103a70 msgr2=0x7fc588198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103a70 0x7fc588198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 -- 172.21.15.57:0/887206268 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58819d710 con 0x7fc588103150 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc586ffd640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103a70 0x7fc588198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc5877fe640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 0x7fc588198720 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00e9f0 tx=0x7fc57c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc584ff9640 1 -- 172.21.15.57:0/887206268 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c00cd80 con 0x7fc588103150 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc584ff9640 1 -- 172.21.15.57:0/887206268 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc57c0108e0 con 0x7fc588103150 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58819da00 con 0x7fc588103150 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc584ff9640 1 -- 172.21.15.57:0/887206268 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c018980 con 0x7fc588103150 2024-04-21T08:53:34.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.079+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5881a5290 con 0x7fc588103150 2024-04-21T08:53:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc584ff9640 1 -- 172.21.15.57:0/887206268 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc57c010430 con 0x7fc588103150 2024-04-21T08:53:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 Processor -- start 2024-04-21T08:53:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 -- start start 2024-04-21T08:53:34.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 0x5647de456960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456ea0 0x5647de45f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de461e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5647de3bb1e0 con 0x5647de45fa80 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5647de358390 con 0x5647de456ea0 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.082+0000 7fc590bc3740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5647de3b71a0 con 0x5647de456580 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de461e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 0x5647de456960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 0x5647de456960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42056/0 (socket says 172.21.15.57:42056) 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de461e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41726/0 (socket says 172.21.15.57:41726) 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 -- 172.21.15.57:0/3279841565 learned_addr learned my addr 172.21.15.57:0/3279841565 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc584ff9640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc5700a3d20 0x7fc5700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc5877fe640 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456ea0 0x5647de45f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc5700a3d20 0x7fc5700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 -- 172.21.15.57:0/3279841565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456ea0 msgr2=0x5647de45f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456ea0 0x5647de45f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 -- 172.21.15.57:0/3279841565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 msgr2=0x5647de461e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de461e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 -- 172.21.15.57:0/3279841565 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647de1977d0 con 0x5647de456580 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc584ff9640 1 -- 172.21.15.57:0/887206268 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7fc57c0d0fe0 con 0x7fc588103150 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 0x5647de456960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc580005c40 tx=0x7fc580013f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f87e4df4766a7734 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.083+0000 7fc587fff640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc5700a3d20 0x7fc5700a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc58001d000 tx=0x7fc58001f000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc56dffb640 1 -- 172.21.15.57:0/3279841565 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58000fde0 con 0x5647de456580 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc56dffb640 1 -- 172.21.15.57:0/3279841565 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc580022020 con 0x5647de456580 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc56dffb640 1 -- 172.21.15.57:0/3279841565 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580010690 con 0x5647de456580 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 -- 172.21.15.57:0/3279841565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 msgr2=0x5647de456960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 0x5647de456960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc580005c40 tx=0x7fc580013f50 comp rx=0 tx=0).stop 2024-04-21T08:53:34.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 -- 172.21.15.57:0/3279841565 shutdown_connections 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de461e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456ea0 0x5647de45f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3279841565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456580 0x5647de456960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 -- 172.21.15.57:0/3279841565 >> 172.21.15.57:0/3279841565 conn(0x5647de450cb0 msgr2=0x5647de451090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 -- 172.21.15.57:0/3279841565 shutdown_connections 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.084+0000 7fc590bc3740 1 -- 172.21.15.57:0/3279841565 wait complete. 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 -- start start 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 Processor -- start 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de458d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 0x5647de459240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de45fa80 0x5647de45d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5647de462620 con 0x5647de456ea0 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5647de3742f0 con 0x5647de456580 2024-04-21T08:53:34.091 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5647de4627a0 con 0x5647de45fa80 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc590bc3740 10 client.0 _ll_register_callbacks cb 0x5647de381580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 0x5647de459240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc587fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de45fa80 0x5647de45d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 0x5647de459240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41742/0 (socket says 172.21.15.57:41742) 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de458d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.085+0000 7fc586ffd640 1 -- 172.21.15.57:0/817765777 learned_addr learned my addr 172.21.15.57:0/817765777 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc587fff640 1 -- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 msgr2=0x5647de458d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc587fff640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de458d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc587fff640 1 -- 172.21.15.57:0/817765777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 msgr2=0x5647de459240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc587fff640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 0x5647de459240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc587fff640 1 -- 172.21.15.57:0/817765777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647de0cd3f0 con 0x5647de45fa80 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc586ffd640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 0x5647de459240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc5877fe640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de458d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:34.092 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc587fff640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de45fa80 0x5647de45d4c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc580024040 tx=0x7fc580009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e362a9c7b769e58b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc566ffd640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc566ffd640 1 -- 172.21.15.57:0/817765777 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580034060 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc580022020 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.086+0000 7fc566ffd640 1 -- 172.21.15.57:0/817765777 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7fc5800049d0 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc566ffd640 1 -- 172.21.15.57:0/817765777 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc58003b050 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 10 client.26239 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5647de08d0f0 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc566ffd640 1 -- 172.21.15.57:0/817765777 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc580022020 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc566ffd640 1 -- 172.21.15.57:0/817765777 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc580005670 con 0x5647de45fa80 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 10 client.26239 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 10 client.26239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 20 client.26239 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4781c0 0x5647de47a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.093 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 4 client.26239 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:34.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647de3c8c00 con 0x5647de4781c0 2024-04-21T08:53:34.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.087+0000 7fc5877fe640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4781c0 0x5647de47a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.088+0000 7fc5877fe640 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4781c0 0x5647de47a5b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3db0a919bb99f218 server_cookie=e28d566e50190051 in_seq=0 out_seq=0 2024-04-21T08:53:34.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.088+0000 7fc566ffd640 10 client.26239 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:34.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc566ffd640 1 -- 172.21.15.57:0/817765777 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647de3c8c00 con 0x5647de4781c0 2024-04-21T08:53:34.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc566ffd640 10 client.26239 handle_command_reply: tid=0 2024-04-21T08:53:34.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 client.26239 shutdown 2024-04-21T08:53:34.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de45fa80 msgr2=0x5647de45d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de45fa80 0x5647de45d4c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc580024040 tx=0x7fc580009f80 comp rx=0 tx=0).stop 2024-04-21T08:53:34.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 shutdown_connections 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4781c0 0x5647de47a5b0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de45fa80 0x5647de45d4c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de456ea0 0x5647de459240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc566ffd640 0 client.26239 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 --2- 172.21.15.57:0/817765777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de458d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 >> 172.21.15.57:0/817765777 conn(0x5647de450cb0 msgr2=0x5647de452a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 shutdown_connections 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 1 -- 172.21.15.57:0/817765777 wait complete. 2024-04-21T08:53:34.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.089+0000 7fc590bc3740 20 client.26239 trim_cache size 0 max 16384 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 Processor -- start 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 -- start start 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de45f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de47fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de480280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881ab3f0 con 0x5647de45fa80 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881adc50 con 0x5647de456580 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc590bc3740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881aba40 con 0x5647de456ea0 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc587fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de480280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc587fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de480280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41752/0 (socket says 172.21.15.57:41752) 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc587fff640 1 -- 172.21.15.57:0/3568816933 learned_addr learned my addr 172.21.15.57:0/3568816933 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc586ffd640 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de47fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.113 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.111+0000 7fc5877fe640 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de45f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc587fff640 1 -- 172.21.15.57:0/3568816933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 msgr2=0x5647de47fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc587fff640 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de47fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc587fff640 1 -- 172.21.15.57:0/3568816933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 msgr2=0x5647de45f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc587fff640 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de45f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc587fff640 1 -- 172.21.15.57:0/3568816933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647de4621a0 con 0x5647de45fa80 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc587fff640 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de480280 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc580005a70 tx=0x7fc580013560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c244fc8e753102ce server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc56d7fa640 1 -- 172.21.15.57:0/3568816933 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580034040 con 0x5647de45fa80 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc56d7fa640 1 -- 172.21.15.57:0/3568816933 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc580022030 con 0x5647de45fa80 2024-04-21T08:53:34.114 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.112+0000 7fc56d7fa640 1 -- 172.21.15.57:0/3568816933 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580010430 con 0x5647de45fa80 2024-04-21T08:53:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 -- 172.21.15.57:0/3568816933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 msgr2=0x5647de480280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de480280 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc580005a70 tx=0x7fc580013560 comp rx=0 tx=0).stop 2024-04-21T08:53:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 -- 172.21.15.57:0/3568816933 shutdown_connections 2024-04-21T08:53:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de480280 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.118 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de47fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 --2- 172.21.15.57:0/3568816933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de45f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 -- 172.21.15.57:0/3568816933 >> 172.21.15.57:0/3568816933 conn(0x5647de450cb0 msgr2=0x5647de457d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 -- 172.21.15.57:0/3568816933 shutdown_connections 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 -- 172.21.15.57:0/3568816933 wait complete. 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 -- start start 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.113+0000 7fc590bc3740 1 Processor -- start 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de474a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de46fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de470310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881ac110 con 0x5647de45fa80 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881ad580 con 0x5647de456580 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881ac7e0 con 0x5647de456ea0 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc590bc3740 10 client.0 _ll_register_callbacks cb 0x7fc57c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de470310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.119 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc586ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de46fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc5877fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de474a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de470310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41760/0 (socket says 172.21.15.57:41760) 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 -- 172.21.15.57:0/4205851222 learned_addr learned my addr 172.21.15.57:0/4205851222 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 -- 172.21.15.57:0/4205851222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 msgr2=0x5647de46fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de46fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 -- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 msgr2=0x5647de474a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de474a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 -- 172.21.15.57:0/4205851222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647de470a00 con 0x5647de45fa80 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc587fff640 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de470310 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc58002f450 tx=0x7fc5800011c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef5ff5ff4caa538 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.114+0000 7fc566ffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc566ffd640 1 -- 172.21.15.57:0/4205851222 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580034050 con 0x5647de45fa80 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc566ffd640 1 -- 172.21.15.57:0/4205851222 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7fc58002f720 con 0x5647de45fa80 2024-04-21T08:53:34.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc580022030 con 0x5647de45fa80 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc566ffd640 1 -- 172.21.15.57:0/4205851222 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc580053050 con 0x5647de45fa80 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 10 client.16899 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5647de4de4c0 con 0x5647de45fa80 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc566ffd640 1 -- 172.21.15.57:0/4205851222 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc580022030 con 0x5647de45fa80 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc566ffd640 1 -- 172.21.15.57:0/4205851222 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc580040610 con 0x5647de45fa80 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 10 client.16899 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 10 client.16899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 20 client.16899 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4dead0 0x5647de4e0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 4 client.16899 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.115+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5647de3c2a70 con 0x5647de4dead0 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.116+0000 7fc5877fe640 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4dead0 0x5647de4e0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.116+0000 7fc566ffd640 10 client.16899 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:34.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.116+0000 7fc5877fe640 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4dead0 0x5647de4e0ef0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a290dce91ef05ad server_cookie=c7a37ab8dd114019 in_seq=0 out_seq=0 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc566ffd640 1 -- 172.21.15.57:0/4205851222 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5647de3c2a70 con 0x5647de4dead0 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc566ffd640 10 client.16899 handle_command_reply: tid=0 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 client.16899 shutdown 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 msgr2=0x5647de470310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de470310 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc58002f450 tx=0x7fc5800011c0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 shutdown_connections 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5647de4dead0 0x5647de4e0ef0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5647de45fa80 0x5647de470310 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5647de456ea0 0x5647de46fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 --2- 172.21.15.57:0/4205851222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5647de456580 0x5647de474a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 >> 172.21.15.57:0/4205851222 conn(0x5647de450cb0 msgr2=0x5647de44f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc566ffd640 0 client.16899 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:34.122 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 shutdown_connections 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 1 -- 172.21.15.57:0/4205851222 wait complete. 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.117+0000 7fc590bc3740 20 client.16899 trim_cache size 0 max 16384 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (16623 inodes in the stack)", 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: "0b874346-6e34-4cca-8a18-946870a8591d": { 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "0b874346-6e34-4cca-8a18-946870a8591d", 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:53:34.123 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:53:34.124 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:53:34.124 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:53:34.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc5700a3d20 msgr2=0x7fc5700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc5700a3d20 0x7fc5700a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc58001d000 tx=0x7fc58001f000 comp rx=0 tx=0).stop 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 msgr2=0x7fc588198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 0x7fc588198720 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00e9f0 tx=0x7fc57c00eeb0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 shutdown_connections 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc5700a3d20 0x7fc5700a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58810c520 0x7fc58819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc588103a70 0x7fc588198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 --2- 172.21.15.57:0/887206268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc588103150 0x7fc588198720 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 >> 172.21.15.57:0/887206268 conn(0x7fc588077c00 msgr2=0x7fc5880fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 shutdown_connections 2024-04-21T08:53:34.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:53:34.118+0000 7fc58def2640 1 -- 172.21.15.57:0/887206268 wait complete. 2024-04-21T08:53:34.127 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTLD scripts/kconfig/conf 2024-04-21T08:53:34.132 INFO:tasks.cephfs.filesystem:scrub status for tag:0b874346-6e34-4cca-8a18-946870a8591d - {'path': '/', 'tag': '0b874346-6e34-4cca-8a18-946870a8591d', 'options': 'recursive,force'} 2024-04-21T08:53:34.154 INFO:tasks.workunit.client.0.smithi057.stdout:*** Default configuration is based on 'x86_64_defconfig' 2024-04-21T08:53:34.339 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:34 smithi057 ceph-mon[27968]: pgmap v252: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 1.68k op/s 2024-04-21T08:53:34.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:34 smithi129 ceph-mon[31272]: pgmap v252: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 1.68k op/s 2024-04-21T08:53:34.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:34 smithi136 ceph-mon[31791]: pgmap v252: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.2 MiB/s wr, 1.68k op/s 2024-04-21T08:53:36.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:36 smithi129 ceph-mon[31272]: pgmap v253: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 1.35k op/s 2024-04-21T08:53:36.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:36 smithi057 ceph-mon[27968]: pgmap v253: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 1.35k op/s 2024-04-21T08:53:36.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:36 smithi136 ceph-mon[31791]: pgmap v253: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 1.35k op/s 2024-04-21T08:53:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 -- 172.21.15.57:0/1547538793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 msgr2=0x7f7fbc1025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1547538793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1025c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009900 tx=0x7f7fb002bc30 comp rx=0 tx=0).stop 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 -- 172.21.15.57:0/1547538793 shutdown_connections 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1547538793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fbc1072b0 0x7f7fbc10ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1547538793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc102b90 0x7f7fbc106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1547538793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1025c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 -- 172.21.15.57:0/1547538793 >> 172.21.15.57:0/1547538793 conn(0x7f7fbc0fda00 msgr2=0x7f7fbc0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 -- 172.21.15.57:0/1547538793 shutdown_connections 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.969+0000 7f7fc3011640 1 -- 172.21.15.57:0/1547538793 wait complete. 2024-04-21T08:53:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 Processor -- start 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 -- start start 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc1072b0 0x7f7fbc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fbc19cfb0 0x7f7fbc19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1121f0 con 0x7f7fbc1021e0 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc112070 con 0x7f7fbc19cfb0 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc3011640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc112370 con 0x7f7fbc1072b0 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.970+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41780/0 (socket says 172.21.15.57:41780) 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 -- 172.21.15.57:0/1633045132 learned_addr learned my addr 172.21.15.57:0/1633045132 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc1072b0 0x7f7fbc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fbc19cfb0 0x7f7fbc19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 -- 172.21.15.57:0/1633045132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc1072b0 msgr2=0x7f7fbc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc1072b0 0x7f7fbc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 -- 172.21.15.57:0/1633045132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fbc19cfb0 msgr2=0x7f7fbc19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fbc19cfb0 0x7f7fbc19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 -- 172.21.15.57:0/1633045132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc19daa0 con 0x7f7fbc1021e0 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc1072b0 0x7f7fbc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1986f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0005940 tx=0x7f7fb002c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fb9ffb640 1 -- 172.21.15.57:0/1633045132 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0043020 con 0x7f7fbc1021e0 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc19dd30 con 0x7f7fbc1021e0 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fb9ffb640 1 -- 172.21.15.57:0/1633045132 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb002ca70 con 0x7f7fbc1021e0 2024-04-21T08:53:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.971+0000 7f7fb9ffb640 1 -- 172.21.15.57:0/1633045132 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00039f0 con 0x7f7fbc1021e0 2024-04-21T08:53:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.972+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc1a5580 con 0x7f7fbc1021e0 2024-04-21T08:53:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.974+0000 7f7fb9ffb640 1 -- 172.21.15.57:0/1633045132 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7fb0046020 con 0x7f7fbc1021e0 2024-04-21T08:53:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 Processor -- start 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 -- start start 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fb9ffb640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7fac0a3e40 0x7f7fac0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc418700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc409220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fdbc3708d0 con 0x55fdbc418320 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fdbc30de80 con 0x55fdbc408da0 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fdbc36c890 con 0x55fdbc409760 2024-04-21T08:53:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7fac0a3e40 0x7f7fac0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc418700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc418700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41790/0 (socket says 172.21.15.57:41790) 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc0dad640 1 -- 172.21.15.57:0/2929627996 learned_addr learned my addr 172.21.15.57:0/2929627996 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fb9ffb640 1 -- 172.21.15.57:0/1633045132 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7f7fb00f2ab0 con 0x7f7fbc1021e0 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.975+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc409220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc0dad640 1 -- 172.21.15.57:0/2929627996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 msgr2=0x55fdbc41ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc0dad640 1 -- 172.21.15.57:0/2929627996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 msgr2=0x55fdbc409220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc409220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc0dad640 1 -- 172.21.15.57:0/2929627996 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdbc32f320 con 0x55fdbc418320 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7fac0a3e40 0x7f7fac0a6300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009fd0 tx=0x7f7fb4009de0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc418700 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00be450 tx=0x7f7fb00be290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7e217c279a7d7f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fa6ffd640 1 -- 172.21.15.57:0/2929627996 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00bee50 con 0x55fdbc418320 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fa6ffd640 1 -- 172.21.15.57:0/2929627996 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb008e910 con 0x55fdbc418320 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.976+0000 7f7fa6ffd640 1 -- 172.21.15.57:0/2929627996 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb008e540 con 0x55fdbc418320 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 -- 172.21.15.57:0/2929627996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 msgr2=0x55fdbc418700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc418700 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00be450 tx=0x7f7fb00be290 comp rx=0 tx=0).stop 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 -- 172.21.15.57:0/2929627996 shutdown_connections 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc409220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/2929627996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc418700 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 -- 172.21.15.57:0/2929627996 >> 172.21.15.57:0/2929627996 conn(0x55fdbc3d2c90 msgr2=0x55fdbc407c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 -- 172.21.15.57:0/2929627996 shutdown_connections 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 -- 172.21.15.57:0/2929627996 wait complete. 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 -- start start 2024-04-21T08:53:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.977+0000 7f7fc5c03740 1 Processor -- start 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc41fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fdbc4204d0 con 0x55fdbc418320 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fdbc2f8eb0 con 0x55fdbc408da0 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fdbc420650 con 0x55fdbc409760 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc5c03740 10 client.0 _ll_register_callbacks cb 0x55fdbc325b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc15ae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc15ae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41806/0 (socket says 172.21.15.57:41806) 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc15ae640 1 -- 172.21.15.57:0/3783663583 learned_addr learned my addr 172.21.15.57:0/3783663583 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:37.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc41fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fbbfff640 1 -- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 msgr2=0x55fdbc41fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc41fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fbbfff640 1 -- 172.21.15.57:0/3783663583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 msgr2=0x55fdbc41d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fbbfff640 1 -- 172.21.15.57:0/3783663583 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdbc083210 con 0x55fdbc409760 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.978+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41d160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0fdeb0 tx=0x7f7fa800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4680b0643b7777d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7f9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7f9bfff640 1 -- 172.21.15.57:0/3783663583 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8009e50 con 0x55fdbc409760 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb008e910 con 0x55fdbc409760 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7f9bfff640 1 -- 172.21.15.57:0/3783663583 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7f7fa8012110 con 0x55fdbc409760 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7f9bfff640 1 -- 172.21.15.57:0/3783663583 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7fa8025050 con 0x55fdbc409760 2024-04-21T08:53:37.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7fc5c03740 10 client.26257 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fdbc14f620 con 0x55fdbc409760 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.979+0000 7f7f9bfff640 1 -- 172.21.15.57:0/3783663583 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb008e910 con 0x55fdbc409760 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7f9bfff640 1 -- 172.21.15.57:0/3783663583 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7fa8003c60 con 0x55fdbc409760 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc5c03740 10 client.26257 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc5c03740 10 client.26257 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc5c03740 20 client.26257 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc42fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc5c03740 4 client.26257 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdbc384880 con 0x55fdbc42d670 2024-04-21T08:53:37.986 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc42fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc42fa60 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44d0dbf9c4f18fce server_cookie=3459f1ff28d629b5 in_seq=0 out_seq=0 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.980+0000 7f7f9bfff640 10 client.26257 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7f9bfff640 1 -- 172.21.15.57:0/3783663583 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fdbc384880 con 0x55fdbc42d670 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7f9bfff640 10 client.26257 handle_command_reply: tid=0 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 client.26257 shutdown 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 msgr2=0x55fdbc41d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41d160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc0fdeb0 tx=0x7f7fa800f790 comp rx=0 tx=0).stop 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 shutdown_connections 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc42fa60 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7f9bfff640 0 client.26257 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.981+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc41d160 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.982+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3783663583 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc41fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.982+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 >> 172.21.15.57:0/3783663583 conn(0x55fdbc3d2c90 msgr2=0x55fdbc406b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.982+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 shutdown_connections 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.982+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3783663583 wait complete. 2024-04-21T08:53:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:37.982+0000 7f7fc5c03740 20 client.26257 trim_cache size 0 max 16384 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 Processor -- start 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 -- start start 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc405b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc429270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ab270 con 0x55fdbc418320 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1adad0 con 0x55fdbc408da0 2024-04-21T08:53:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ab8c0 con 0x55fdbc409760 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc429270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc405b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc429270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42102/0 (socket says 172.21.15.57:42102) 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc405b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:51976/0 (socket says 172.21.15.57:51976) 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:41814/0 (socket says 172.21.15.57:41814) 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fbbfff640 1 -- 172.21.15.57:0/3480305838 learned_addr learned my addr 172.21.15.57:0/3480305838 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 -- 172.21.15.57:0/3480305838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 msgr2=0x55fdbc429270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc429270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 -- 172.21.15.57:0/3480305838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 msgr2=0x55fdbc405b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc405b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 -- 172.21.15.57:0/3480305838 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdbc408780 con 0x55fdbc418320 2024-04-21T08:53:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41f890 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4014fd0 tx=0x7f7fb4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303d0ead302979c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fa67fc640 1 -- 172.21.15.57:0/3480305838 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4004100 con 0x55fdbc418320 2024-04-21T08:53:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fa67fc640 1 -- 172.21.15.57:0/3480305838 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb4004750 con 0x55fdbc418320 2024-04-21T08:53:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.002+0000 7f7fa67fc640 1 -- 172.21.15.57:0/3480305838 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4012a30 con 0x55fdbc418320 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3480305838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 msgr2=0x55fdbc41f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41f890 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4014fd0 tx=0x7f7fb4009500 comp rx=0 tx=0).stop 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3480305838 shutdown_connections 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc418320 0x55fdbc41f890 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc409760 0x55fdbc429270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/3480305838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc408da0 0x55fdbc405b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3480305838 >> 172.21.15.57:0/3480305838 conn(0x55fdbc3d2c90 msgr2=0x55fdbc406b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3480305838 shutdown_connections 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 -- 172.21.15.57:0/3480305838 wait complete. 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 -- start start 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.003+0000 7f7fc5c03740 1 Processor -- start 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 0x55fdbc425d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc409760 0x55fdbc416a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc418320 0x55fdbc417160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1abf90 con 0x55fdbc409760 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ad400 con 0x55fdbc418320 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc1ac660 con 0x55fdbc408da0 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc5c03740 10 client.0 _ll_register_callbacks cb 0x7f7fa8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 0x55fdbc425d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 0x55fdbc425d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42116/0 (socket says 172.21.15.57:42116) 2024-04-21T08:53:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 -- 172.21.15.57:0/4229597814 learned_addr learned my addr 172.21.15.57:0/4229597814 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc15ae640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc418320 0x55fdbc417160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc409760 0x55fdbc416a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 -- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc418320 msgr2=0x55fdbc417160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc418320 0x55fdbc417160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 -- 172.21.15.57:0/4229597814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc409760 msgr2=0x55fdbc416a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc409760 0x55fdbc416a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.004+0000 7f7fc0dad640 1 -- 172.21.15.57:0/4229597814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdbc423850 con 0x55fdbc408da0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7fc0dad640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 0x55fdbc425d10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00f0370 tx=0x7f7fb008d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=583c7c6c7528ddf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7f9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7f9bfff640 1 -- 172.21.15.57:0/4229597814 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb00bee50 con 0x55fdbc408da0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7f9bfff640 1 -- 172.21.15.57:0/4229597814 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9563+0+0 (secure 0 0 0) 0x7f7fb00badc0 con 0x55fdbc408da0 2024-04-21T08:53:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb4004750 con 0x55fdbc408da0 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7f9bfff640 1 -- 172.21.15.57:0/4229597814 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7fb00c2b00 con 0x55fdbc408da0 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7fc5c03740 10 client.26263 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fdbc493dc0 con 0x55fdbc408da0 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7f9bfff640 1 -- 172.21.15.57:0/4229597814 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7fb4004750 con 0x55fdbc408da0 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7f9bfff640 1 -- 172.21.15.57:0/4229597814 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f7fb00d6070 con 0x55fdbc408da0 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.005+0000 7f7fc5c03740 10 client.26263 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fc5c03740 10 client.26263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fc5c03740 20 client.26263 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc496330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fc5c03740 4 client.26263 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdbc3103d0 con 0x55fdbc42d670 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc496330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7f9bfff640 10 client.26263 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:38.012 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:38.006+0000 7f7fbbfff640 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc496330 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ef3023dac667c73 server_cookie=f8f020c2d630d14 in_seq=0 out_seq=0 2024-04-21T08:53:38.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:38 smithi129 ceph-mon[31272]: pgmap v254: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 1.47k op/s 2024-04-21T08:53:38.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:38 smithi057 ceph-mon[27968]: pgmap v254: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 1.47k op/s 2024-04-21T08:53:38.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:38 smithi136 ceph-mon[31791]: pgmap v254: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 1.47k op/s 2024-04-21T08:53:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:39 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:39.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:39 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:39.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:39 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:53:40.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:40 smithi129 ceph-mon[31272]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 2.40k op/s 2024-04-21T08:53:40.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:40 smithi057 ceph-mon[27968]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 2.40k op/s 2024-04-21T08:53:40.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:40 smithi136 ceph-mon[31791]: pgmap v255: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 2.40k op/s 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.763+0000 7f7f9bfff640 1 -- 172.21.15.57:0/4229597814 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdbc3103d0 con 0x55fdbc42d670 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.763+0000 7f7f9bfff640 10 client.26263 handle_command_reply: tid=0 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.763+0000 7f7fc5c03740 1 client.26263 shutdown 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 msgr2=0x55fdbc425d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 0x55fdbc425d10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7fb00f0370 tx=0x7f7fb008d040 comp rx=0 tx=0).stop 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 shutdown_connections 2024-04-21T08:53:41.764 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fdbc42d670 0x55fdbc496330 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fdbc418320 0x55fdbc417160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fdbc409760 0x55fdbc416a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 --2- 172.21.15.57:0/4229597814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fdbc408da0 0x55fdbc425d10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 >> 172.21.15.57:0/4229597814 conn(0x55fdbc3d2c90 msgr2=0x55fdbc41c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7f9bfff640 0 client.26263 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 shutdown_connections 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 1 -- 172.21.15.57:0/4229597814 wait complete. 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.764+0000 7f7fc5c03740 20 client.26263 trim_cache size 0 max 16384 2024-04-21T08:53:41.765 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:53:41.766 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:53:41.766 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:53:41.766 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:53:41.766 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7fac0a3e40 msgr2=0x7f7fac0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7fac0a3e40 0x7f7fac0a6300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009fd0 tx=0x7f7fb4009de0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 msgr2=0x7f7fbc1986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1986f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0005940 tx=0x7f7fb002c5f0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 shutdown_connections 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7fac0a3e40 0x7f7fac0a6300 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7fbc19cfb0 0x7f7fbc19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fbc1072b0 0x7f7fbc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 --2- 172.21.15.57:0/1633045132 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fbc1021e0 0x7f7fbc1986f0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 >> 172.21.15.57:0/1633045132 conn(0x7f7fbc0fda00 msgr2=0x7f7fbc0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 shutdown_connections 2024-04-21T08:53:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:41.765+0000 7f7fc3011640 1 -- 172.21.15.57:0/1633045132 wait complete. 2024-04-21T08:53:42.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:42 smithi129 ceph-mon[31272]: pgmap v256: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 2.77k op/s 2024-04-21T08:53:42.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:42 smithi057 ceph-mon[27968]: pgmap v256: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 2.77k op/s 2024-04-21T08:53:42.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:42 smithi136 ceph-mon[31791]: pgmap v256: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.9 MiB/s wr, 2.77k op/s 2024-04-21T08:53:43.971 INFO:tasks.workunit.client.0.smithi057.stdout:# 2024-04-21T08:53:43.971 INFO:tasks.workunit.client.0.smithi057.stdout:# configuration written to .config 2024-04-21T08:53:43.971 INFO:tasks.workunit.client.0.smithi057.stdout:# 2024-04-21T08:53:43.973 INFO:tasks.workunit.client.0.smithi057.stderr:++ grep -c processor /proc/cpuinfo 2024-04-21T08:53:43.974 INFO:tasks.workunit.client.0.smithi057.stderr:+ make -j8 2024-04-21T08:53:44.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:44 smithi129 ceph-mon[31272]: pgmap v257: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 3.76k op/s 2024-04-21T08:53:44.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:44 smithi057 ceph-mon[27968]: pgmap v257: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 3.76k op/s 2024-04-21T08:53:44.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:44 smithi136 ceph-mon[31791]: pgmap v257: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.0 MiB/s wr, 3.76k op/s 2024-04-21T08:53:45.451 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/errno.h 2024-04-21T08:53:45.451 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/bpf_perf_event.h 2024-04-21T08:53:45.452 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/fcntl.h 2024-04-21T08:53:45.452 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctls.h 2024-04-21T08:53:45.452 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctl.h 2024-04-21T08:53:45.453 INFO:tasks.workunit.client.0.smithi057.stdout: GEN arch/x86/include/generated/asm/orc_hash.h 2024-04-21T08:53:45.478 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/ipcbuf.h 2024-04-21T08:53:45.481 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/param.h 2024-04-21T08:53:45.482 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/poll.h 2024-04-21T08:53:45.483 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/resource.h 2024-04-21T08:53:45.486 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/socket.h 2024-04-21T08:53:45.487 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/sockios.h 2024-04-21T08:53:45.489 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/termbits.h 2024-04-21T08:53:45.491 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/termios.h 2024-04-21T08:53:45.501 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/uapi/asm/types.h 2024-04-21T08:53:45.505 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC arch/x86/tools/relocs_32.o 2024-04-21T08:53:45.507 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC arch/x86/tools/relocs_64.o 2024-04-21T08:53:45.507 INFO:tasks.workunit.client.0.smithi057.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_32.h 2024-04-21T08:53:45.512 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/generated/uapi/linux/version.h 2024-04-21T08:53:45.517 INFO:tasks.workunit.client.0.smithi057.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h 2024-04-21T08:53:45.519 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/generated/compile.h 2024-04-21T08:53:45.523 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/config/kernel.release 2024-04-21T08:53:45.540 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/early_ioremap.h 2024-04-21T08:53:45.542 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/export.h 2024-04-21T08:53:45.545 INFO:tasks.workunit.client.0.smithi057.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h 2024-04-21T08:53:45.547 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/mcs_spinlock.h 2024-04-21T08:53:45.552 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/irq_regs.h 2024-04-21T08:53:45.559 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/kmap_size.h 2024-04-21T08:53:45.564 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/local64.h 2024-04-21T08:53:45.566 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/generated/utsrelease.h 2024-04-21T08:53:45.568 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/mmiowb.h 2024-04-21T08:53:45.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:45 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:45.575 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/module.lds.h 2024-04-21T08:53:45.578 INFO:tasks.workunit.client.0.smithi057.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_32.h 2024-04-21T08:53:45.581 INFO:tasks.workunit.client.0.smithi057.stdout: SYSHDR arch/x86/include/generated/asm/unistd_32_ia32.h 2024-04-21T08:53:45.582 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC arch/x86/tools/relocs_common.o 2024-04-21T08:53:45.585 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/rwonce.h 2024-04-21T08:53:45.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:45 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:45.591 INFO:tasks.workunit.client.0.smithi057.stdout: WRAP arch/x86/include/generated/asm/unaligned.h 2024-04-21T08:53:45.597 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/kallsyms 2024-04-21T08:53:45.612 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/sorttable 2024-04-21T08:53:45.615 INFO:tasks.workunit.client.0.smithi057.stdout: SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h 2024-04-21T08:53:45.624 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/selinux/genheaders/genheaders 2024-04-21T08:53:45.627 INFO:tasks.workunit.client.0.smithi057.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_64.h 2024-04-21T08:53:45.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:45 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:53:45.666 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/selinux/mdp/mdp 2024-04-21T08:53:45.787 INFO:tasks.workunit.client.0.smithi057.stdout: DESCEND objtool 2024-04-21T08:53:45.831 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/asn1_compiler 2024-04-21T08:53:45.913 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep.o 2024-04-21T08:53:45.944 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTLD arch/x86/tools/relocs 2024-04-21T08:53:45.985 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTLD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep-in.o 2024-04-21T08:53:45.991 INFO:tasks.workunit.client.0.smithi057.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep 2024-04-21T08:53:46.032 INFO:tasks.workunit.client.0.smithi057.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/exec-cmd.h 2024-04-21T08:53:46.033 INFO:tasks.workunit.client.0.smithi057.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/help.h 2024-04-21T08:53:46.033 INFO:tasks.workunit.client.0.smithi057.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/pager.h 2024-04-21T08:53:46.033 INFO:tasks.workunit.client.0.smithi057.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/parse-options.h 2024-04-21T08:53:46.035 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/exec-cmd.o 2024-04-21T08:53:46.046 INFO:tasks.workunit.client.0.smithi057.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/run-command.h 2024-04-21T08:53:46.046 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/help.o 2024-04-21T08:53:46.047 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/pager.o 2024-04-21T08:53:46.048 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/parse-options.o 2024-04-21T08:53:46.052 INFO:tasks.workunit.client.0.smithi057.stdout: INSTALL libsubcmd_headers 2024-04-21T08:53:46.055 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/run-command.o 2024-04-21T08:53:46.207 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/sigchain.o 2024-04-21T08:53:46.251 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/subcmd-config.o 2024-04-21T08:53:46.479 INFO:tasks.workunit.client.0.smithi057.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/libsubcmd-in.o 2024-04-21T08:53:46.494 INFO:tasks.workunit.client.0.smithi057.stdout: AR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/libsubcmd.a 2024-04-21T08:53:46.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:46 smithi129 ceph-mon[31272]: pgmap v258: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 908 KiB/s wr, 3.54k op/s 2024-04-21T08:53:46.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:46 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:53:46.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:46 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:53:46.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:46 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:53:46.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:46 smithi129 ceph-mon[31272]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:53:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:46 smithi057 ceph-mon[27968]: pgmap v258: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 908 KiB/s wr, 3.54k op/s 2024-04-21T08:53:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:46 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:53:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:46 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:53:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:46 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:53:46.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:46 smithi057 ceph-mon[27968]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:53:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:46 smithi136 ceph-mon[31791]: pgmap v258: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 908 KiB/s wr, 3.54k op/s 2024-04-21T08:53:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:46 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:53:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:46 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:53:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:46 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:53:46.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:46 smithi136 ceph-mon[31791]: osdmap e100: 12 total, 12 up, 12 in 2024-04-21T08:53:47.420 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/weak.o 2024-04-21T08:53:47.420 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/check.o 2024-04-21T08:53:47.420 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/special.o 2024-04-21T08:53:47.421 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/builtin-check.o 2024-04-21T08:53:47.422 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/elf.o 2024-04-21T08:53:47.422 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool.o 2024-04-21T08:53:47.426 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/special.o 2024-04-21T08:53:47.503 INFO:tasks.workunit.client.0.smithi057.stdout: MKDIR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/lib/ 2024-04-21T08:53:47.510 INFO:tasks.workunit.client.0.smithi057.stdout: GEN /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/lib/inat-tables.c 2024-04-21T08:53:47.517 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/orc_gen.o 2024-04-21T08:53:47.524 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/orc_dump.o 2024-04-21T08:53:47.534 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/decode.o 2024-04-21T08:53:47.558 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libstring.o 2024-04-21T08:53:47.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:47 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:53:47.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:47 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:53:47.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:47 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:53:47.660 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libctype.o 2024-04-21T08:53:47.674 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/str_error_r.o 2024-04-21T08:53:47.675 INFO:tasks.workunit.client.0.smithi057.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/librbtree.o 2024-04-21T08:53:48.070 INFO:tasks.workunit.client.0.smithi057.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/objtool-in.o 2024-04-21T08:53:48.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:48 smithi129 ceph-mon[31272]: pgmap v260: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 966 KiB/s wr, 3.53k op/s 2024-04-21T08:53:48.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:48 smithi057 ceph-mon[27968]: pgmap v260: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 966 KiB/s wr, 3.53k op/s 2024-04-21T08:53:48.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:48 smithi136 ceph-mon[31791]: pgmap v260: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 966 KiB/s wr, 3.53k op/s 2024-04-21T08:53:49.255 INFO:tasks.workunit.client.0.smithi057.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool-in.o 2024-04-21T08:53:49.271 INFO:tasks.workunit.client.0.smithi057.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool 2024-04-21T08:53:50.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:50 smithi129 ceph-mon[31272]: pgmap v261: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 2.51k op/s 2024-04-21T08:53:50.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:50 smithi057 ceph-mon[27968]: pgmap v261: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 2.51k op/s 2024-04-21T08:53:50.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:50 smithi136 ceph-mon[31791]: pgmap v261: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.1 MiB/s wr, 2.51k op/s 2024-04-21T08:53:52.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:52 smithi129 ceph-mon[31272]: pgmap v262: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.77k op/s 2024-04-21T08:53:52.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:52 smithi057 ceph-mon[27968]: pgmap v262: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.77k op/s 2024-04-21T08:53:52.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:52 smithi136 ceph-mon[31791]: pgmap v262: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 1.77k op/s 2024-04-21T08:53:54.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:54 smithi129 ceph-mon[31272]: pgmap v263: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 646 op/s 2024-04-21T08:53:54.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:54 smithi057 ceph-mon[27968]: pgmap v263: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 646 op/s 2024-04-21T08:53:54.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:54 smithi136 ceph-mon[31791]: pgmap v263: 129 pgs: 129 active+clean; 1.4 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 646 op/s 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.861+0000 7f8086414640 1 -- 172.21.15.57:0/2229656424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 msgr2=0x7f8080102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.861+0000 7f8086414640 1 --2- 172.21.15.57:0/2229656424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080102580 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f8070009900 tx=0x7f807002bc30 comp rx=0 tx=0).stop 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 -- 172.21.15.57:0/2229656424 shutdown_connections 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 --2- 172.21.15.57:0/2229656424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8080107270 0x7f808010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 --2- 172.21.15.57:0/2229656424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8080102b50 0x7f8080106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 --2- 172.21.15.57:0/2229656424 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080102580 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 -- 172.21.15.57:0/2229656424 >> 172.21.15.57:0/2229656424 conn(0x7f80800fda00 msgr2=0x7f80800ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 -- 172.21.15.57:0/2229656424 shutdown_connections 2024-04-21T08:53:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 -- 172.21.15.57:0/2229656424 wait complete. 2024-04-21T08:53:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 Processor -- start 2024-04-21T08:53:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.862+0000 7f8086414640 1 -- start start 2024-04-21T08:53:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f8086414640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f8086414640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8080107270 0x7f8080074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f8086414640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8080074cd0 0x7f80801a6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f8086414640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f80801121e0 con 0x7f80801021a0 2024-04-21T08:53:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f8086414640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112060 con 0x7f8080074cd0 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f8086414640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8080112360 con 0x7f8080107270 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56960/0 (socket says 172.21.15.57:56960) 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 -- 172.21.15.57:0/1424652633 learned_addr learned my addr 172.21.15.57:0/1424652633 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f80849b1640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8080074cd0 0x7f80801a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807f7fe640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8080107270 0x7f8080074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 -- 172.21.15.57:0/1424652633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8080107270 msgr2=0x7f8080074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8080107270 0x7f8080074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 -- 172.21.15.57:0/1424652633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8080074cd0 msgr2=0x7f80801a6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8080074cd0 0x7f80801a6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 -- 172.21.15.57:0/1424652633 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80801a6fd0 con 0x7f80801021a0 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.863+0000 7f807ffff640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080074250 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8070005940 tx=0x7f80700045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.864+0000 7f807d7fa640 1 -- 172.21.15.57:0/1424652633 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070043020 con 0x7f80801021a0 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.864+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80801a7260 con 0x7f80801021a0 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.864+0000 7f807d7fa640 1 -- 172.21.15.57:0/1424652633 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f807002caf0 con 0x7f80801021a0 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.864+0000 7f807d7fa640 1 -- 172.21.15.57:0/1424652633 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070003af0 con 0x7f80801021a0 2024-04-21T08:53:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.864+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a7800 con 0x7f80801021a0 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.867+0000 7f807d7fa640 1 -- 172.21.15.57:0/1424652633 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8070046020 con 0x7f80801021a0 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.867+0000 7f8089007740 1 Processor -- start 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.867+0000 7f8089007740 1 -- start start 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f8089007740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57d8d0 0x55d75a57c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f8089007740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57cb60 0x55d75a57cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f8089007740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a5860e0 0x55d75a5884d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.875 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f8089007740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55d75a4e18d0 con 0x55d75a57d8d0 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f8089007740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d75a47ee80 con 0x55d75a57cb60 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f8089007740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d75a4dd890 con 0x55d75a5860e0 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f80849b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57d8d0 0x55d75a57c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807d7fa640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80680a3dc0 0x7f80680a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57cb60 0x55d75a57cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57cb60 0x55d75a57cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:33708/0 (socket says 172.21.15.57:33708) 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807ffff640 1 -- 172.21.15.57:0/1474398443 learned_addr learned my addr 172.21.15.57:0/1474398443 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:54.876 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f80849b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57d8d0 0x55d75a57c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56968/0 (socket says 172.21.15.57:56968) 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a5860e0 0x55d75a5884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f80849b1640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80680a3dc0 0x7f80680a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807d7fa640 1 -- 172.21.15.57:0/1424652633 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9475+0+0 (secure 0 0 0) 0x7f80700f2bd0 con 0x7f80801021a0 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 -- 172.21.15.57:0/1474398443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57cb60 msgr2=0x55d75a57cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57cb60 0x55d75a57cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 -- 172.21.15.57:0/1474398443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57d8d0 msgr2=0x55d75a57c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57d8d0 0x55d75a57c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.877 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 -- 172.21.15.57:0/1474398443 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75a4a0320 con 0x55d75a5860e0 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f80849b1640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80680a3dc0 0x7f80680a6280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8078005fd0 tx=0x7f8078009ad0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.868+0000 7f807f7fe640 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a5860e0 0x55d75a5884d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f808006b5d0 tx=0x7f8074011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3823fdaea6729 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.869+0000 7f80667fc640 1 -- 172.21.15.57:0/1474398443 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074004310 con 0x55d75a5860e0 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.869+0000 7f80667fc640 1 -- 172.21.15.57:0/1474398443 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80740044b0 con 0x55d75a5860e0 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.869+0000 7f80667fc640 1 -- 172.21.15.57:0/1474398443 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80740035f0 con 0x55d75a5860e0 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 -- 172.21.15.57:0/1474398443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a5860e0 msgr2=0x55d75a5884d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a5860e0 0x55d75a5884d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f808006b5d0 tx=0x7f8074011440 comp rx=0 tx=0).stop 2024-04-21T08:53:54.878 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 -- 172.21.15.57:0/1474398443 shutdown_connections 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a5860e0 0x55d75a5884d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57cb60 0x55d75a57cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 --2- 172.21.15.57:0/1474398443 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57d8d0 0x55d75a57c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 -- 172.21.15.57:0/1474398443 >> 172.21.15.57:0/1474398443 conn(0x55d75a543c90 msgr2=0x55d75a577850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 -- 172.21.15.57:0/1474398443 shutdown_connections 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 -- 172.21.15.57:0/1474398443 wait complete. 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 -- start start 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.870+0000 7f8089007740 1 Processor -- start 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 0x55d75a5859c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.879 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a582d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a5860e0 0x55d75a583290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55d75a588e10 con 0x55d75a5860e0 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d75a469eb0 con 0x55d75a57d8d0 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55d75a588f90 con 0x55d75a57cb60 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f8089007740 10 client.0 _ll_register_callbacks cb 0x55d75a496b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 0x55d75a5859c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f80849b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a5860e0 0x55d75a583290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 0x55d75a5859c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47352/0 (socket says 172.21.15.57:47352) 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f80849b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a5860e0 0x55d75a583290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56984/0 (socket says 172.21.15.57:56984) 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a582d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f80849b1640 1 -- 172.21.15.57:0/4001132113 learned_addr learned my addr 172.21.15.57:0/4001132113 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:54.880 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 -- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 msgr2=0x55d75a582d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a582d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 -- 172.21.15.57:0/4001132113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a5860e0 msgr2=0x55d75a583290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a5860e0 0x55d75a583290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 -- 172.21.15.57:0/4001132113 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75a1f4210 con 0x55d75a57cb60 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f807ffff640 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 0x55d75a5859c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55d75a57e030 tx=0x7f807002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e0752a82d9eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.871+0000 7f805f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f805f7fe640 1 -- 172.21.15.57:0/4001132113 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807002cca0 con 0x55d75a57cb60 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80740044b0 con 0x55d75a57cb60 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f805f7fe640 1 -- 172.21.15.57:0/4001132113 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9475+0+0 (secure 0 0 0) 0x7f8070003c90 con 0x55d75a57cb60 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f805f7fe640 1 -- 172.21.15.57:0/4001132113 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f80700befc0 con 0x55d75a57cb60 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 10 client.26275 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:54.881 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d75a2c0620 con 0x55d75a57cb60 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f805f7fe640 1 -- 172.21.15.57:0/4001132113 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80740044b0 con 0x55d75a57cb60 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f805f7fe640 1 -- 172.21.15.57:0/4001132113 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f80700053b0 con 0x55d75a57cb60 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 10 client.26275 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 10 client.26275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 20 client.26275 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a596020 0x55d75a598490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 4 client.26275 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d75a4f5880 con 0x55d75a596020 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.872+0000 7f807f7fe640 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a596020 0x55d75a598490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f805f7fe640 10 client.26275 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:54.882 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f807f7fe640 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a596020 0x55d75a598490 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4c0f3ae5c9f175 server_cookie=384a3797bfe796fa in_seq=0 out_seq=0 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f805f7fe640 1 -- 172.21.15.57:0/4001132113 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d75a4f5880 con 0x55d75a596020 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f805f7fe640 10 client.26275 handle_command_reply: tid=0 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f8089007740 1 client.26275 shutdown 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 msgr2=0x55d75a5859c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.873+0000 7f8089007740 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 0x55d75a5859c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55d75a57e030 tx=0x7f807002bc30 comp rx=0 tx=0).stop 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 shutdown_connections 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a596020 0x55d75a598490 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a5860e0 0x55d75a583290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a582d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f805f7fe640 0 client.26275 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 --2- 172.21.15.57:0/4001132113 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57cb60 0x55d75a5859c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 >> 172.21.15.57:0/4001132113 conn(0x55d75a543c90 msgr2=0x55d75a5795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 shutdown_connections 2024-04-21T08:53:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 1 -- 172.21.15.57:0/4001132113 wait complete. 2024-04-21T08:53:54.884 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.874+0000 7f8089007740 20 client.26275 trim_cache size 0 max 16384 2024-04-21T08:53:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 Processor -- start 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 -- start start 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a543c90 0x55d75a591d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a572760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a585480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f80801af520 con 0x55d75a57cb60 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b1d80 con 0x55d75a57d8d0 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.890+0000 7f8089007740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f80801afb70 con 0x55d75a543c90 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a572760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a572760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56996/0 (socket says 172.21.15.57:56996) 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 -- 172.21.15.57:0/2947671973 learned_addr learned my addr 172.21.15.57:0/2947671973 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:54.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a543c90 0x55d75a591d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f80849b1640 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a585480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 -- 172.21.15.57:0/2947671973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a543c90 msgr2=0x55d75a591d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a543c90 0x55d75a591d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 -- 172.21.15.57:0/2947671973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 msgr2=0x55d75a585480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a585480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 -- 172.21.15.57:0/2947671973 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d75a572f30 con 0x55d75a57cb60 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807ffff640 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a543c90 0x55d75a591d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f807f7fe640 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a572760 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x55d75a57d600 tx=0x7f8074004050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0262d15541e7f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f8065ffb640 1 -- 172.21.15.57:0/2947671973 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074030070 con 0x55d75a57cb60 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f8065ffb640 1 -- 172.21.15.57:0/2947671973 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8074029030 con 0x55d75a57cb60 2024-04-21T08:53:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.891+0000 7f8065ffb640 1 -- 172.21.15.57:0/2947671973 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807402a5b0 con 0x55d75a57cb60 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 -- 172.21.15.57:0/2947671973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 msgr2=0x55d75a572760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a572760 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x55d75a57d600 tx=0x7f8074004050 comp rx=0 tx=0).stop 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 -- 172.21.15.57:0/2947671973 shutdown_connections 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a57d8d0 0x55d75a585480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a572760 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 --2- 172.21.15.57:0/2947671973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a543c90 0x55d75a591d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 -- 172.21.15.57:0/2947671973 >> 172.21.15.57:0/2947671973 conn(0x55d75a57e740 msgr2=0x55d75a57c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 -- 172.21.15.57:0/2947671973 shutdown_connections 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 -- 172.21.15.57:0/2947671973 wait complete. 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.892+0000 7f8089007740 1 -- start start 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 Processor -- start 2024-04-21T08:53:54.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a543c90 0x55d75a58e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a57ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57d8d0 0x55d75a580510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b0240 con 0x55d75a57cb60 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b16b0 con 0x55d75a543c90 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f80801b0910 con 0x55d75a57d8d0 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f8089007740 10 client.0 _ll_register_callbacks cb 0x7f80700f0010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a57ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f80849b1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57d8d0 0x55d75a580510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a57ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:57004/0 (socket says 172.21.15.57:57004) 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 -- 172.21.15.57:0/3696719836 learned_addr learned my addr 172.21.15.57:0/3696719836 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807ffff640 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a543c90 0x55d75a58e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 -- 172.21.15.57:0/3696719836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57d8d0 msgr2=0x55d75a580510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57d8d0 0x55d75a580510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 -- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a543c90 msgr2=0x55d75a58e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a543c90 0x55d75a58e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 -- 172.21.15.57:0/3696719836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d75a580d10 con 0x55d75a57cb60 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f807f7fe640 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a57ffd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f807402e040 tx=0x7f8074011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef287b18d8ec4b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f805f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.893+0000 7f805f7fe640 1 -- 172.21.15.57:0/3696719836 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074030070 con 0x55d75a57cb60 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f805f7fe640 1 -- 172.21.15.57:0/3696719836 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9475+0+0 (secure 0 0 0) 0x7f8074011040 con 0x55d75a57cb60 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8074029030 con 0x55d75a57cb60 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f805f7fe640 1 -- 172.21.15.57:0/3696719836 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8074011240 con 0x55d75a57cb60 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 10 client.16935 fetch_fsmap learned FSMap version 21 2024-04-21T08:53:54.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55d75a5fc610 con 0x55d75a57cb60 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f805f7fe640 1 -- 172.21.15.57:0/3696719836 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8074029030 con 0x55d75a57cb60 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f805f7fe640 1 -- 172.21.15.57:0/3696719836 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8074018070 con 0x55d75a57cb60 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 10 client.16935 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 20 client.16935 populate_metadata read hostname 'smithi057' 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a5860e0 0x55d75a5feb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 4 client.16935 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d75a4813d0 con 0x55d75a5860e0 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.894+0000 7f807ffff640 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a5860e0 0x55d75a5feb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.895+0000 7f807ffff640 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a5860e0 0x55d75a5feb20 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743410a73b94465c server_cookie=9d2b5c7d9edfa32f in_seq=0 out_seq=0 2024-04-21T08:53:54.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:54.895+0000 7f805f7fe640 10 client.16935 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:56.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:56 smithi129 ceph-mon[31272]: pgmap v264: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 387 op/s 2024-04-21T08:53:56.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:56 smithi057 ceph-mon[27968]: pgmap v264: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 387 op/s 2024-04-21T08:53:56.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:56 smithi136 ceph-mon[31791]: pgmap v264: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 387 op/s 2024-04-21T08:53:57.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f805f7fe640 1 -- 172.21.15.57:0/3696719836 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d75a4813d0 con 0x55d75a5860e0 2024-04-21T08:53:57.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f805f7fe640 10 client.16935 handle_command_reply: tid=0 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 client.16935 shutdown 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 msgr2=0x55d75a57ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a57ffd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f807402e040 tx=0x7f8074011a60 comp rx=0 tx=0).stop 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 shutdown_connections 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55d75a5860e0 0x55d75a5feb20 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55d75a57d8d0 0x55d75a580510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55d75a57cb60 0x55d75a57ffd0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 --2- 172.21.15.57:0/3696719836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d75a543c90 0x55d75a58e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f805f7fe640 0 client.16935 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:53:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 >> 172.21.15.57:0/3696719836 conn(0x55d75a57e740 msgr2=0x55d75a5815e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 shutdown_connections 2024-04-21T08:53:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 1 -- 172.21.15.57:0/3696719836 wait complete. 2024-04-21T08:53:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.083+0000 7f8089007740 20 client.16935 trim_cache size 0 max 16384 2024-04-21T08:53:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:53:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:53:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:53:57.086 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:53:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80680a3dc0 msgr2=0x7f80680a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80680a3dc0 0x7f80680a6280 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8078005fd0 tx=0x7f8078009ad0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 msgr2=0x7f8080074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:53:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080074250 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8070005940 tx=0x7f80700045e0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.088 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 shutdown_connections 2024-04-21T08:53:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f80680a3dc0 0x7f80680a6280 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8080074cd0 0x7f80801a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8080107270 0x7f8080074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 --2- 172.21.15.57:0/1424652633 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80801021a0 0x7f8080074250 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:53:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 >> 172.21.15.57:0/1424652633 conn(0x7f80800fda00 msgr2=0x7f80800fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:53:57.089 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 shutdown_connections 2024-04-21T08:53:57.090 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:53:57.084+0000 7f8086414640 1 -- 172.21.15.57:0/1424652633 wait complete. 2024-04-21T08:53:58.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:53:58 smithi129 ceph-mon[31272]: pgmap v265: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 548 op/s 2024-04-21T08:53:58.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:53:58 smithi057 ceph-mon[27968]: pgmap v265: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 548 op/s 2024-04-21T08:53:58.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:53:58 smithi136 ceph-mon[31791]: pgmap v265: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 548 op/s 2024-04-21T08:54:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:00 smithi129 ceph-mon[31272]: pgmap v266: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 821 op/s 2024-04-21T08:54:00.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:00 smithi057 ceph-mon[27968]: pgmap v266: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 821 op/s 2024-04-21T08:54:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:00 smithi136 ceph-mon[31791]: pgmap v266: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 821 op/s 2024-04-21T08:54:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:02 smithi129 ceph-mon[31272]: pgmap v267: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.1 MiB/s wr, 778 op/s 2024-04-21T08:54:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:02 smithi057 ceph-mon[27968]: pgmap v267: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.1 MiB/s wr, 778 op/s 2024-04-21T08:54:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:02 smithi136 ceph-mon[31791]: pgmap v267: 129 pgs: 129 active+clean; 1.4 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.1 MiB/s wr, 778 op/s 2024-04-21T08:54:03.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:03 smithi129 ceph-mon[31272]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:54:03.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:03 smithi057 ceph-mon[27968]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:54:03.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:03 smithi136 ceph-mon[31791]: osdmap e101: 12 total, 12 up, 12 in 2024-04-21T08:54:04.133 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:04.133 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:04.334 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.334+0000 7f2a358c1640 1 -- 172.21.15.57:0/3253033323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30106150 msgr2=0x7f2a3010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.334 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.334+0000 7f2a358c1640 1 --2- 172.21.15.57:0/3253033323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30106150 0x7f2a3010dd20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009be0 tx=0x7f2a2802bf30 comp rx=0 tx=0).stop 2024-04-21T08:54:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.336+0000 7f2a358c1640 1 -- 172.21.15.57:0/3253033323 shutdown_connections 2024-04-21T08:54:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.336+0000 7f2a358c1640 1 --2- 172.21.15.57:0/3253033323 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30106150 0x7f2a3010dd20 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.336+0000 7f2a358c1640 1 --2- 172.21.15.57:0/3253033323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30101a30 0x7f2a30105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.336+0000 7f2a358c1640 1 --2- 172.21.15.57:0/3253033323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a30101080 0x7f2a30101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.338 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.336+0000 7f2a358c1640 1 -- 172.21.15.57:0/3253033323 >> 172.21.15.57:0/3253033323 conn(0x7f2a30078000 msgr2=0x7f2a30078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.340+0000 7f2a358c1640 1 -- 172.21.15.57:0/3253033323 shutdown_connections 2024-04-21T08:54:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.340+0000 7f2a358c1640 1 -- 172.21.15.57:0/3253033323 wait complete. 2024-04-21T08:54:04.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.341+0000 7f2a358c1640 1 Processor -- start 2024-04-21T08:54:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.349+0000 7f2a358c1640 1 -- start start 2024-04-21T08:54:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.350+0000 7f2a358c1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a30101080 0x7f2a3010c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.351 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.350+0000 7f2a358c1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 0x7f2a3010cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.350+0000 7f2a358c1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30117260 0x7f2a301176e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.350+0000 7f2a358c1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010fa10 con 0x7f2a30117260 2024-04-21T08:54:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.350+0000 7f2a358c1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010f890 con 0x7f2a30101080 2024-04-21T08:54:04.352 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.350+0000 7f2a358c1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3010fb90 con 0x7f2a30106150 2024-04-21T08:54:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.351+0000 7f2a2e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 0x7f2a3010cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.352+0000 7f2a2e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 0x7f2a3010cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57586/0 (socket says 172.21.15.57:57586) 2024-04-21T08:54:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.352+0000 7f2a2e7fc640 1 -- 172.21.15.57:0/1867835383 learned_addr learned my addr 172.21.15.57:0/1867835383 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2f7fe640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30117260 0x7f2a301176e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2effd640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a30101080 0x7f2a3010c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2e7fc640 1 -- 172.21.15.57:0/1867835383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a30101080 msgr2=0x7f2a3010c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a30101080 0x7f2a3010c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2e7fc640 1 -- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30117260 msgr2=0x7f2a301176e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30117260 0x7f2a301176e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2e7fc640 1 -- 172.21.15.57:0/1867835383 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a30117d20 con 0x7f2a30106150 2024-04-21T08:54:04.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.353+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 0x7f2a3010cd60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800d770 tx=0x7f2a1800dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.354+0000 7f2a348bf640 1 -- 172.21.15.57:0/1867835383 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a18017070 con 0x7f2a30106150 2024-04-21T08:54:04.358 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.354+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a30118010 con 0x7f2a30106150 2024-04-21T08:54:04.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.354+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3011a670 con 0x7f2a30106150 2024-04-21T08:54:04.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.354+0000 7f2a348bf640 1 -- 172.21.15.57:0/1867835383 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a18009c30 con 0x7f2a30106150 2024-04-21T08:54:04.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.354+0000 7f2a348bf640 1 -- 172.21.15.57:0/1867835383 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1800f620 con 0x7f2a30106150 2024-04-21T08:54:04.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.358+0000 7f2a348bf640 1 -- 172.21.15.57:0/1867835383 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2a1801a070 con 0x7f2a30106150 2024-04-21T08:54:04.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.359+0000 7f2a348bf640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f2a1c0a3e60 0x7f2a1c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.359+0000 7f2a348bf640 1 -- 172.21.15.57:0/1867835383 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f2a180d15e0 con 0x7f2a30106150 2024-04-21T08:54:04.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 Processor -- start 2024-04-21T08:54:04.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 -- start start 2024-04-21T08:54:04.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcd580 0x55c796fcd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c796fd6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55c796f321e0 con 0x55c796fcdea0 2024-04-21T08:54:04.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c796ecf390 con 0x55c796fd6a80 2024-04-21T08:54:04.361 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.360+0000 7f2a3858d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55c796f2e1a0 con 0x55c796fcd580 2024-04-21T08:54:04.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.362+0000 7f2a2f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.362+0000 7f2a2f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53620/0 (socket says 172.21.15.57:53620) 2024-04-21T08:54:04.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.362+0000 7f2a2f7fe640 1 -- 172.21.15.57:0/1396362731 learned_addr learned my addr 172.21.15.57:0/1396362731 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:04.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.362+0000 7f2a2effd640 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcd580 0x55c796fcd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.362+0000 7f2a2effd640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f2a1c0a3e60 0x7f2a1c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.362+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c796fd6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.364+0000 7f2a2f7fe640 1 -- 172.21.15.57:0/1396362731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcd580 msgr2=0x55c796fcd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.364+0000 7f2a2f7fe640 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcd580 0x55c796fcd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.364+0000 7f2a2f7fe640 1 -- 172.21.15.57:0/1396362731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 msgr2=0x55c796fd6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.364+0000 7f2a2f7fe640 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c796fd6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.364+0000 7f2a2f7fe640 1 -- 172.21.15.57:0/1396362731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c796d0e7d0 con 0x55c796fd6a80 2024-04-21T08:54:04.365 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.364+0000 7f2a2f7fe640 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd8e70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009be0 tx=0x7f2a28009940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861c92bee35c3bac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.367+0000 7f2a25ffb640 1 -- 172.21.15.57:0/1396362731 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28043070 con 0x55c796fd6a80 2024-04-21T08:54:04.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.367+0000 7f2a25ffb640 1 -- 172.21.15.57:0/1396362731 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a280333e0 con 0x55c796fd6a80 2024-04-21T08:54:04.367 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.367+0000 7f2a25ffb640 1 -- 172.21.15.57:0/1396362731 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2802c8d0 con 0x55c796fd6a80 2024-04-21T08:54:04.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.368+0000 7f2a3858d740 1 -- 172.21.15.57:0/1396362731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 msgr2=0x55c796fd8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.368+0000 7f2a3858d740 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd8e70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009be0 tx=0x7f2a28009940 comp rx=0 tx=0).stop 2024-04-21T08:54:04.370 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.370+0000 7f2a3858d740 1 -- 172.21.15.57:0/1396362731 shutdown_connections 2024-04-21T08:54:04.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.370+0000 7f2a3858d740 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd8e70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.370+0000 7f2a3858d740 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c796fd6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.370+0000 7f2a3858d740 1 --2- 172.21.15.57:0/1396362731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcd580 0x55c796fcd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.371 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.370+0000 7f2a3858d740 1 -- 172.21.15.57:0/1396362731 >> 172.21.15.57:0/1396362731 conn(0x55c796fc7cb0 msgr2=0x55c796fc8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.374+0000 7f2a2effd640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f2a1c0a3e60 0x7f2a1c0a6320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2a20005ab0 tx=0x7f2a20009260 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.379+0000 7f2a3858d740 1 -- 172.21.15.57:0/1396362731 shutdown_connections 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.386+0000 7f2a3858d740 1 -- 172.21.15.57:0/1396362731 wait complete. 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.386+0000 7f2a3858d740 1 -- start start 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.386+0000 7f2a3858d740 1 Processor -- start 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.387+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 0x55c796fd39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.387+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcdea0 0x55c796fd3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.387+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.387+0000 7f2a3858d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55c796fd9860 con 0x55c796fcd580 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.387+0000 7f2a3858d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c796eeb2f0 con 0x55c796fd6a80 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.387+0000 7f2a3858d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55c796fd99e0 con 0x55c796fcdea0 2024-04-21T08:54:04.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.389+0000 7f2a3858d740 10 client.0 _ll_register_callbacks cb 0x55c796ef8580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:04.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.389+0000 7f2a2effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 0x55c796fd39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.393+0000 7f2a2effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 0x55c796fd39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50134/0 (socket says 172.21.15.57:50134) 2024-04-21T08:54:04.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.393+0000 7f2a2effd640 1 -- 172.21.15.57:0/2129392279 learned_addr learned my addr 172.21.15.57:0/2129392279 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:04.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.389+0000 7f2a2f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.395 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.396+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcdea0 0x55c796fd3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.397+0000 7f2a2effd640 1 -- 172.21.15.57:0/2129392279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcdea0 msgr2=0x55c796fd3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.397+0000 7f2a2effd640 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcdea0 0x55c796fd3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.397+0000 7f2a2effd640 1 -- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 msgr2=0x55c796fd4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.397+0000 7f2a2effd640 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.397+0000 7f2a2effd640 1 -- 172.21.15.57:0/2129392279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c796c443f0 con 0x55c796fcd580 2024-04-21T08:54:04.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.399+0000 7f2a2effd640 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 0x55c796fd39a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d70 tx=0x7f2a20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735c3c9b0bae9461 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.399+0000 7f2a0effd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:04.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.400+0000 7f2a0effd640 1 -- 172.21.15.57:0/2129392279 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a20004320 con 0x55c796fcd580 2024-04-21T08:54:04.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.401+0000 7f2a0effd640 1 -- 172.21.15.57:0/2129392279 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f2a200035f0 con 0x55c796fcd580 2024-04-21T08:54:04.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.401+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a280333e0 con 0x55c796fcd580 2024-04-21T08:54:04.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.401+0000 7f2a0effd640 1 -- 172.21.15.57:0/2129392279 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2a200044c0 con 0x55c796fcd580 2024-04-21T08:54:04.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.401+0000 7f2a3858d740 10 client.16947 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:04.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.401+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c796c040f0 con 0x55c796fcd580 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a0effd640 1 -- 172.21.15.57:0/2129392279 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a280333e0 con 0x55c796fcd580 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a0effd640 1 -- 172.21.15.57:0/2129392279 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2a20026020 con 0x55c796fcd580 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a3858d740 10 client.16947 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a3858d740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a3858d740 20 client.16947 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797008a10 0x55c79700ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a3858d740 4 client.16947 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:04.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.402+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c796f3fc00 con 0x55c797008a10 2024-04-21T08:54:04.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.405+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797008a10 0x55c79700ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.410 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.411+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797008a10 0x55c79700ae90 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c5f005ff0b1ece server_cookie=33cf962233d814d in_seq=0 out_seq=0 2024-04-21T08:54:04.410 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.411+0000 7f2a0effd640 10 client.16947 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:04.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.415+0000 7f2a0effd640 1 -- 172.21.15.57:0/2129392279 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c796f3fc00 con 0x55c797008a10 2024-04-21T08:54:04.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.415+0000 7f2a0effd640 10 client.16947 handle_command_reply: tid=0 2024-04-21T08:54:04.416 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.416+0000 7f2a3858d740 1 client.16947 shutdown 2024-04-21T08:54:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.424+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 msgr2=0x55c796fd39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.424+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 0x55c796fd39a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d70 tx=0x7f2a20004730 comp rx=0 tx=0).stop 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.425+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 shutdown_connections 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.425+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797008a10 0x55c79700ae90 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.425+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fd6a80 0x55c796fd4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.425+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fcdea0 0x55c796fd3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.425+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2129392279 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcd580 0x55c796fd39a0 secure :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d70 tx=0x7f2a20004730 comp rx=0 tx=0).stop 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.425+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 >> 172.21.15.57:0/2129392279 conn(0x55c796fc7cb0 msgr2=0x55c796fc9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.426+0000 7f2a0effd640 0 client.16947 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:04.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.433+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 shutdown_connections 2024-04-21T08:54:04.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.436+0000 7f2a3858d740 1 -- 172.21.15.57:0/2129392279 wait complete. 2024-04-21T08:54:04.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.440+0000 7f2a3858d740 20 client.16947 trim_cache size 0 max 16384 2024-04-21T08:54:04.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.477+0000 7f2a3858d740 1 Processor -- start 2024-04-21T08:54:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 -- start start 2024-04-21T08:54:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 0x55c796fd60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fe2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.479 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fc97f0 0x55c7970107a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a2effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 0x55c796fd60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a2effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 0x55c796fd60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53632/0 (socket says 172.21.15.57:53632) 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a2effd640 1 -- 172.21.15.57:0/3740385104 learned_addr learned my addr 172.21.15.57:0/3740385104 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1000e140 con 0x55c796fc97f0 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a100109a0 con 0x55c796fcdea0 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1000e790 con 0x55c796fd6a80 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.479+0000 7f2a2f7fe640 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fc97f0 0x55c7970107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.480+0000 7f2a2effd640 1 -- 172.21.15.57:0/3740385104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 msgr2=0x55c796fe2780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.480+0000 7f2a2effd640 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fe2780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.480+0000 7f2a2effd640 1 -- 172.21.15.57:0/3740385104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fc97f0 msgr2=0x55c7970107a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.480+0000 7f2a2effd640 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fc97f0 0x55c7970107a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.480+0000 7f2a2effd640 1 -- 172.21.15.57:0/3740385104 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c796fd68b0 con 0x55c796fcdea0 2024-04-21T08:54:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.480+0000 7f2a2effd640 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 0x55c796fd60d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2a2001fab0 tx=0x7f2a2000d750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa0d4c79f635af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.481+0000 7f2a257fa640 1 -- 172.21.15.57:0/3740385104 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2003a070 con 0x55c796fcdea0 2024-04-21T08:54:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.481+0000 7f2a257fa640 1 -- 172.21.15.57:0/3740385104 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a200035f0 con 0x55c796fcdea0 2024-04-21T08:54:04.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.481+0000 7f2a257fa640 1 -- 172.21.15.57:0/3740385104 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2001d780 con 0x55c796fcdea0 2024-04-21T08:54:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.483+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 msgr2=0x55c796fd60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.483+0000 7f2a3858d740 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 0x55c796fd60d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2a2001fab0 tx=0x7f2a2000d750 comp rx=0 tx=0).stop 2024-04-21T08:54:04.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.485+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 shutdown_connections 2024-04-21T08:54:04.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.485+0000 7f2a3858d740 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fc97f0 0x55c7970107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.485+0000 7f2a3858d740 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fe2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.485+0000 7f2a3858d740 1 --2- 172.21.15.57:0/3740385104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fcdea0 0x55c796fd60d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.485+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 >> 172.21.15.57:0/3740385104 conn(0x55c796fc7cb0 msgr2=0x55c796fc90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.487+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 shutdown_connections 2024-04-21T08:54:04.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.487+0000 7f2a3858d740 1 -- 172.21.15.57:0/3740385104 wait complete. 2024-04-21T08:54:04.487 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.488+0000 7f2a3858d740 1 -- start start 2024-04-21T08:54:04.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.488+0000 7f2a3858d740 1 Processor -- start 2024-04-21T08:54:04.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.489+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 0x55c797068390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.489 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.489+0000 7f2a3858d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c7970688d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.489+0000 7f2a2effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 0x55c797068390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.489+0000 7f2a2effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 0x55c797068390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53638/0 (socket says 172.21.15.57:53638) 2024-04-21T08:54:04.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.489+0000 7f2a2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c7970688d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c7970688d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50148/0 (socket says 172.21.15.57:50148) 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a2effd640 1 -- 172.21.15.57:0/2951945849 learned_addr learned my addr 172.21.15.57:0/2951945849 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fdaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1000ee60 con 0x55c796fcdea0 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a100102d0 con 0x55c796fc97f0 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1000f530 con 0x55c796fd6a80 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.490+0000 7f2a2f7fe640 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fdaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.491+0000 7f2a3858d740 10 client.0 _ll_register_callbacks cb 0x7f2a2000d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:04.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.492+0000 7f2a2effd640 1 -- 172.21.15.57:0/2951945849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 msgr2=0x55c796fdaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.492+0000 7f2a2effd640 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fdaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.492+0000 7f2a2effd640 1 -- 172.21.15.57:0/2951945849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 msgr2=0x55c7970688d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:04.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.492+0000 7f2a2effd640 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c7970688d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.492+0000 7f2a2effd640 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c796fdb590 con 0x55c796fc97f0 2024-04-21T08:54:04.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.498+0000 7f2a2effd640 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 0x55c797068390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000d540 tx=0x7f2a20005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f0a9146e165b504 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.500+0000 7f2a0effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.503+0000 7f2a0effd640 1 -- 172.21.15.57:0/2951945849 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2003a030 con 0x55c796fc97f0 2024-04-21T08:54:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.503+0000 7f2a0effd640 1 -- 172.21.15.57:0/2951945849 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f2a20016040 con 0x55c796fc97f0 2024-04-21T08:54:04.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.504+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a200035f0 con 0x55c796fc97f0 2024-04-21T08:54:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.509+0000 7f2a0effd640 1 -- 172.21.15.57:0/2951945849 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f2a20009440 con 0x55c796fc97f0 2024-04-21T08:54:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.509+0000 7f2a3858d740 10 client.26417 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:04.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.509+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c79706fab0 con 0x55c796fc97f0 2024-04-21T08:54:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.516+0000 7f2a0effd640 1 -- 172.21.15.57:0/2951945849 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a200035f0 con 0x55c796fc97f0 2024-04-21T08:54:04.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.516+0000 7f2a0effd640 1 -- 172.21.15.57:0/2951945849 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f2a20026040 con 0x55c796fc97f0 2024-04-21T08:54:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.516+0000 7f2a3858d740 10 client.26417 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.516+0000 7f2a3858d740 10 client.26417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.516+0000 7f2a3858d740 20 client.26417 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.517+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797070070 0x55c797072460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.517+0000 7f2a3858d740 4 client.26417 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:04.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.517+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c796f39a70 con 0x55c797070070 2024-04-21T08:54:04.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.520+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797070070 0x55c797072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.521+0000 7f2a2e7fc640 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797070070 0x55c797072460 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11181b6d47014957 server_cookie=22d033fc1ac9a1da in_seq=0 out_seq=0 2024-04-21T08:54:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.523+0000 7f2a0effd640 10 client.26417 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.523+0000 7f2a0effd640 1 -- 172.21.15.57:0/2951945849 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c796f39a70 con 0x55c797070070 2024-04-21T08:54:04.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.523+0000 7f2a0effd640 10 client.26417 handle_command_reply: tid=0 2024-04-21T08:54:04.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.525+0000 7f2a3858d740 1 client.26417 shutdown 2024-04-21T08:54:04.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 msgr2=0x55c797068390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 0x55c797068390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000d540 tx=0x7f2a20005270 comp rx=0 tx=0).stop 2024-04-21T08:54:04.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 shutdown_connections 2024-04-21T08:54:04.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c797070070 0x55c797072460 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c796fd6a80 0x55c796fdaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c796fcdea0 0x55c7970688d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 --2- 172.21.15.57:0/2951945849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c796fc97f0 0x55c797068390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 >> 172.21.15.57:0/2951945849 conn(0x55c796fc7cb0 msgr2=0x55c796fd3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.527+0000 7f2a0effd640 0 client.26417 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:04.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.528+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 shutdown_connections 2024-04-21T08:54:04.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.531+0000 7f2a3858d740 1 -- 172.21.15.57:0/2951945849 wait complete. 2024-04-21T08:54:04.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.532+0000 7f2a3858d740 20 client.26417 trim_cache size 0 max 16384 2024-04-21T08:54:04.535 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:54:04.536 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:54:04.536 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:54:04.536 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:54:04.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.536+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f2a1c0a3e60 msgr2=0x7f2a1c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.536+0000 7f2a358c1640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f2a1c0a3e60 0x7f2a1c0a6320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2a20005ab0 tx=0x7f2a20009260 comp rx=0 tx=0).stop 2024-04-21T08:54:04.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.536+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 msgr2=0x7f2a3010cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.536+0000 7f2a358c1640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 0x7f2a3010cd60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800d770 tx=0x7f2a1800dc30 comp rx=0 tx=0).stop 2024-04-21T08:54:04.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.538+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 shutdown_connections 2024-04-21T08:54:04.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.538+0000 7f2a358c1640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f2a1c0a3e60 0x7f2a1c0a6320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.538+0000 7f2a358c1640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a30117260 0x7f2a301176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.538+0000 7f2a358c1640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a30106150 0x7f2a3010cd60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.538+0000 7f2a358c1640 1 --2- 172.21.15.57:0/1867835383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a30101080 0x7f2a3010c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.540 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.538+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 >> 172.21.15.57:0/1867835383 conn(0x7f2a30078000 msgr2=0x7f2a301004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.540+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 shutdown_connections 2024-04-21T08:54:04.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.540+0000 7f2a358c1640 1 -- 172.21.15.57:0/1867835383 wait complete. 2024-04-21T08:54:04.560 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:54:04.560 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:04.561 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:04 smithi057 ceph-mon[27968]: pgmap v269: 129 pgs: 12 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 82 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 922 op/s 2024-04-21T08:54:04.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:04 smithi129 ceph-mon[31272]: pgmap v269: 129 pgs: 12 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 82 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 922 op/s 2024-04-21T08:54:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:04 smithi136 ceph-mon[31791]: pgmap v269: 129 pgs: 12 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 82 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 922 op/s 2024-04-21T08:54:04.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.758+0000 7fda33a81640 1 -- 172.21.15.57:0/3633180140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c06bcd0 msgr2=0x7fda2c06c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.758+0000 7fda33a81640 1 --2- 172.21.15.57:0/3633180140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c06bcd0 0x7fda2c06c130 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fda24009be0 tx=0x7fda2402bfa0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.766+0000 7fda33a81640 1 -- 172.21.15.57:0/3633180140 shutdown_connections 2024-04-21T08:54:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.766+0000 7fda33a81640 1 --2- 172.21.15.57:0/3633180140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c11b490 0x7fda2c11d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.766+0000 7fda33a81640 1 --2- 172.21.15.57:0/3633180140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c06bcd0 0x7fda2c06c130 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.766+0000 7fda33a81640 1 --2- 172.21.15.57:0/3633180140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c06d4d0 0x7fda2c06b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.766+0000 7fda33a81640 1 -- 172.21.15.57:0/3633180140 >> 172.21.15.57:0/3633180140 conn(0x7fda2c070640 msgr2=0x7fda2c108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:04.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.770+0000 7fda33a81640 1 -- 172.21.15.57:0/3633180140 shutdown_connections 2024-04-21T08:54:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.771+0000 7fda33a81640 1 -- 172.21.15.57:0/3633180140 wait complete. 2024-04-21T08:54:04.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.771+0000 7fda33a81640 1 Processor -- start 2024-04-21T08:54:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 -- start start 2024-04-21T08:54:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c06bcd0 0x7fda2c119ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c06d4d0 0x7fda2c112b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c11b490 0x7fda2c113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c120fb0 con 0x7fda2c06d4d0 2024-04-21T08:54:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c120e30 con 0x7fda2c06bcd0 2024-04-21T08:54:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.775+0000 7fda33a81640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c121130 con 0x7fda2c11b490 2024-04-21T08:54:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3101c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c06d4d0 0x7fda2c112b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3201e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c11b490 0x7fda2c113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.780 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3201e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c11b490 0x7fda2c113080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57638/0 (socket says 172.21.15.57:57638) 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3201e640 1 -- 172.21.15.57:0/4173024793 learned_addr learned my addr 172.21.15.57:0/4173024793 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3101c640 1 -- 172.21.15.57:0/4173024793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c11b490 msgr2=0x7fda2c113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3101c640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c11b490 0x7fda2c113080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3101c640 1 -- 172.21.15.57:0/4173024793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c06bcd0 msgr2=0x7fda2c119ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3101c640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c06bcd0 0x7fda2c119ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.776+0000 7fda3101c640 1 -- 172.21.15.57:0/4173024793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c113610 con 0x7fda2c06d4d0 2024-04-21T08:54:04.781 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.777+0000 7fda3101c640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c06d4d0 0x7fda2c112b40 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fda24036040 tx=0x7fda24004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.779+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24043020 con 0x7fda2c06d4d0 2024-04-21T08:54:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.779+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda24004670 con 0x7fda2c06d4d0 2024-04-21T08:54:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.779+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2403e6b0 con 0x7fda2c06d4d0 2024-04-21T08:54:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.779+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2c1138a0 con 0x7fda2c06d4d0 2024-04-21T08:54:04.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.779+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda2c117870 con 0x7fda2c06d4d0 2024-04-21T08:54:04.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.780+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2c06cd70 con 0x7fda2c06d4d0 2024-04-21T08:54:04.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.791+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fda24046020 con 0x7fda2c06d4d0 2024-04-21T08:54:04.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.792+0000 7fda22ffd640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fda180a3f20 0x7fda180a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:04.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.793+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7fda240f3640 con 0x7fda2c06d4d0 2024-04-21T08:54:04.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.793+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda240f3b00 con 0x7fda2c06d4d0 2024-04-21T08:54:04.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.796+0000 7fda3181d640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fda180a3f20 0x7fda180a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:04.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:04.809+0000 7fda3181d640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fda180a3f20 0x7fda180a63e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda28005fd0 tx=0x7fda28009260 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.242+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda2c068de0 con 0x7fda2c06d4d0 2024-04-21T08:54:05.244 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:54:05.244 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:54:05.244 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:54:05.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.243+0000 7fda22ffd640 1 -- 172.21.15.57:0/4173024793 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fda24004820 con 0x7fda2c06d4d0 2024-04-21T08:54:05.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.247+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fda180a3f20 msgr2=0x7fda180a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.247+0000 7fda33a81640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fda180a3f20 0x7fda180a63e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda28005fd0 tx=0x7fda28009260 comp rx=0 tx=0).stop 2024-04-21T08:54:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.247+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c06d4d0 msgr2=0x7fda2c112b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.247+0000 7fda33a81640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c06d4d0 0x7fda2c112b40 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fda24036040 tx=0x7fda24004270 comp rx=0 tx=0).stop 2024-04-21T08:54:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.249+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 shutdown_connections 2024-04-21T08:54:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.249+0000 7fda33a81640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fda180a3f20 0x7fda180a63e0 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda28005fd0 tx=0x7fda28009260 comp rx=0 tx=0).stop 2024-04-21T08:54:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.249+0000 7fda33a81640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda2c11b490 0x7fda2c113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.249+0000 7fda33a81640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda2c06d4d0 0x7fda2c112b40 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.249+0000 7fda33a81640 1 --2- 172.21.15.57:0/4173024793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda2c06bcd0 0x7fda2c119ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.249+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 >> 172.21.15.57:0/4173024793 conn(0x7fda2c070640 msgr2=0x7fda2c072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.252+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 shutdown_connections 2024-04-21T08:54:05.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.253+0000 7fda33a81640 1 -- 172.21.15.57:0/4173024793 wait complete. 2024-04-21T08:54:05.270 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:54:05.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.465+0000 7f525a92e640 1 -- 172.21.15.57:0/2050672849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5254101a50 msgr2=0x7f5254105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.465+0000 7f525a92e640 1 --2- 172.21.15.57:0/2050672849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5254101a50 0x7f5254105a90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5244009900 tx=0x7f524402bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.472+0000 7f525a92e640 1 -- 172.21.15.57:0/2050672849 shutdown_connections 2024-04-21T08:54:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.472+0000 7f525a92e640 1 --2- 172.21.15.57:0/2050672849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5254106170 0x7f525410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.472+0000 7f525a92e640 1 --2- 172.21.15.57:0/2050672849 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5254101a50 0x7f5254105a90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.472 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.472+0000 7f525a92e640 1 --2- 172.21.15.57:0/2050672849 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52541010a0 0x7f5254101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.472+0000 7f525a92e640 1 -- 172.21.15.57:0/2050672849 >> 172.21.15.57:0/2050672849 conn(0x7f52540771d0 msgr2=0x7f52540775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.479+0000 7f525a92e640 1 -- 172.21.15.57:0/2050672849 shutdown_connections 2024-04-21T08:54:05.482 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.482+0000 7f525a92e640 1 -- 172.21.15.57:0/2050672849 wait complete. 2024-04-21T08:54:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.484+0000 7f525a92e640 1 Processor -- start 2024-04-21T08:54:05.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.484+0000 7f525a92e640 1 -- start start 2024-04-21T08:54:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.484+0000 7f525a92e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52541010a0 0x7f5254197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f525a92e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5254106170 0x7f5254197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f525a92e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525419bf20 0x7f525419c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f525a92e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f52541110e0 con 0x7f525419bf20 2024-04-21T08:54:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f525a92e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5254110f60 con 0x7f52541010a0 2024-04-21T08:54:05.485 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f525a92e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5254111260 con 0x7f5254106170 2024-04-21T08:54:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525419bf20 0x7f525419c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525419bf20 0x7f525419c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50186/0 (socket says 172.21.15.57:50186) 2024-04-21T08:54:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.485+0000 7f5258ecb640 1 -- 172.21.15.57:0/1113332308 learned_addr learned my addr 172.21.15.57:0/1113332308 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.486+0000 7f5253fff640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52541010a0 0x7f5254197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.486 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.487+0000 7f52537fe640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5254106170 0x7f5254197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.489+0000 7f5253fff640 1 -- 172.21.15.57:0/1113332308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5254106170 msgr2=0x7f5254197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.489+0000 7f5253fff640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5254106170 0x7f5254197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.489+0000 7f5253fff640 1 -- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525419bf20 msgr2=0x7f525419c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.489+0000 7f5253fff640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525419bf20 0x7f525419c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.489+0000 7f5253fff640 1 -- 172.21.15.57:0/1113332308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525419ca10 con 0x7f52541010a0 2024-04-21T08:54:05.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.490+0000 7f5253fff640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52541010a0 0x7f5254197660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f524800d6d0 tx=0x7f524800db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.493+0000 7f52517fa640 1 -- 172.21.15.57:0/1113332308 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248017070 con 0x7f52541010a0 2024-04-21T08:54:05.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.493+0000 7f52517fa640 1 -- 172.21.15.57:0/1113332308 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5248009820 con 0x7f52541010a0 2024-04-21T08:54:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.493+0000 7f52517fa640 1 -- 172.21.15.57:0/1113332308 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5248004c40 con 0x7f52541010a0 2024-04-21T08:54:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.493+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525419cd00 con 0x7f52541010a0 2024-04-21T08:54:05.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.493+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525406a790 con 0x7f52541010a0 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f52517fa640 1 -- 172.21.15.57:0/1113332308 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5248010460 con 0x7f52541010a0 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f525d5f4740 1 Processor -- start 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f525d5f4740 1 -- start start 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28d99750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 0x55bb28db3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28db5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.502+0000 7f525d5f4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55bb28cfe1f0 con 0x55bb28d99d20 2024-04-21T08:54:05.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f525d5f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bb28c9b4b0 con 0x55bb28db3a10 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f525d5f4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55bb28cfa1b0 con 0x55bb28d99370 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28d99750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28d99750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57666/0 (socket says 172.21.15.57:57666) 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 -- 172.21.15.57:0/58695054 learned_addr learned my addr 172.21.15.57:0/58695054 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 -- 172.21.15.57:0/58695054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 msgr2=0x55bb28db5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28db5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 -- 172.21.15.57:0/58695054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 msgr2=0x55bb28db3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 0x55bb28db3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.503+0000 7f5258ecb640 1 -- 172.21.15.57:0/58695054 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb28c8d1b0 con 0x55bb28d99370 2024-04-21T08:54:05.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.504+0000 7f52517fa640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f523c0a3f20 0x7f523c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.504+0000 7f52517fa640 1 -- 172.21.15.57:0/1113332308 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f52480d1340 con 0x7f52541010a0 2024-04-21T08:54:05.507 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.507+0000 7f5258ecb640 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28d99750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f524c00e530 tx=0x7f524c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4356a71ecece796 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.510+0000 7f5253fff640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f523c0a3f20 0x7f523c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.510+0000 7f5253fff640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f523c0a3f20 0x7f523c0a63e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x55bb28d9a380 tx=0x7f5248012000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.510+0000 7f523a7fc640 1 -- 172.21.15.57:0/58695054 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c03a890 con 0x55bb28d99370 2024-04-21T08:54:05.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.511+0000 7f523a7fc640 1 -- 172.21.15.57:0/58695054 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f524c0413b0 con 0x55bb28d99370 2024-04-21T08:54:05.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.512+0000 7f525d5f4740 1 -- 172.21.15.57:0/58695054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 msgr2=0x55bb28d99750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.512+0000 7f525d5f4740 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28d99750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f524c00e530 tx=0x7f524c009500 comp rx=0 tx=0).stop 2024-04-21T08:54:05.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.518+0000 7f525d5f4740 1 -- 172.21.15.57:0/58695054 shutdown_connections 2024-04-21T08:54:05.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.518+0000 7f525d5f4740 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28db5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.518+0000 7f525d5f4740 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 0x55bb28db3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.518+0000 7f525d5f4740 1 --2- 172.21.15.57:0/58695054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28d99750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.518+0000 7f525d5f4740 1 -- 172.21.15.57:0/58695054 >> 172.21.15.57:0/58695054 conn(0x55bb28d93ba0 msgr2=0x55bb28d93f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.523+0000 7f525d5f4740 1 -- 172.21.15.57:0/58695054 shutdown_connections 2024-04-21T08:54:05.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.525+0000 7f525d5f4740 1 -- 172.21.15.57:0/58695054 wait complete. 2024-04-21T08:54:05.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.526+0000 7f525d5f4740 1 -- start start 2024-04-21T08:54:05.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.526+0000 7f525d5f4740 1 Processor -- start 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99370 0x55bb28dad070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 0x55bb28dad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28db3a10 0x55bb28db2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55bb28d9a660 con 0x55bb28d99370 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55bb28cb7500 con 0x55bb28d99d20 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55bb28d9a7e0 con 0x55bb28db3a10 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f525d5f4740 10 client.0 _ll_register_callbacks cb 0x55bb28c85b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f52537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 0x55bb28dad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f52537fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 0x55bb28dad5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53666/0 (socket says 172.21.15.57:53666) 2024-04-21T08:54:05.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.527+0000 7f52537fe640 1 -- 172.21.15.57:0/882397303 learned_addr learned my addr 172.21.15.57:0/882397303 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f5258ecb640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28db3a10 0x55bb28db2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f5258ecb640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99370 0x55bb28dad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f52537fe640 1 -- 172.21.15.57:0/882397303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28db3a10 msgr2=0x55bb28db2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f52537fe640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28db3a10 0x55bb28db2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f52537fe640 1 -- 172.21.15.57:0/882397303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99370 msgr2=0x55bb28dad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f52537fe640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99370 0x55bb28dad070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f52537fe640 1 -- 172.21.15.57:0/882397303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb28a103f0 con 0x55bb28d99d20 2024-04-21T08:54:05.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.528+0000 7f52537fe640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 0x55bb28dad5b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5244005ab0 tx=0x7f524402c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc1df56bb54309f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.532+0000 7f52337fe640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:05.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.532+0000 7f52337fe640 1 -- 172.21.15.57:0/882397303 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244045020 con 0x55bb28d99d20 2024-04-21T08:54:05.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.532+0000 7f52337fe640 1 -- 172.21.15.57:0/882397303 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f5244005040 con 0x55bb28d99d20 2024-04-21T08:54:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.533+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f524c0413b0 con 0x55bb28d99d20 2024-04-21T08:54:05.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.536+0000 7f52337fe640 1 -- 172.21.15.57:0/882397303 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f5244005240 con 0x55bb28d99d20 2024-04-21T08:54:05.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.538+0000 7f525d5f4740 10 client.26429 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:05.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.538+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb28ada7d0 con 0x55bb28d99d20 2024-04-21T08:54:05.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.548+0000 7f52337fe640 1 -- 172.21.15.57:0/882397303 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f524c0413b0 con 0x55bb28d99d20 2024-04-21T08:54:05.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.548+0000 7f52337fe640 1 -- 172.21.15.57:0/882397303 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f52440035f0 con 0x55bb28d99d20 2024-04-21T08:54:05.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.549+0000 7f525d5f4740 10 client.26429 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:05.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.549+0000 7f525d5f4740 10 client.26429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:05.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.549+0000 7f525d5f4740 20 client.26429 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:05.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.549+0000 7f525d5f4740 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28da27c0 0x55bb28da4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.549+0000 7f525d5f4740 4 client.26429 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:05.551 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.549+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb28c9d930 con 0x55bb28da27c0 2024-04-21T08:54:05.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.552+0000 7f5258ecb640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28da27c0 0x55bb28da4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.556 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.554+0000 7f5258ecb640 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28da27c0 0x55bb28da4c20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c287481376440b server_cookie=bcdd62afe18165cb in_seq=0 out_seq=0 2024-04-21T08:54:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.557+0000 7f52337fe640 10 client.26429 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.557+0000 7f52337fe640 1 -- 172.21.15.57:0/882397303 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb28c9d930 con 0x55bb28da27c0 2024-04-21T08:54:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.557+0000 7f52337fe640 10 client.26429 handle_command_reply: tid=0 2024-04-21T08:54:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.557+0000 7f525d5f4740 1 client.26429 shutdown 2024-04-21T08:54:05.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.559+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 msgr2=0x55bb28dad5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.559+0000 7f525d5f4740 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 0x55bb28dad5b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5244005ab0 tx=0x7f524402c9f0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.561+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 shutdown_connections 2024-04-21T08:54:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.561+0000 7f525d5f4740 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28da27c0 0x55bb28da4c20 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.561+0000 7f525d5f4740 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28db3a10 0x55bb28db2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.561+0000 7f525d5f4740 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99d20 0x55bb28dad5b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.561+0000 7f525d5f4740 1 --2- 172.21.15.57:0/882397303 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99370 0x55bb28dad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.561+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 >> 172.21.15.57:0/882397303 conn(0x55bb28d93ba0 msgr2=0x55bb28d958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.562+0000 7f52337fe640 0 client.26429 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:05.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.568+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 shutdown_connections 2024-04-21T08:54:05.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.572+0000 7f525d5f4740 1 -- 172.21.15.57:0/882397303 wait complete. 2024-04-21T08:54:05.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.574+0000 7f525d5f4740 20 client.26429 trim_cache size 0 max 16384 2024-04-21T08:54:05.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:05 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4173024793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:05.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:05 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4173024793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:05.601 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.602+0000 7f525d5f4740 1 Processor -- start 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.604+0000 7f525d5f4740 1 -- start start 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.605+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28db2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.605+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 0x55bb28d9b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.605+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28d9eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.605+0000 7f525d5f4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f524000e180 con 0x55bb28d99d20 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.605+0000 7f525d5f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52400109e0 con 0x55bb28db3a10 2024-04-21T08:54:05.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.605+0000 7f525d5f4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f524000e7d0 con 0x55bb28d99370 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28db2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28d9eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28db2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57674/0 (socket says 172.21.15.57:57674) 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 -- 172.21.15.57:0/2513688445 learned_addr learned my addr 172.21.15.57:0/2513688445 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 -- 172.21.15.57:0/2513688445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 msgr2=0x55bb28d9eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28d9eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 -- 172.21.15.57:0/2513688445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 msgr2=0x55bb28d9b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 0x55bb28d9b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.608+0000 7f5258ecb640 1 -- 172.21.15.57:0/2513688445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb28d99ac0 con 0x55bb28d99370 2024-04-21T08:54:05.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.611+0000 7f5258ecb640 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28db2d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f524c041d30 tx=0x7f524c009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a1793e61b80108c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.613+0000 7f5239ffb640 1 -- 172.21.15.57:0/2513688445 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c04c040 con 0x55bb28d99370 2024-04-21T08:54:05.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.613+0000 7f5239ffb640 1 -- 172.21.15.57:0/2513688445 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f524c00bcb0 con 0x55bb28d99370 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 -- 172.21.15.57:0/2513688445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 msgr2=0x55bb28db2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28db2d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f524c041d30 tx=0x7f524c009fd0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 -- 172.21.15.57:0/2513688445 shutdown_connections 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28db3a10 0x55bb28d9eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28d99d20 0x55bb28d9b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 --2- 172.21.15.57:0/2513688445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99370 0x55bb28db2d00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.614+0000 7f525d5f4740 1 -- 172.21.15.57:0/2513688445 >> 172.21.15.57:0/2513688445 conn(0x55bb28d93ba0 msgr2=0x55bb28d9e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.618+0000 7f525d5f4740 1 -- 172.21.15.57:0/2513688445 shutdown_connections 2024-04-21T08:54:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.618+0000 7f525d5f4740 1 -- 172.21.15.57:0/2513688445 wait complete. 2024-04-21T08:54:05.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.619+0000 7f525d5f4740 1 -- start start 2024-04-21T08:54:05.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.619+0000 7f525d5f4740 1 Processor -- start 2024-04-21T08:54:05.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99370 0x55bb28dd6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99d20 0x55bb28dd7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28db3a10 0x55bb28dd77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f524000eea0 con 0x55bb28db3a10 2024-04-21T08:54:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5240010310 con 0x55bb28d99370 2024-04-21T08:54:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f524000f570 con 0x55bb28d99d20 2024-04-21T08:54:05.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.621+0000 7f525d5f4740 10 client.0 _ll_register_callbacks cb 0x7f52440043e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.623+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99370 0x55bb28dd6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.623+0000 7f5258ecb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99370 0x55bb28dd6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53686/0 (socket says 172.21.15.57:53686) 2024-04-21T08:54:05.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.623+0000 7f5258ecb640 1 -- 172.21.15.57:0/3921528141 learned_addr learned my addr 172.21.15.57:0/3921528141 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:05.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.623+0000 7f5258ecb640 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28db3a10 0x55bb28dd77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.624+0000 7f5258ecb640 1 -- 172.21.15.57:0/3921528141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99d20 msgr2=0x55bb28dd7250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.624+0000 7f5258ecb640 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99d20 0x55bb28dd7250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.624+0000 7f5258ecb640 1 -- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99370 msgr2=0x55bb28dd6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.624+0000 7f5258ecb640 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99370 0x55bb28dd6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.624+0000 7f5258ecb640 1 -- 172.21.15.57:0/3921528141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb28e28d80 con 0x55bb28db3a10 2024-04-21T08:54:05.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.626+0000 7f5258ecb640 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28db3a10 0x55bb28dd77d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f524c03a5a0 tx=0x7f524c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ac9ae78d2d0a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:05.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.626+0000 7f52337fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:05.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.627+0000 7f52337fe640 1 -- 172.21.15.57:0/3921528141 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c05f060 con 0x55bb28db3a10 2024-04-21T08:54:05.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.627+0000 7f52337fe640 1 -- 172.21.15.57:0/3921528141 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f524c00b800 con 0x55bb28db3a10 2024-04-21T08:54:05.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.627+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f524c00bcb0 con 0x55bb28db3a10 2024-04-21T08:54:05.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.633+0000 7f52337fe640 1 -- 172.21.15.57:0/3921528141 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f524c00be60 con 0x55bb28db3a10 2024-04-21T08:54:05.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:05 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4173024793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:05.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.638+0000 7f525d5f4740 10 client.16971 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:05.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.638+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55bb28e2a080 con 0x55bb28db3a10 2024-04-21T08:54:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.639+0000 7f52337fe640 1 -- 172.21.15.57:0/3921528141 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f524c00bcb0 con 0x55bb28db3a10 2024-04-21T08:54:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f52337fe640 1 -- 172.21.15.57:0/3921528141 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f524c03a690 con 0x55bb28db3a10 2024-04-21T08:54:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f525d5f4740 10 client.16971 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f525d5f4740 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:05.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f525d5f4740 20 client.16971 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f525d5f4740 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28e2a630 0x55bb28e2ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f525d5f4740 4 client.16971 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:05.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.640+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb28d0bc10 con 0x55bb28e2a630 2024-04-21T08:54:05.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.643+0000 7f52537fe640 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28e2a630 0x55bb28e2ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:05.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.645+0000 7f52537fe640 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28e2a630 0x55bb28e2ca50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f00e940cda4e03 server_cookie=735aa8253c4189e9 in_seq=0 out_seq=0 2024-04-21T08:54:05.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.646+0000 7f52337fe640 10 client.16971 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.648+0000 7f52337fe640 1 -- 172.21.15.57:0/3921528141 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb28d0bc10 con 0x55bb28e2a630 2024-04-21T08:54:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.648+0000 7f52337fe640 10 client.16971 handle_command_reply: tid=0 2024-04-21T08:54:05.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.648+0000 7f525d5f4740 1 client.16971 shutdown 2024-04-21T08:54:05.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.655+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28db3a10 msgr2=0x55bb28dd77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.655+0000 7f525d5f4740 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28db3a10 0x55bb28dd77d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f524c03a5a0 tx=0x7f524c000f40 comp rx=0 tx=0).stop 2024-04-21T08:54:05.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 shutdown_connections 2024-04-21T08:54:05.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f525d5f4740 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55bb28e2a630 0x55bb28e2ca50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f525d5f4740 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55bb28db3a10 0x55bb28dd77d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f525d5f4740 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55bb28d99d20 0x55bb28dd7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f525d5f4740 1 --2- 172.21.15.57:0/3921528141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55bb28d99370 0x55bb28dd6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 >> 172.21.15.57:0/3921528141 conn(0x55bb28d93ba0 msgr2=0x55bb28d92c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.659+0000 7f52337fe640 0 client.16971 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:05.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.664+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 shutdown_connections 2024-04-21T08:54:05.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.668+0000 7f525d5f4740 1 -- 172.21.15.57:0/3921528141 wait complete. 2024-04-21T08:54:05.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.669+0000 7f525d5f4740 20 client.16971 trim_cache size 0 max 16384 2024-04-21T08:54:05.669 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:54:05.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.673+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f523c0a3f20 msgr2=0x7f523c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.673+0000 7f525a92e640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f523c0a3f20 0x7f523c0a63e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x55bb28d9a380 tx=0x7f5248012000 comp rx=0 tx=0).stop 2024-04-21T08:54:05.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.673+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52541010a0 msgr2=0x7f5254197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.673+0000 7f525a92e640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52541010a0 0x7f5254197660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f524800d6d0 tx=0x7f524800db90 comp rx=0 tx=0).stop 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 shutdown_connections 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f523c0a3f20 0x7f523c0a63e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f525419bf20 0x7f525419c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5254106170 0x7f5254197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 --2- 172.21.15.57:0/1113332308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52541010a0 0x7f5254197660 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:05.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 >> 172.21.15.57:0/1113332308 conn(0x7f52540771d0 msgr2=0x7f52540fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:05.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 shutdown_connections 2024-04-21T08:54:05.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:05.674+0000 7f525a92e640 1 -- 172.21.15.57:0/1113332308 wait complete. 2024-04-21T08:54:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:06 smithi129 ceph-mon[31272]: pgmap v270: 129 pgs: 16 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 67 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 927 op/s 2024-04-21T08:54:06.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:06 smithi057 ceph-mon[27968]: pgmap v270: 129 pgs: 16 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 67 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 927 op/s 2024-04-21T08:54:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:06 smithi136 ceph-mon[31791]: pgmap v270: 129 pgs: 16 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 67 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.3 MiB/s wr, 927 op/s 2024-04-21T08:54:06.693 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:54:06.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:54:06.693 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:54:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.908+0000 7ff572ce0640 1 -- 172.21.15.57:0/1319971601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 msgr2=0x7ff56c10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.908+0000 7ff572ce0640 1 --2- 172.21.15.57:0/1319971601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c10ec50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff55c009890 tx=0x7ff55c02bb70 comp rx=0 tx=0).stop 2024-04-21T08:54:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.914+0000 7ff572ce0640 1 -- 172.21.15.57:0/1319971601 shutdown_connections 2024-04-21T08:54:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.914+0000 7ff572ce0640 1 --2- 172.21.15.57:0/1319971601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff56c11b730 0x7ff56c11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.914+0000 7ff572ce0640 1 --2- 172.21.15.57:0/1319971601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c10ec50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.914+0000 7ff572ce0640 1 --2- 172.21.15.57:0/1319971601 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.914+0000 7ff572ce0640 1 -- 172.21.15.57:0/1319971601 >> 172.21.15.57:0/1319971601 conn(0x7ff56c06d060 msgr2=0x7ff56c06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:06.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.918+0000 7ff572ce0640 1 -- 172.21.15.57:0/1319971601 shutdown_connections 2024-04-21T08:54:06.920 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.920+0000 7ff572ce0640 1 -- 172.21.15.57:0/1319971601 wait complete. 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.921+0000 7ff572ce0640 1 Processor -- start 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 -- start start 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c1a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c1a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff56c11b730 0x7ff56c1abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c120fc0 con 0x7ff56c1107a0 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c120e40 con 0x7ff56c10e7d0 2024-04-21T08:54:06.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.923+0000 7ff572ce0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c121140 con 0x7ff56c11b730 2024-04-21T08:54:06.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.925+0000 7ff570a7c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c1a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.928+0000 7ff56bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c1a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.928+0000 7ff56bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c1a7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50244/0 (socket says 172.21.15.57:50244) 2024-04-21T08:54:06.930 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.928+0000 7ff56bfff640 1 -- 172.21.15.57:0/228263600 learned_addr learned my addr 172.21.15.57:0/228263600 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:06.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.933+0000 7ff56bfff640 1 -- 172.21.15.57:0/228263600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff56c11b730 msgr2=0x7ff56c1abf10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:54:06.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.933+0000 7ff56bfff640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff56c11b730 0x7ff56c1abf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.933+0000 7ff56bfff640 1 -- 172.21.15.57:0/228263600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 msgr2=0x7ff56c1a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.933+0000 7ff56bfff640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c1a7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.933+0000 7ff56bfff640 1 -- 172.21.15.57:0/228263600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c1ac640 con 0x7ff56c1107a0 2024-04-21T08:54:06.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.933+0000 7ff570a7c640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c1a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:06.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.937+0000 7ff56bfff640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c1a7b90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff55c037040 tx=0x7ff55c035bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.938+0000 7ff569ffb640 1 -- 172.21.15.57:0/228263600 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c041020 con 0x7ff56c1107a0 2024-04-21T08:54:06.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.939+0000 7ff569ffb640 1 -- 172.21.15.57:0/228263600 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff55c02c9c0 con 0x7ff56c1107a0 2024-04-21T08:54:06.939 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.939+0000 7ff569ffb640 1 -- 172.21.15.57:0/228263600 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c039040 con 0x7ff56c1107a0 2024-04-21T08:54:06.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.940+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56c1ac8d0 con 0x7ff56c1107a0 2024-04-21T08:54:06.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.940+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff56c1b41c0 con 0x7ff56c1107a0 2024-04-21T08:54:06.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.945+0000 7ff5759ab740 1 Processor -- start 2024-04-21T08:54:06.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.947+0000 7ff569ffb640 1 -- 172.21.15.57:0/228263600 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff55c044020 con 0x7ff56c1107a0 2024-04-21T08:54:06.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.948+0000 7ff569ffb640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5540a3ea0 0x7ff5540a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.950+0000 7ff569ffb640 1 -- 172.21.15.57:0/228263600 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7ff55c02cb70 con 0x7ff56c1107a0 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 -- start start 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac419fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a04840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a07260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ac4195ff00 con 0x55ac419fc290 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ac418fd590 con 0x55ac419fb8e0 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.951+0000 7ff5759ab740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ac4195bec0 con 0x55ac41a04e10 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.952+0000 7ff57127d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a07260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.952+0000 7ff57127d640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5540a3ea0 0x7ff5540a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.952+0000 7ff57127d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a07260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57696/0 (socket says 172.21.15.57:57696) 2024-04-21T08:54:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.952+0000 7ff57127d640 1 -- 172.21.15.57:0/841888263 learned_addr learned my addr 172.21.15.57:0/841888263 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.953+0000 7ff57127d640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5540a3ea0 0x7ff5540a6360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff56400fa40 tx=0x7ff564009520 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.953+0000 7ff570a7c640 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac419fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:06.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.954+0000 7ff570a7c640 1 -- 172.21.15.57:0/841888263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 msgr2=0x55ac41a07260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.954+0000 7ff570a7c640 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a07260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.954+0000 7ff570a7c640 1 -- 172.21.15.57:0/841888263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 msgr2=0x55ac41a04840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.954+0000 7ff570a7c640 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a04840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.954+0000 7ff570a7c640 1 -- 172.21.15.57:0/841888263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac4173a7e0 con 0x55ac419fb8e0 2024-04-21T08:54:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.955+0000 7ff570a7c640 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac419fbcc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff560009d70 tx=0x7ff56000e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88182f693742c67e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:06.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.955+0000 7ff57127d640 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a07260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:06.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.959+0000 7ff552ffd640 1 -- 172.21.15.57:0/841888263 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560017070 con 0x55ac419fb8e0 2024-04-21T08:54:06.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.959+0000 7ff552ffd640 1 -- 172.21.15.57:0/841888263 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff560013920 con 0x55ac419fb8e0 2024-04-21T08:54:06.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.959+0000 7ff552ffd640 1 -- 172.21.15.57:0/841888263 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560012890 con 0x55ac419fb8e0 2024-04-21T08:54:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.968+0000 7ff5759ab740 1 -- 172.21.15.57:0/841888263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 msgr2=0x55ac419fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:06.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.968+0000 7ff5759ab740 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac419fbcc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff560009d70 tx=0x7ff56000e5a0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.981+0000 7ff5759ab740 1 -- 172.21.15.57:0/841888263 shutdown_connections 2024-04-21T08:54:06.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.981+0000 7ff5759ab740 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a07260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.980 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.981+0000 7ff5759ab740 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a04840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.981+0000 7ff5759ab740 1 --2- 172.21.15.57:0/841888263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac419fbcc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:06.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.981+0000 7ff5759ab740 1 -- 172.21.15.57:0/841888263 >> 172.21.15.57:0/841888263 conn(0x55ac419f5fb0 msgr2=0x55ac419f6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:06.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.988+0000 7ff5759ab740 1 -- 172.21.15.57:0/841888263 shutdown_connections 2024-04-21T08:54:06.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.993+0000 7ff5759ab740 1 -- 172.21.15.57:0/841888263 wait complete. 2024-04-21T08:54:06.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.994+0000 7ff5759ab740 1 -- start start 2024-04-21T08:54:06.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.994+0000 7ff5759ab740 1 Processor -- start 2024-04-21T08:54:06.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.997+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 0x55ac41a01e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.997+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fc290 0x55ac41a02360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.997+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac41a04e10 0x55ac41a028a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:06.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.997+0000 7ff5759ab740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ac41a07d80 con 0x55ac41a04e10 2024-04-21T08:54:06.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.997+0000 7ff5759ab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ac41919580 con 0x55ac419fc290 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.997+0000 7ff5759ab740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ac41a07f00 con 0x55ac419fb8e0 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.998+0000 7ff5759ab740 10 client.0 _ll_register_callbacks cb 0x55ac4191f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.999+0000 7ff570a7c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 0x55ac41a01e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.999+0000 7ff570a7c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 0x55ac41a01e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57698/0 (socket says 172.21.15.57:57698) 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:06.999+0000 7ff570a7c640 1 -- 172.21.15.57:0/844954691 learned_addr learned my addr 172.21.15.57:0/844954691 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.000+0000 7ff570a7c640 1 -- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fc290 msgr2=0x55ac41a02360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.000+0000 7ff570a7c640 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fc290 0x55ac41a02360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.000+0000 7ff570a7c640 1 -- 172.21.15.57:0/844954691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac41a04e10 msgr2=0x55ac41a028a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:07.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.000+0000 7ff570a7c640 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac41a04e10 0x55ac41a028a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.000+0000 7ff570a7c640 1 -- 172.21.15.57:0/844954691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac4179af70 con 0x55ac419fb8e0 2024-04-21T08:54:07.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.001+0000 7ff570a7c640 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 0x55ac41a01e20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff560000c00 tx=0x7ff560013d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=439f05e1ebfb63a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.002+0000 7ff54bfff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:07.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.002+0000 7ff54bfff640 1 -- 172.21.15.57:0/844954691 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560017040 con 0x55ac419fb8e0 2024-04-21T08:54:07.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.003+0000 7ff54bfff640 1 -- 172.21.15.57:0/844954691 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7ff560012a20 con 0x55ac419fb8e0 2024-04-21T08:54:07.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.004+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff560013920 con 0x55ac419fb8e0 2024-04-21T08:54:07.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.007+0000 7ff54bfff640 1 -- 172.21.15.57:0/844954691 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff560009960 con 0x55ac419fb8e0 2024-04-21T08:54:07.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.008+0000 7ff5759ab740 10 client.26326 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:07.008 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.008+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ac41632170 con 0x55ac419fb8e0 2024-04-21T08:54:07.009 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.008+0000 7ff54bfff640 1 -- 172.21.15.57:0/844954691 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff560013920 con 0x55ac419fb8e0 2024-04-21T08:54:07.009 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.008+0000 7ff54bfff640 1 -- 172.21.15.57:0/844954691 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff560023020 con 0x55ac419fb8e0 2024-04-21T08:54:07.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.012+0000 7ff5759ab740 10 client.26326 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:07.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.012+0000 7ff5759ab740 10 client.26326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:07.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.012+0000 7ff5759ab740 20 client.26326 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:07.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.012+0000 7ff5759ab740 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a26240 0x55ac41a28630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.012+0000 7ff5759ab740 4 client.26326 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:07.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.012+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac4196d920 con 0x55ac41a26240 2024-04-21T08:54:07.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.013+0000 7ff56bfff640 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a26240 0x55ac41a28630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.019+0000 7ff56bfff640 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a26240 0x55ac41a28630 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1112ef1f1fd64f20 server_cookie=3c94ef66b98faa37 in_seq=0 out_seq=0 2024-04-21T08:54:07.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.019+0000 7ff54bfff640 10 client.26326 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.020+0000 7ff54bfff640 1 -- 172.21.15.57:0/844954691 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac4196d920 con 0x55ac41a26240 2024-04-21T08:54:07.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.020+0000 7ff54bfff640 10 client.26326 handle_command_reply: tid=0 2024-04-21T08:54:07.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.022+0000 7ff5759ab740 1 client.26326 shutdown 2024-04-21T08:54:07.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.026+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 msgr2=0x55ac41a01e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.026+0000 7ff5759ab740 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 0x55ac41a01e20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff560000c00 tx=0x7ff560013d00 comp rx=0 tx=0).stop 2024-04-21T08:54:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.033+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 shutdown_connections 2024-04-21T08:54:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.033+0000 7ff5759ab740 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a26240 0x55ac41a28630 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.033+0000 7ff5759ab740 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac41a04e10 0x55ac41a028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.033+0000 7ff5759ab740 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fc290 0x55ac41a02360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.033+0000 7ff5759ab740 1 --2- 172.21.15.57:0/844954691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fb8e0 0x55ac41a01e20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.033+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 >> 172.21.15.57:0/844954691 conn(0x55ac419f5fb0 msgr2=0x55ac419f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.034+0000 7ff54bfff640 0 client.26326 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.041+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 shutdown_connections 2024-04-21T08:54:07.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.044+0000 7ff5759ab740 1 -- 172.21.15.57:0/844954691 wait complete. 2024-04-21T08:54:07.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.048+0000 7ff5759ab740 20 client.26326 trim_cache size 0 max 16384 2024-04-21T08:54:07.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.084+0000 7ff5759ab740 1 Processor -- start 2024-04-21T08:54:07.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 -- start start 2024-04-21T08:54:07.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 0x55ac41a04550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fc290 0x55ac41a10c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac41a04e10 0x55ac419f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.087 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff570a7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 0x55ac41a04550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff570a7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 0x55ac41a04550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50270/0 (socket says 172.21.15.57:50270) 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff570a7c640 1 -- 172.21.15.57:0/2575219248 learned_addr learned my addr 172.21.15.57:0/2575219248 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff55800e100 con 0x55ac419fb8e0 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff558010960 con 0x55ac41a04e10 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.086+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff55800e750 con 0x55ac419fc290 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.087+0000 7ff570a7c640 1 -- 172.21.15.57:0/2575219248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fc290 msgr2=0x55ac41a10c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.087+0000 7ff570a7c640 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fc290 0x55ac41a10c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.087+0000 7ff570a7c640 1 -- 172.21.15.57:0/2575219248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac41a04e10 msgr2=0x55ac419f4e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.087+0000 7ff570a7c640 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac41a04e10 0x55ac419f4e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.087+0000 7ff570a7c640 1 -- 172.21.15.57:0/2575219248 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac41a02df0 con 0x55ac419fb8e0 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.087+0000 7ff570a7c640 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 0x55ac41a04550 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff56000eaa0 tx=0x7ff56000af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585ed0ea105182e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.088 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.088+0000 7ff5527fc640 1 -- 172.21.15.57:0/2575219248 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560033030 con 0x55ac419fb8e0 2024-04-21T08:54:07.089 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.089+0000 7ff5527fc640 1 -- 172.21.15.57:0/2575219248 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff560009bc0 con 0x55ac419fb8e0 2024-04-21T08:54:07.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.090+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 msgr2=0x55ac41a04550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.090 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.090+0000 7ff5759ab740 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 0x55ac41a04550 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff56000eaa0 tx=0x7ff56000af90 comp rx=0 tx=0).stop 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.094+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 shutdown_connections 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.094+0000 7ff5759ab740 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac41a04e10 0x55ac419f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.094+0000 7ff5759ab740 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac419fc290 0x55ac41a10c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.094+0000 7ff5759ab740 1 --2- 172.21.15.57:0/2575219248 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fb8e0 0x55ac41a04550 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.094+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 >> 172.21.15.57:0/2575219248 conn(0x55ac419f5fb0 msgr2=0x55ac419f4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.095+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 shutdown_connections 2024-04-21T08:54:07.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.095+0000 7ff5759ab740 1 -- 172.21.15.57:0/2575219248 wait complete. 2024-04-21T08:54:07.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.095+0000 7ff5759ab740 1 -- start start 2024-04-21T08:54:07.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.095+0000 7ff5759ab740 1 Processor -- start 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac41a0d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a08fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff55800ee20 con 0x55ac419fc290 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff558010290 con 0x55ac419fb8e0 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff55800f4f0 con 0x55ac41a04e10 2024-04-21T08:54:07.096 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff5759ab740 10 client.0 _ll_register_callbacks cb 0x7ff564009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff57127d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a08fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff57127d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a08fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57712/0 (socket says 172.21.15.57:57712) 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff57127d640 1 -- 172.21.15.57:0/4057245527 learned_addr learned my addr 172.21.15.57:0/4057245527 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff570a7c640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac41a0d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.096+0000 7ff56bfff640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.097+0000 7ff57127d640 1 -- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 msgr2=0x55ac41a0d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.097+0000 7ff57127d640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac41a0d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.097+0000 7ff57127d640 1 -- 172.21.15.57:0/4057245527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 msgr2=0x55ac41a089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.097+0000 7ff57127d640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a089d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.097 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.097+0000 7ff57127d640 1 -- 172.21.15.57:0/4057245527 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac41a09810 con 0x55ac41a04e10 2024-04-21T08:54:07.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.097+0000 7ff57127d640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a08fe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff56400cf50 tx=0x7ff564019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cc48931c3055226 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.098 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.098+0000 7ff54bfff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:07.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.100+0000 7ff54bfff640 1 -- 172.21.15.57:0/4057245527 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56402a060 con 0x55ac41a04e10 2024-04-21T08:54:07.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.100+0000 7ff54bfff640 1 -- 172.21.15.57:0/4057245527 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7ff564025470 con 0x55ac41a04e10 2024-04-21T08:54:07.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.100+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff560009bc0 con 0x55ac41a04e10 2024-04-21T08:54:07.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.103+0000 7ff54bfff640 1 -- 172.21.15.57:0/4057245527 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff564009aa0 con 0x55ac41a04e10 2024-04-21T08:54:07.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.105+0000 7ff5759ab740 10 client.26329 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:07.105 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.105+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ac41a8c670 con 0x55ac41a04e10 2024-04-21T08:54:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.115+0000 7ff54bfff640 1 -- 172.21.15.57:0/4057245527 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff560009bc0 con 0x55ac41a04e10 2024-04-21T08:54:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.115+0000 7ff54bfff640 1 -- 172.21.15.57:0/4057245527 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff56401c050 con 0x55ac41a04e10 2024-04-21T08:54:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.116+0000 7ff5759ab740 10 client.26329 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.116+0000 7ff5759ab740 10 client.26329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:07.115 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.116+0000 7ff5759ab740 20 client.26329 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.116+0000 7ff5759ab740 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a8cd20 0x55ac41a8f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.116+0000 7ff5759ab740 4 client.26329 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:54:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.116+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ac41967790 con 0x55ac41a8cd20 2024-04-21T08:54:07.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.117+0000 7ff570a7c640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a8cd20 0x55ac41a8f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.126+0000 7ff570a7c640 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a8cd20 0x55ac41a8f110 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10dda9904dd84839 server_cookie=fc37716bc714e211 in_seq=0 out_seq=0 2024-04-21T08:54:07.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.126+0000 7ff54bfff640 10 client.26329 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.130+0000 7ff54bfff640 1 -- 172.21.15.57:0/4057245527 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ac41967790 con 0x55ac41a8cd20 2024-04-21T08:54:07.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.130+0000 7ff54bfff640 10 client.26329 handle_command_reply: tid=0 2024-04-21T08:54:07.130 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.130+0000 7ff5759ab740 1 client.26329 shutdown 2024-04-21T08:54:07.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.132+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 msgr2=0x55ac41a08fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.132 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.132+0000 7ff5759ab740 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a08fe0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff56400cf50 tx=0x7ff564019000 comp rx=0 tx=0).stop 2024-04-21T08:54:07.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 shutdown_connections 2024-04-21T08:54:07.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff5759ab740 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ac41a8cd20 0x55ac41a8f110 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff5759ab740 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ac41a04e10 0x55ac41a08fe0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff5759ab740 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ac419fc290 0x55ac41a089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff5759ab740 1 --2- 172.21.15.57:0/4057245527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ac419fb8e0 0x55ac41a0d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff54bfff640 0 client.26329 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.134 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.134+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 >> 172.21.15.57:0/4057245527 conn(0x55ac419f5fb0 msgr2=0x55ac419f5010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.135+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 shutdown_connections 2024-04-21T08:54:07.135 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.135+0000 7ff5759ab740 1 -- 172.21.15.57:0/4057245527 wait complete. 2024-04-21T08:54:07.137 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.137+0000 7ff5759ab740 20 client.26329 trim_cache size 0 max 16384 2024-04-21T08:54:07.140 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:54:07.140 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:54:07.140 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "8206b271-55b7-4fbb-8695-a695ac8d2315", 2024-04-21T08:54:07.140 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:54:07.140 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:54:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.141+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5540a3ea0 msgr2=0x7ff5540a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.141+0000 7ff572ce0640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5540a3ea0 0x7ff5540a6360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff56400fa40 tx=0x7ff564009520 comp rx=0 tx=0).stop 2024-04-21T08:54:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.141+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 msgr2=0x7ff56c1a7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.141+0000 7ff572ce0640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c1a7b90 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff55c037040 tx=0x7ff55c035bd0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.142+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 shutdown_connections 2024-04-21T08:54:07.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.142+0000 7ff572ce0640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff5540a3ea0 0x7ff5540a6360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.142+0000 7ff572ce0640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff56c11b730 0x7ff56c1abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.142+0000 7ff572ce0640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff56c1107a0 0x7ff56c1a7b90 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.142+0000 7ff572ce0640 1 --2- 172.21.15.57:0/228263600 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff56c10e7d0 0x7ff56c1a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.142+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 >> 172.21.15.57:0/228263600 conn(0x7ff56c06d060 msgr2=0x7ff56c072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.142 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.143+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 shutdown_connections 2024-04-21T08:54:07.144 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.144+0000 7ff572ce0640 1 -- 172.21.15.57:0/228263600 wait complete. 2024-04-21T08:54:07.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:07.160 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:07.359 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.354+0000 7f92d8a5c640 1 -- 172.21.15.57:0/1013575177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4100f70 msgr2=0x7f92d410d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.354+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/1013575177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4100f70 0x7f92d410d430 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f92cc01acb0 tx=0x7f92cc03d170 comp rx=0 tx=0).stop 2024-04-21T08:54:07.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.356+0000 7f92d8a5c640 1 -- 172.21.15.57:0/1013575177 shutdown_connections 2024-04-21T08:54:07.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.356+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/1013575177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4100f70 0x7f92d410d430 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.356+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/1013575177 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d41005b0 0x7f92d4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.356+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/1013575177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d4106670 0x7f92d4106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.360 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.356+0000 7f92d8a5c640 1 -- 172.21.15.57:0/1013575177 >> 172.21.15.57:0/1013575177 conn(0x7f92d40773b0 msgr2=0x7f92d40fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.360+0000 7f92d8a5c640 1 -- 172.21.15.57:0/1013575177 shutdown_connections 2024-04-21T08:54:07.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.361+0000 7f92d8a5c640 1 -- 172.21.15.57:0/1013575177 wait complete. 2024-04-21T08:54:07.362 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 Processor -- start 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 -- start start 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d41005b0 0x7f92d41088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100f70 0x7f92d4108de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4106670 0x7f92d4109340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f92d406d9f0 con 0x7f92d4106670 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92d406d870 con 0x7f92d41005b0 2024-04-21T08:54:07.363 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.362+0000 7f92d8a5c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f92d406db70 con 0x7f92d4100f70 2024-04-21T08:54:07.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.364+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100f70 0x7f92d4108de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.364+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100f70 0x7f92d4108de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57726/0 (socket says 172.21.15.57:57726) 2024-04-21T08:54:07.364 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.364+0000 7f92cbfff640 1 -- 172.21.15.57:0/2220211098 learned_addr learned my addr 172.21.15.57:0/2220211098 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.366+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4106670 0x7f92d4109340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.366 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d259c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d41005b0 0x7f92d41088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d2d9d640 1 -- 172.21.15.57:0/2220211098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100f70 msgr2=0x7f92d4108de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100f70 0x7f92d4108de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d2d9d640 1 -- 172.21.15.57:0/2220211098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d41005b0 msgr2=0x7f92d41088a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d41005b0 0x7f92d41088a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d2d9d640 1 -- 172.21.15.57:0/2220211098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d4192e60 con 0x7f92d4106670 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.367+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4106670 0x7f92d4109340 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f92cc01ac00 tx=0x7f92cc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.369 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.369+0000 7f92d259c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d41005b0 0x7f92d41088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.371+0000 7f92cb7fe640 1 -- 172.21.15.57:0/2220211098 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc04a040 con 0x7f92d4106670 2024-04-21T08:54:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.371+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d41930f0 con 0x7f92d4106670 2024-04-21T08:54:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.372+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d4193660 con 0x7f92d4106670 2024-04-21T08:54:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.372+0000 7f92cb7fe640 1 -- 172.21.15.57:0/2220211098 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92cc055070 con 0x7f92d4106670 2024-04-21T08:54:07.374 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.372+0000 7f92cb7fe640 1 -- 172.21.15.57:0/2220211098 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc045680 con 0x7f92d4106670 2024-04-21T08:54:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.377+0000 7f92db654740 1 Processor -- start 2024-04-21T08:54:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.377+0000 7f92cb7fe640 1 -- 172.21.15.57:0/2220211098 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f92cc045820 con 0x7f92d4106670 2024-04-21T08:54:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.378+0000 7f92db654740 1 -- start start 2024-04-21T08:54:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.378+0000 7f92cb7fe640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f92c00a3f30 0x7f92c00a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.379+0000 7f92cb7fe640 1 -- 172.21.15.57:0/2220211098 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f92cc104a00 con 0x7f92d4106670 2024-04-21T08:54:07.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.379+0000 7f92db654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e57960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.379+0000 7f92db654740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e60540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.379+0000 7f92db654740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 0x5556e4e62e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.380+0000 7f92db654740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5556e4dbc1e0 con 0x5556e4e60a80 2024-04-21T08:54:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.380+0000 7f92db654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5556e4d59390 con 0x5556e4e57580 2024-04-21T08:54:07.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.380+0000 7f92db654740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5556e4db81a0 con 0x5556e4e57ea0 2024-04-21T08:54:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.381+0000 7f92d2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 0x5556e4e62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.381+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e60540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.381+0000 7f92d2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 0x5556e4e62e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50310/0 (socket says 172.21.15.57:50310) 2024-04-21T08:54:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.381+0000 7f92d2d9d640 1 -- 172.21.15.57:0/2582206928 learned_addr learned my addr 172.21.15.57:0/2582206928 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.381+0000 7f92cbfff640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f92c00a3f30 0x7f92c00a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.381+0000 7f92d259c640 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e57960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.384+0000 7f92cbfff640 1 -- 172.21.15.57:0/2582206928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 msgr2=0x5556e4e57960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.384+0000 7f92cbfff640 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e57960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.384+0000 7f92cbfff640 1 -- 172.21.15.57:0/2582206928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 msgr2=0x5556e4e62e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.384+0000 7f92cbfff640 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 0x5556e4e62e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.384+0000 7f92cbfff640 1 -- 172.21.15.57:0/2582206928 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e4b987d0 con 0x5556e4e57ea0 2024-04-21T08:54:07.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.384+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 0x5556e4e62e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:07.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.385+0000 7f92cbfff640 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e60540 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92c4006a50 tx=0x7f92c4014900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c592c23cb5d47623 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.386+0000 7f92c8ff9640 1 -- 172.21.15.57:0/2582206928 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4017470 con 0x5556e4e57ea0 2024-04-21T08:54:07.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.386+0000 7f92c8ff9640 1 -- 172.21.15.57:0/2582206928 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c401b070 con 0x5556e4e57ea0 2024-04-21T08:54:07.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.386+0000 7f92c8ff9640 1 -- 172.21.15.57:0/2582206928 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4021680 con 0x5556e4e57ea0 2024-04-21T08:54:07.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.387+0000 7f92db654740 1 -- 172.21.15.57:0/2582206928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 msgr2=0x5556e4e60540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.387+0000 7f92db654740 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e60540 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92c4006a50 tx=0x7f92c4014900 comp rx=0 tx=0).stop 2024-04-21T08:54:07.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.388+0000 7f92cbfff640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f92c00a3f30 0x7f92c00a63f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92d410a060 tx=0x7f92c4019000 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.389+0000 7f92db654740 1 -- 172.21.15.57:0/2582206928 shutdown_connections 2024-04-21T08:54:07.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.389+0000 7f92db654740 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e60a80 0x5556e4e62e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.389+0000 7f92db654740 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e60540 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.389+0000 7f92db654740 1 --2- 172.21.15.57:0/2582206928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e57960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.389+0000 7f92db654740 1 -- 172.21.15.57:0/2582206928 >> 172.21.15.57:0/2582206928 conn(0x5556e4e51cb0 msgr2=0x5556e4e52090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.393+0000 7f92db654740 1 -- 172.21.15.57:0/2582206928 shutdown_connections 2024-04-21T08:54:07.393 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.394+0000 7f92db654740 1 -- 172.21.15.57:0/2582206928 wait complete. 2024-04-21T08:54:07.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.394+0000 7f92db654740 1 -- start start 2024-04-21T08:54:07.394 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.394+0000 7f92db654740 1 Processor -- start 2024-04-21T08:54:07.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57580 0x5556e4e5da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e5df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5556e4e63860 con 0x5556e4e57ea0 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5556e4d752f0 con 0x5556e4e60a80 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5556e4e639e0 con 0x5556e4e57580 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.402+0000 7f92db654740 10 client.0 _ll_register_callbacks cb 0x5556e4d82580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92d259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57580 0x5556e4e5da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e5df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e5df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50318/0 (socket says 172.21.15.57:50318) 2024-04-21T08:54:07.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 -- 172.21.15.57:0/487935046 learned_addr learned my addr 172.21.15.57:0/487935046 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 -- 172.21.15.57:0/487935046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57580 msgr2=0x5556e4e5da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57580 0x5556e4e5da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 -- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 msgr2=0x5556e4e5e4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:07.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.403+0000 7f92cbfff640 1 -- 172.21.15.57:0/487935046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e4ace3f0 con 0x5556e4e57ea0 2024-04-21T08:54:07.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.404+0000 7f92cbfff640 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e5df70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f92c4006e30 tx=0x7f92c4016f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554f0ef79d845825 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.404 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.404+0000 7f92b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:07.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.406+0000 7f92b9ffb640 1 -- 172.21.15.57:0/487935046 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4025040 con 0x5556e4e57ea0 2024-04-21T08:54:07.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.406+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92c401b070 con 0x5556e4e57ea0 2024-04-21T08:54:07.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.406+0000 7f92b9ffb640 1 -- 172.21.15.57:0/487935046 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f92c4021cf0 con 0x5556e4e57ea0 2024-04-21T08:54:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.411+0000 7f92b9ffb640 1 -- 172.21.15.57:0/487935046 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f92c4005990 con 0x5556e4e57ea0 2024-04-21T08:54:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.411+0000 7f92db654740 10 client.16992 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.411+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5556e4a8e0f0 con 0x5556e4e57ea0 2024-04-21T08:54:07.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92b9ffb640 1 -- 172.21.15.57:0/487935046 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c401b070 con 0x5556e4e57ea0 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92b9ffb640 1 -- 172.21.15.57:0/487935046 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f92c4015040 con 0x5556e4e57ea0 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92db654740 10 client.16992 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92db654740 10 client.16992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92db654740 20 client.16992 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92db654740 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4e70e80 0x5556e4e73300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92db654740 4 client.16992 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:07.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.412+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556e4dc9c00 con 0x5556e4e70e80 2024-04-21T08:54:07.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.426+0000 7f92d259c640 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4e70e80 0x5556e4e73300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.436+0000 7f92d259c640 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4e70e80 0x5556e4e73300 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aca05461f1894897 server_cookie=71ed6c22ecf8e839 in_seq=0 out_seq=0 2024-04-21T08:54:07.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.437+0000 7f92b9ffb640 10 client.16992 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.440+0000 7f92b9ffb640 1 -- 172.21.15.57:0/487935046 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556e4dc9c00 con 0x5556e4e70e80 2024-04-21T08:54:07.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.440+0000 7f92b9ffb640 10 client.16992 handle_command_reply: tid=0 2024-04-21T08:54:07.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.442+0000 7f92db654740 1 client.16992 shutdown 2024-04-21T08:54:07.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.450+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 msgr2=0x5556e4e5df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.450+0000 7f92db654740 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e5df70 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f92c4006e30 tx=0x7f92c4016f50 comp rx=0 tx=0).stop 2024-04-21T08:54:07.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 shutdown_connections 2024-04-21T08:54:07.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92db654740 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4e70e80 0x5556e4e73300 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92db654740 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92db654740 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e5df70 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92db654740 1 --2- 172.21.15.57:0/487935046 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57580 0x5556e4e5da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 >> 172.21.15.57:0/487935046 conn(0x5556e4e51cb0 msgr2=0x5556e4e53a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.455+0000 7f92b9ffb640 0 client.16992 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.458 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.459+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 shutdown_connections 2024-04-21T08:54:07.463 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.463+0000 7f92db654740 1 -- 172.21.15.57:0/487935046 wait complete. 2024-04-21T08:54:07.465 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.465+0000 7f92db654740 20 client.16992 trim_cache size 0 max 16384 2024-04-21T08:54:07.500 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.500+0000 7f92db654740 1 Processor -- start 2024-04-21T08:54:07.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.503+0000 7f92db654740 1 -- start start 2024-04-21T08:54:07.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.503+0000 7f92db654740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57580 0x5556e4e60160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.503+0000 7f92db654740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e6c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.503 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92db654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92db654740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400e140 con 0x5556e4e57580 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92db654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92b40109a0 con 0x5556e4e60a80 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92db654740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400e790 con 0x5556e4e57ea0 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e6c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e6c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57752/0 (socket says 172.21.15.57:57752) 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92cbfff640 1 -- 172.21.15.57:0/3908869435 learned_addr learned my addr 172.21.15.57:0/3908869435 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.504+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92d259c640 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57580 0x5556e4e60160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92cbfff640 1 -- 172.21.15.57:0/3908869435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 msgr2=0x5556e4e5e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92cbfff640 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92cbfff640 1 -- 172.21.15.57:0/3908869435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57580 msgr2=0x5556e4e60160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92cbfff640 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57580 0x5556e4e60160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92cbfff640 1 -- 172.21.15.57:0/3908869435 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556e4e62e70 con 0x5556e4e57ea0 2024-04-21T08:54:07.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92d259c640 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57580 0x5556e4e60160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:07.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.506+0000 7f92cbfff640 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e6c7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f92c400d190 tx=0x7f92c4014e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef0b06b03d9fc635 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.508+0000 7f92bbfff640 1 -- 172.21.15.57:0/3908869435 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4040020 con 0x5556e4e57ea0 2024-04-21T08:54:07.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.509+0000 7f92bbfff640 1 -- 172.21.15.57:0/3908869435 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c401b070 con 0x5556e4e57ea0 2024-04-21T08:54:07.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.509+0000 7f92db654740 1 -- 172.21.15.57:0/3908869435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 msgr2=0x5556e4e6c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.509 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.509+0000 7f92db654740 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e6c7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f92c400d190 tx=0x7f92c4014e80 comp rx=0 tx=0).stop 2024-04-21T08:54:07.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.512+0000 7f92db654740 1 -- 172.21.15.57:0/3908869435 shutdown_connections 2024-04-21T08:54:07.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.512+0000 7f92db654740 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e60a80 0x5556e4e5e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.512+0000 7f92db654740 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e57ea0 0x5556e4e6c7d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.512+0000 7f92db654740 1 --2- 172.21.15.57:0/3908869435 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57580 0x5556e4e60160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.512+0000 7f92db654740 1 -- 172.21.15.57:0/3908869435 >> 172.21.15.57:0/3908869435 conn(0x5556e4e51cb0 msgr2=0x5556e4e5d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.519+0000 7f92db654740 1 -- 172.21.15.57:0/3908869435 shutdown_connections 2024-04-21T08:54:07.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.520+0000 7f92db654740 1 -- 172.21.15.57:0/3908869435 wait complete. 2024-04-21T08:54:07.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 -- start start 2024-04-21T08:54:07.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 Processor -- start 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e68ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e69220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 0x5556e4e5a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400ee60 con 0x5556e4e57ea0 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92b40102d0 con 0x5556e4e57580 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f92b400f530 con 0x5556e4e60a80 2024-04-21T08:54:07.521 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.521+0000 7f92db654740 10 client.0 _ll_register_callbacks cb 0x7f92c40142f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.522+0000 7f92d2d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 0x5556e4e5a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.522+0000 7f92d2d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 0x5556e4e5a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:57762/0 (socket says 172.21.15.57:57762) 2024-04-21T08:54:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.522+0000 7f92d2d9d640 1 -- 172.21.15.57:0/1732267503 learned_addr learned my addr 172.21.15.57:0/1732267503 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.523+0000 7f92d259c640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e68ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.523+0000 7f92cbfff640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e69220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.526+0000 7f92d2d9d640 1 -- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 msgr2=0x5556e4e68ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.526+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e68ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.526+0000 7f92d2d9d640 1 -- 172.21.15.57:0/1732267503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 msgr2=0x5556e4e69220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.526+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e69220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.527+0000 7f92d2d9d640 1 -- 172.21.15.57:0/1732267503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556e4e5a940 con 0x5556e4e60a80 2024-04-21T08:54:07.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.530+0000 7f92d2d9d640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 0x5556e4e5a110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0ccaa0 tx=0x7f92cc0d0180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ccd06587d98a65 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:07.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.531+0000 7f92b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:07.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.532+0000 7f92b9ffb640 1 -- 172.21.15.57:0/1732267503 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc01a8e0 con 0x5556e4e60a80 2024-04-21T08:54:07.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.533+0000 7f92b9ffb640 1 -- 172.21.15.57:0/1732267503 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f92cc101f60 con 0x5556e4e60a80 2024-04-21T08:54:07.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.533+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92c401b070 con 0x5556e4e60a80 2024-04-21T08:54:07.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.538+0000 7f92b9ffb640 1 -- 172.21.15.57:0/1732267503 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f92cc102160 con 0x5556e4e60a80 2024-04-21T08:54:07.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.541+0000 7f92db654740 10 client.26344 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:07.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.541+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5556e4ed6fc0 con 0x5556e4e60a80 2024-04-21T08:54:07.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.545+0000 7f92b9ffb640 1 -- 172.21.15.57:0/1732267503 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c401b070 con 0x5556e4e60a80 2024-04-21T08:54:07.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.545+0000 7f92b9ffb640 1 -- 172.21.15.57:0/1732267503 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f92cc003c10 con 0x5556e4e60a80 2024-04-21T08:54:07.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.547+0000 7f92db654740 10 client.26344 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:07.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.547+0000 7f92db654740 10 client.26344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:07.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.547+0000 7f92db654740 20 client.26344 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:07.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.547+0000 7f92db654740 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4ed76f0 0x5556e4ed9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:07.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.547+0000 7f92db654740 4 client.26344 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:07.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.547+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5556e4dc3a70 con 0x5556e4ed76f0 2024-04-21T08:54:07.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.553+0000 7f92d259c640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4ed76f0 0x5556e4ed9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:07.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.555+0000 7f92d259c640 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4ed76f0 0x5556e4ed9b10 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad511cb41b252700 server_cookie=4bcdea2cc85bcab7 in_seq=0 out_seq=0 2024-04-21T08:54:07.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.556+0000 7f92b9ffb640 10 client.26344 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.559+0000 7f92b9ffb640 1 -- 172.21.15.57:0/1732267503 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x5556e4dc3a70 con 0x5556e4ed76f0 2024-04-21T08:54:07.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.559+0000 7f92b9ffb640 10 client.26344 handle_command_reply: tid=0 2024-04-21T08:54:07.559 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.560+0000 7f92db654740 1 client.26344 shutdown 2024-04-21T08:54:07.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.566+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 msgr2=0x5556e4e5a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.565 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.566+0000 7f92db654740 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 0x5556e4e5a110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f92cc0ccaa0 tx=0x7f92cc0d0180 comp rx=0 tx=0).stop 2024-04-21T08:54:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 shutdown_connections 2024-04-21T08:54:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92db654740 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5556e4ed76f0 0x5556e4ed9b10 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92db654740 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5556e4e60a80 0x5556e4e5a110 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92db654740 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5556e4e57ea0 0x5556e4e69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92db654740 1 --2- 172.21.15.57:0/1732267503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5556e4e57580 0x5556e4e68ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 >> 172.21.15.57:0/1732267503 conn(0x5556e4e51cb0 msgr2=0x5556e4e5e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.570+0000 7f92b9ffb640 0 client.26344 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:07.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.573+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 shutdown_connections 2024-04-21T08:54:07.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.573+0000 7f92db654740 1 -- 172.21.15.57:0/1732267503 wait complete. 2024-04-21T08:54:07.573 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.573+0000 7f92db654740 20 client.26344 trim_cache size 0 max 16384 2024-04-21T08:54:07.576 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:54:07.576 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (3775 inodes in the stack)", 2024-04-21T08:54:07.576 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:54:07.576 INFO:teuthology.orchestra.run.smithi057.stdout: "8206b271-55b7-4fbb-8695-a695ac8d2315": { 2024-04-21T08:54:07.577 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:54:07.577 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "8206b271-55b7-4fbb-8695-a695ac8d2315", 2024-04-21T08:54:07.577 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:54:07.577 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:54:07.577 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:54:07.577 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:54:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.579+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f92c00a3f30 msgr2=0x7f92c00a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.579+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f92c00a3f30 0x7f92c00a63f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f92d410a060 tx=0x7f92c4019000 comp rx=0 tx=0).stop 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.579+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4106670 msgr2=0x7f92d4109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.579+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4106670 0x7f92d4109340 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f92cc01ac00 tx=0x7f92cc003ba0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.579+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 shutdown_connections 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.580+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f92c00a3f30 0x7f92c00a63f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.580+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f92d4106670 0x7f92d4109340 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.580+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92d4100f70 0x7f92d4108de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.580+0000 7f92d8a5c640 1 --2- 172.21.15.57:0/2220211098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92d41005b0 0x7f92d41088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:07.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.580+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 >> 172.21.15.57:0/2220211098 conn(0x7f92d40773b0 msgr2=0x7f92d40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.581+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 shutdown_connections 2024-04-21T08:54:07.580 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:07.581+0000 7f92d8a5c640 1 -- 172.21.15.57:0/2220211098 wait complete. 2024-04-21T08:54:07.599 INFO:tasks.cephfs.filesystem:scrub status for tag:8206b271-55b7-4fbb-8695-a695ac8d2315 - {'path': '/', 'tag': '8206b271-55b7-4fbb-8695-a695ac8d2315', 'options': 'recursive,force'} 2024-04-21T08:54:08.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:08 smithi129 ceph-mon[31272]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 54 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.5 MiB/s wr, 675 op/s 2024-04-21T08:54:08.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:08 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:54:08.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:08 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:08.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:08 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:54:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:08 smithi057 ceph-mon[27968]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 54 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.5 MiB/s wr, 675 op/s 2024-04-21T08:54:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:08 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:54:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:08 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:08 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:54:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:08 smithi136 ceph-mon[31791]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 54 active+clean; 1.4 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1.5 MiB/s wr, 675 op/s 2024-04-21T08:54:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:08 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:54:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:08 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:08 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:54:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.234+0000 7f10a5d40640 1 -- 172.21.15.57:0/447382322 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0101120 msgr2=0x7f10a0101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.234+0000 7f10a5d40640 1 --2- 172.21.15.57:0/447382322 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0101120 0x7f10a0101500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1098009ca0 tx=0x7f109802c120 comp rx=0 tx=0).stop 2024-04-21T08:54:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.237+0000 7f10a5d40640 1 -- 172.21.15.57:0/447382322 shutdown_connections 2024-04-21T08:54:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.237+0000 7f10a5d40640 1 --2- 172.21.15.57:0/447382322 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a01061a0 0x7f10a010dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.238+0000 7f10a5d40640 1 --2- 172.21.15.57:0/447382322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a0101ad0 0x7f10a0105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.238+0000 7f10a5d40640 1 --2- 172.21.15.57:0/447382322 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0101120 0x7f10a0101500 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.238+0000 7f10a5d40640 1 -- 172.21.15.57:0/447382322 >> 172.21.15.57:0/447382322 conn(0x7f10a0077fa0 msgr2=0x7f10a00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.240+0000 7f10a5d40640 1 -- 172.21.15.57:0/447382322 shutdown_connections 2024-04-21T08:54:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.243+0000 7f10a5d40640 1 -- 172.21.15.57:0/447382322 wait complete. 2024-04-21T08:54:10.243 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.243+0000 7f10a5d40640 1 Processor -- start 2024-04-21T08:54:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 -- start start 2024-04-21T08:54:10.245 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a0101120 0x7f10a019d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 0x7f10a0196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0196a60 0x7f10a0196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0110140 con 0x7f10a0101120 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10a010ffc0 con 0x7f10a01061a0 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.245+0000 7f10a5d40640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01102c0 con 0x7f10a0196a60 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.246+0000 7f109effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 0x7f10a0196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.246+0000 7f109effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 0x7f10a0196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53778/0 (socket says 172.21.15.57:53778) 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.246+0000 7f109effd640 1 -- 172.21.15.57:0/4253515703 learned_addr learned my addr 172.21.15.57:0/4253515703 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:10.246 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.246+0000 7f109ffff640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0196a60 0x7f10a0196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.248+0000 7f109effd640 1 -- 172.21.15.57:0/4253515703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0196a60 msgr2=0x7f10a0196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.248+0000 7f109effd640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0196a60 0x7f10a0196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.248+0000 7f109effd640 1 -- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a0101120 msgr2=0x7f10a019d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.248+0000 7f109effd640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a0101120 0x7f10a019d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.249+0000 7f109effd640 1 -- 172.21.15.57:0/4253515703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a019afc0 con 0x7f10a01061a0 2024-04-21T08:54:10.255 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.256+0000 7f109effd640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 0x7f10a0196520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f109400b450 tx=0x7f109400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.258+0000 7f109cff9640 1 -- 172.21.15.57:0/4253515703 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094015020 con 0x7f10a01061a0 2024-04-21T08:54:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.258+0000 7f109cff9640 1 -- 172.21.15.57:0/4253515703 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1094011d40 con 0x7f10a01061a0 2024-04-21T08:54:10.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.258+0000 7f109cff9640 1 -- 172.21.15.57:0/4253515703 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109400f340 con 0x7f10a01061a0 2024-04-21T08:54:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.259+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a019b2b0 con 0x7f10a01061a0 2024-04-21T08:54:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.259+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a019b7b0 con 0x7f10a01061a0 2024-04-21T08:54:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.267+0000 7f109cff9640 1 -- 172.21.15.57:0/4253515703 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1094018020 con 0x7f10a01061a0 2024-04-21T08:54:10.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.268+0000 7f10a8a0d740 1 Processor -- start 2024-04-21T08:54:10.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.274+0000 7f10a8a0d740 1 -- start start 2024-04-21T08:54:10.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.274+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb11a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.274 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb11afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1264d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5650eb07f8d0 con 0x5650eb11b8d0 2024-04-21T08:54:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5650eb01ce80 con 0x5650eb1240e0 2024-04-21T08:54:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5650eb07b890 con 0x5650eb11ab60 2024-04-21T08:54:10.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f109cff9640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10880a3fe0 0x7f10880a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f109ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb11a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f109ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb11a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:50362/0 (socket says 172.21.15.57:50362) 2024-04-21T08:54:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f109ffff640 1 -- 172.21.15.57:0/458077261 learned_addr learned my addr 172.21.15.57:0/458077261 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f109ffff640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10880a3fe0 0x7f10880a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.275+0000 7f109cff9640 1 -- 172.21.15.57:0/4253515703 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f10940d1550 con 0x7f10a01061a0 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 -- 172.21.15.57:0/458077261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 msgr2=0x5650eb11afe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb11afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 -- 172.21.15.57:0/458077261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 msgr2=0x5650eb1264d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1264d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 -- 172.21.15.57:0/458077261 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650eb03e320 con 0x5650eb11b8d0 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10880a3fe0 0x7f10880a64a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1090006fd0 tx=0x7f109000a500 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.276+0000 7f109ffff640 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb11a620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f109000a8c0 tx=0x7f1090019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f6eda16cfc6464 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.277+0000 7f1085ffb640 1 -- 172.21.15.57:0/458077261 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090018e60 con 0x5650eb11b8d0 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.277+0000 7f1085ffb640 1 -- 172.21.15.57:0/458077261 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109001c070 con 0x5650eb11b8d0 2024-04-21T08:54:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.277+0000 7f1085ffb640 1 -- 172.21.15.57:0/458077261 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090014430 con 0x5650eb11b8d0 2024-04-21T08:54:10.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.278+0000 7f10a8a0d740 1 -- 172.21.15.57:0/458077261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 msgr2=0x5650eb11a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.279 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.278+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb11a620 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f109000a8c0 tx=0x7f1090019000 comp rx=0 tx=0).stop 2024-04-21T08:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.282+0000 7f10a8a0d740 1 -- 172.21.15.57:0/458077261 shutdown_connections 2024-04-21T08:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.282+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1264d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.282+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb11afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.282+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/458077261 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb11a620 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.282+0000 7f10a8a0d740 1 -- 172.21.15.57:0/458077261 >> 172.21.15.57:0/458077261 conn(0x5650eb0e1c90 msgr2=0x5650eb115850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.286+0000 7f10a8a0d740 1 -- 172.21.15.57:0/458077261 shutdown_connections 2024-04-21T08:54:10.288 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.288+0000 7f10a8a0d740 1 -- 172.21.15.57:0/458077261 wait complete. 2024-04-21T08:54:10.288 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.288+0000 7f10a8a0d740 1 -- start start 2024-04-21T08:54:10.288 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 Processor -- start 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb123b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb120e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1214a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5650eb137dc0 con 0x5650eb11b8d0 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5650eb007eb0 con 0x5650eb1240e0 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5650eb137f40 con 0x5650eb11ab60 2024-04-21T08:54:10.289 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.289+0000 7f10a8a0d740 10 client.0 _ll_register_callbacks cb 0x5650eb034b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.291+0000 7f109ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1214a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.291+0000 7f109ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1214a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53796/0 (socket says 172.21.15.57:53796) 2024-04-21T08:54:10.291 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.291+0000 7f109ffff640 1 -- 172.21.15.57:0/3982489957 learned_addr learned my addr 172.21.15.57:0/3982489957 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.292+0000 7f109ffff640 1 -- 172.21.15.57:0/3982489957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 msgr2=0x5650eb123b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.292+0000 7f109ffff640 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb123b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.292+0000 7f109ffff640 1 -- 172.21.15.57:0/3982489957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 msgr2=0x5650eb120e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.292+0000 7f109ffff640 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb120e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.292 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.292+0000 7f109ffff640 1 -- 172.21.15.57:0/3982489957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650ead92210 con 0x5650eb1240e0 2024-04-21T08:54:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.296+0000 7f109ffff640 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1214a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1090009040 tx=0x7f1090004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90bb8747cc7152d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.297+0000 7f107effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.297+0000 7f107effd640 1 -- 172.21.15.57:0/3982489957 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109002d020 con 0x5650eb1240e0 2024-04-21T08:54:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.297+0000 7f107effd640 1 -- 172.21.15.57:0/3982489957 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f10900065f0 con 0x5650eb1240e0 2024-04-21T08:54:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.297+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f109001c070 con 0x5650eb1240e0 2024-04-21T08:54:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.302+0000 7f107effd640 1 -- 172.21.15.57:0/3982489957 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1090018e60 con 0x5650eb1240e0 2024-04-21T08:54:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.304+0000 7f10a8a0d740 10 client.26459 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:10.303 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.304+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5650eae5e620 con 0x5650eb1240e0 2024-04-21T08:54:10.308 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.308+0000 7f107effd640 1 -- 172.21.15.57:0/3982489957 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109001c070 con 0x5650eb1240e0 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.309+0000 7f107effd640 1 -- 172.21.15.57:0/3982489957 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1090015040 con 0x5650eb1240e0 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.309+0000 7f10a8a0d740 10 client.26459 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.309+0000 7f10a8a0d740 10 client.26459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.309+0000 7f10a8a0d740 20 client.26459 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.309+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb145030 0x5650eb147420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.310+0000 7f10a8a0d740 4 client.26459 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:10.309 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.310+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650eb093880 con 0x5650eb145030 2024-04-21T08:54:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.312+0000 7f109f7fe640 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb145030 0x5650eb147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.324 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.325+0000 7f109f7fe640 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb145030 0x5650eb147420 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b8299de488ee23 server_cookie=46417f04d2ef5376 in_seq=0 out_seq=0 2024-04-21T08:54:10.328 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.325+0000 7f107effd640 10 client.26459 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.329+0000 7f107effd640 1 -- 172.21.15.57:0/3982489957 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5650eb093880 con 0x5650eb145030 2024-04-21T08:54:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.329+0000 7f107effd640 10 client.26459 handle_command_reply: tid=0 2024-04-21T08:54:10.330 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.329+0000 7f10a8a0d740 1 client.26459 shutdown 2024-04-21T08:54:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.334+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 msgr2=0x5650eb1214a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.335+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1214a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1090009040 tx=0x7f1090004730 comp rx=0 tx=0).stop 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.335+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 shutdown_connections 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.335+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb145030 0x5650eb147420 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.335+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb1214a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.335+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11b8d0 0x5650eb120e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.335+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/3982489957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb123b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.336+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 >> 172.21.15.57:0/3982489957 conn(0x5650eb0e1c90 msgr2=0x5650eb1175a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.336+0000 7f107effd640 0 client.26459 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.336+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 shutdown_connections 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.336+0000 7f10a8a0d740 1 -- 172.21.15.57:0/3982489957 wait complete. 2024-04-21T08:54:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.336+0000 7f10a8a0d740 20 client.26459 trim_cache size 0 max 16384 2024-04-21T08:54:10.372 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.372+0000 7f10a8a0d740 1 Processor -- start 2024-04-21T08:54:10.373 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.374+0000 7f10a8a0d740 1 -- start start 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.374+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb140ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.374+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 0x5650eb1235c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.374+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb1240e0 0x5650eb123b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f109ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb1240e0 0x5650eb123b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01025a0 con 0x5650eb1240e0 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01b6410 con 0x5650eb11b8d0 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f10a00735a0 con 0x5650eb11ab60 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f109effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 0x5650eb1235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f109effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 0x5650eb1235c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53798/0 (socket says 172.21.15.57:53798) 2024-04-21T08:54:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.375+0000 7f109effd640 1 -- 172.21.15.57:0/2212348732 learned_addr learned my addr 172.21.15.57:0/2212348732 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.377+0000 7f109f7fe640 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb140ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.378+0000 7f109effd640 1 -- 172.21.15.57:0/2212348732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 msgr2=0x5650eb140ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.379+0000 7f109effd640 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb140ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.379+0000 7f109effd640 1 -- 172.21.15.57:0/2212348732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb1240e0 msgr2=0x5650eb123b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.379+0000 7f109effd640 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb1240e0 0x5650eb123b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.379+0000 7f109effd640 1 -- 172.21.15.57:0/2212348732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650eb119e30 con 0x5650eb11b8d0 2024-04-21T08:54:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.379+0000 7f109effd640 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 0x5650eb1235c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1094000c00 tx=0x7f109409cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2d07b8edf81af8a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.380+0000 7f10857fa640 1 -- 172.21.15.57:0/2212348732 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094099430 con 0x5650eb11b8d0 2024-04-21T08:54:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.380+0000 7f10857fa640 1 -- 172.21.15.57:0/2212348732 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10940a5e00 con 0x5650eb11b8d0 2024-04-21T08:54:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.381+0000 7f10a8a0d740 1 -- 172.21.15.57:0/2212348732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 msgr2=0x5650eb1235c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.381+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 0x5650eb1235c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1094000c00 tx=0x7f109409cc10 comp rx=0 tx=0).stop 2024-04-21T08:54:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.384+0000 7f10a8a0d740 1 -- 172.21.15.57:0/2212348732 shutdown_connections 2024-04-21T08:54:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.384+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb1240e0 0x5650eb123b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.384+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb11b8d0 0x5650eb1235c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.384+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/2212348732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11ab60 0x5650eb140ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.384+0000 7f10a8a0d740 1 -- 172.21.15.57:0/2212348732 >> 172.21.15.57:0/2212348732 conn(0x5650eb0e1c90 msgr2=0x5650eb11c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:10.388 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.388+0000 7f10a8a0d740 1 -- 172.21.15.57:0/2212348732 shutdown_connections 2024-04-21T08:54:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.390+0000 7f10a8a0d740 1 -- 172.21.15.57:0/2212348732 wait complete. 2024-04-21T08:54:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.390+0000 7f10a8a0d740 1 -- start start 2024-04-21T08:54:10.390 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.390+0000 7f10a8a0d740 1 Processor -- start 2024-04-21T08:54:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11ab60 0x5650eb124ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11b8d0 0x5650eb125020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb13c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.392 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10a0073c10 con 0x5650eb11ab60 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01b5d40 con 0x5650eb1240e0 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f10a01b4fa0 con 0x5650eb11b8d0 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f10a8a0d740 10 client.0 _ll_register_callbacks cb 0x7f109000a810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.392+0000 7f109ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb13c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb13c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:53808/0 (socket says 172.21.15.57:53808) 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 -- 172.21.15.57:0/1969245263 learned_addr learned my addr 172.21.15.57:0/1969245263 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109effd640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11b8d0 0x5650eb125020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.393 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109f7fe640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11ab60 0x5650eb124ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 -- 172.21.15.57:0/1969245263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11b8d0 msgr2=0x5650eb125020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11b8d0 0x5650eb125020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 -- 172.21.15.57:0/1969245263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11ab60 msgr2=0x5650eb124ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11ab60 0x5650eb124ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 -- 172.21.15.57:0/1969245263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650eb13c730 con 0x5650eb1240e0 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.393+0000 7f109ffff640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb13c1f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1090004700 tx=0x7f1090020650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=512dc3bde26628ca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.394+0000 7f107effd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.394+0000 7f107effd640 1 -- 172.21.15.57:0/1969245263 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109002d040 con 0x5650eb1240e0 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.394+0000 7f107effd640 1 -- 172.21.15.57:0/1969245263 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7f1090005990 con 0x5650eb1240e0 2024-04-21T08:54:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.394+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10940a5e00 con 0x5650eb1240e0 2024-04-21T08:54:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.395+0000 7f107effd640 1 -- 172.21.15.57:0/1969245263 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1090043050 con 0x5650eb1240e0 2024-04-21T08:54:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.395+0000 7f10a8a0d740 10 client.26465 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.395+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5650eb13da40 con 0x5650eb1240e0 2024-04-21T08:54:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f107effd640 1 -- 172.21.15.57:0/1969245263 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10940a5e00 con 0x5650eb1240e0 2024-04-21T08:54:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f107effd640 1 -- 172.21.15.57:0/1969245263 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f109000f040 con 0x5650eb1240e0 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f10a8a0d740 10 client.26465 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f10a8a0d740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f10a8a0d740 20 client.26465 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb1ab6b0 0x5650eb1adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f10a8a0d740 4 client.26465 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5650eb01f3d0 con 0x5650eb1ab6b0 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.397+0000 7f109f7fe640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb1ab6b0 0x5650eb1adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.398+0000 7f107effd640 10 client.26465 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.398+0000 7f109f7fe640 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb1ab6b0 0x5650eb1adad0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac51cd59abf48c1 server_cookie=81987a45fca77ab6 in_seq=0 out_seq=0 2024-04-21T08:54:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:10 smithi129 ceph-mon[31272]: pgmap v272: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 308 op/s 2024-04-21T08:54:10.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:10 smithi057 ceph-mon[27968]: pgmap v272: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 308 op/s 2024-04-21T08:54:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:10 smithi136 ceph-mon[31791]: pgmap v272: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 308 op/s 2024-04-21T08:54:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.865+0000 7f107effd640 1 -- 172.21.15.57:0/1969245263 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5650eb01f3d0 con 0x5650eb1ab6b0 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.865+0000 7f107effd640 10 client.26465 handle_command_reply: tid=0 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.865+0000 7f10a8a0d740 1 client.26465 shutdown 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.865+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 msgr2=0x5650eb13c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.865+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb13c1f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1090004700 tx=0x7f1090020650 comp rx=0 tx=0).stop 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 shutdown_connections 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5650eb1ab6b0 0x5650eb1adad0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5650eb1240e0 0x5650eb13c1f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5650eb11b8d0 0x5650eb125020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 --2- 172.21.15.57:0/1969245263 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5650eb11ab60 0x5650eb124ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 >> 172.21.15.57:0/1969245263 conn(0x5650eb0e1c90 msgr2=0x5650eb11f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f107effd640 0 client.26465 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 shutdown_connections 2024-04-21T08:54:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 1 -- 172.21.15.57:0/1969245263 wait complete. 2024-04-21T08:54:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.866+0000 7f10a8a0d740 20 client.26465 trim_cache size 0 max 16384 2024-04-21T08:54:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:54:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:54:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:54:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.867+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10880a3fe0 msgr2=0x7f10880a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.867+0000 7f10a5d40640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10880a3fe0 0x7f10880a64a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1090006fd0 tx=0x7f109000a500 comp rx=0 tx=0).stop 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.867+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 msgr2=0x7f10a0196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.867+0000 7f10a5d40640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 0x7f10a0196520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f109400b450 tx=0x7f109400b910 comp rx=0 tx=0).stop 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.867+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 shutdown_connections 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.867+0000 7f10a5d40640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10880a3fe0 0x7f10880a64a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.868+0000 7f10a5d40640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10a0196a60 0x7f10a0196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.868+0000 7f10a5d40640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10a01061a0 0x7f10a0196520 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.868+0000 7f10a5d40640 1 --2- 172.21.15.57:0/4253515703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10a0101120 0x7f10a019d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.868+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 >> 172.21.15.57:0/4253515703 conn(0x7f10a0077fa0 msgr2=0x7f10a00ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.868+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 shutdown_connections 2024-04-21T08:54:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:10.868+0000 7f10a5d40640 1 -- 172.21.15.57:0/4253515703 wait complete. 2024-04-21T08:54:12.500 INFO:tasks.workunit.client.0.smithi057.stdout: CC scripts/mod/empty.o 2024-04-21T08:54:12.503 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/mod/mk_elfconfig 2024-04-21T08:54:12.505 INFO:tasks.workunit.client.0.smithi057.stdout: CC scripts/mod/devicetable-offsets.s 2024-04-21T08:54:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:12 smithi129 ceph-mon[31272]: pgmap v273: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 204 op/s 2024-04-21T08:54:12.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:12 smithi057 ceph-mon[27968]: pgmap v273: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 204 op/s 2024-04-21T08:54:12.592 INFO:tasks.workunit.client.0.smithi057.stdout: MKELF scripts/mod/elfconfig.h 2024-04-21T08:54:12.601 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/mod/modpost.o 2024-04-21T08:54:12.602 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/mod/sumversion.o 2024-04-21T08:54:12.619 INFO:tasks.workunit.client.0.smithi057.stdout: UPD scripts/mod/devicetable-offsets.h 2024-04-21T08:54:12.629 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC scripts/mod/file2alias.o 2024-04-21T08:54:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:12 smithi136 ceph-mon[31791]: pgmap v273: 129 pgs: 18 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 62 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 204 op/s 2024-04-21T08:54:13.290 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTLD scripts/mod/modpost 2024-04-21T08:54:13.340 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/bounds.s 2024-04-21T08:54:13.341 INFO:tasks.workunit.client.0.smithi057.stdout: CHKSHA1 include/linux/atomic/atomic-instrumented.h 2024-04-21T08:54:13.341 INFO:tasks.workunit.client.0.smithi057.stdout: CHKSHA1 include/linux/atomic/atomic-arch-fallback.h 2024-04-21T08:54:13.342 INFO:tasks.workunit.client.0.smithi057.stdout: CHKSHA1 include/linux/atomic/atomic-long.h 2024-04-21T08:54:13.428 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/generated/bounds.h 2024-04-21T08:54:13.429 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/generated/timeconst.h 2024-04-21T08:54:13.439 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/asm-offsets.s 2024-04-21T08:54:14.007 INFO:tasks.workunit.client.0.smithi057.stdout: UPD include/generated/asm-offsets.h 2024-04-21T08:54:14.013 INFO:tasks.workunit.client.0.smithi057.stdout: CALL scripts/checksyscalls.sh 2024-04-21T08:54:14.067 INFO:tasks.workunit.client.0.smithi057.stdout: LDS scripts/module.lds 2024-04-21T08:54:14.178 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/main.o 2024-04-21T08:54:14.180 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC usr/gen_init_cpio 2024-04-21T08:54:14.184 INFO:tasks.workunit.client.0.smithi057.stdout: CC certs/system_keyring.o 2024-04-21T08:54:14.184 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/compat.o 2024-04-21T08:54:14.186 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/filemap.o 2024-04-21T08:54:14.214 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/dnotify/dnotify.o 2024-04-21T08:54:14.301 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vma.o 2024-04-21T08:54:14.477 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sched/core.o 2024-04-21T08:54:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:14 smithi129 ceph-mon[31272]: pgmap v274: 129 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 99 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 497 op/s 2024-04-21T08:54:14.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:14 smithi057 ceph-mon[27968]: pgmap v274: 129 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 99 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 497 op/s 2024-04-21T08:54:14.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:14 smithi136 ceph-mon[31791]: pgmap v274: 129 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 99 active+clean; 1.4 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 497 op/s 2024-04-21T08:54:14.636 INFO:tasks.workunit.client.0.smithi057.stdout: GEN usr/initramfs_data.cpio 2024-04-21T08:54:14.670 INFO:tasks.workunit.client.0.smithi057.stdout: COPY usr/initramfs_inc_data 2024-04-21T08:54:14.684 INFO:tasks.workunit.client.0.smithi057.stdout: AS usr/initramfs_data.o 2024-04-21T08:54:14.695 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC certs/extract-cert 2024-04-21T08:54:14.809 INFO:tasks.workunit.client.0.smithi057.stdout: AR usr/built-in.a 2024-04-21T08:54:14.936 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/amd/core.o 2024-04-21T08:54:14.991 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/notify/dnotify/built-in.a 2024-04-21T08:54:14.995 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/util.o 2024-04-21T08:54:15.053 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/inotify/inotify_fsnotify.o 2024-04-21T08:54:15.108 INFO:tasks.workunit.client.0.smithi057.stdout: CERT certs/x509_certificate_list 2024-04-21T08:54:15.117 INFO:tasks.workunit.client.0.smithi057.stdout: CERT certs/signing_key.x509 2024-04-21T08:54:15.128 INFO:tasks.workunit.client.0.smithi057.stdout: AS certs/system_certificates.o 2024-04-21T08:54:15.136 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/extable.o 2024-04-21T08:54:15.192 INFO:tasks.workunit.client.0.smithi057.stdout: AR certs/built-in.a 2024-04-21T08:54:15.272 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vdso32-setup.o 2024-04-21T08:54:15.592 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/inotify/inotify_user.o 2024-04-21T08:54:15.595 INFO:tasks.workunit.client.0.smithi057.stdout: UPD init/utsversion-tmp.h 2024-04-21T08:54:15.609 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/do_mounts.o 2024-04-21T08:54:15.617 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/gc.o 2024-04-21T08:54:15.843 INFO:tasks.workunit.client.0.smithi057.stdout: LDS arch/x86/entry/vdso/vdso.lds 2024-04-21T08:54:15.898 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/vdso/vdso-note.o 2024-04-21T08:54:15.924 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/msgutil.o 2024-04-21T08:54:15.935 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vclock_gettime.o 2024-04-21T08:54:16.011 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/amd/lbr.o 2024-04-21T08:54:16.433 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/key.o 2024-04-21T08:54:16.477 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vgetcpu.o 2024-04-21T08:54:16.552 INFO:tasks.workunit.client.0.smithi057.stdout: HOSTCC arch/x86/entry/vdso/vdso2c 2024-04-21T08:54:16.568 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/notify/inotify/built-in.a 2024-04-21T08:54:16.575 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:16 smithi129 ceph-mon[31272]: pgmap v275: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 640 KiB/s wr, 809 op/s 2024-04-21T08:54:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:16 smithi057 ceph-mon[27968]: pgmap v275: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 640 KiB/s wr, 809 op/s 2024-04-21T08:54:16.591 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/msg.o 2024-04-21T08:54:16.622 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/notify/fanotify/built-in.a 2024-04-21T08:54:16.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:16 smithi136 ceph-mon[31791]: pgmap v275: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 640 KiB/s wr, 809 op/s 2024-04-21T08:54:16.644 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/fsnotify.o 2024-04-21T08:54:16.748 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/amd/ibs.o 2024-04-21T08:54:16.855 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mempool.o 2024-04-21T08:54:16.909 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/do_mounts_initrd.o 2024-04-21T08:54:16.983 INFO:tasks.workunit.client.0.smithi057.stdout: LDS arch/x86/entry/vdso/vdso32/vdso32.lds 2024-04-21T08:54:17.037 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/vdso/vdso32/note.o 2024-04-21T08:54:17.094 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/vdso/vdso32/system_call.o 2024-04-21T08:54:17.144 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/vdso/vdso32/sigreturn.o 2024-04-21T08:54:17.186 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vdso32/vclock_gettime.o 2024-04-21T08:54:17.325 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/keyring.o 2024-04-21T08:54:17.556 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/notification.o 2024-04-21T08:54:17.600 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/oom_kill.o 2024-04-21T08:54:17.681 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/initramfs.o 2024-04-21T08:54:17.737 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/sem.o 2024-04-21T08:54:17.880 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/amd/uncore.o 2024-04-21T08:54:17.919 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vdso32/vgetcpu.o 2024-04-21T08:54:18.282 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/group.o 2024-04-21T08:54:18.353 INFO:tasks.workunit.client.0.smithi057.stdout: VDSO arch/x86/entry/vdso/vdso64.so.dbg 2024-04-21T08:54:18.507 INFO:tasks.workunit.client.0.smithi057.stdout: VDSO arch/x86/entry/vdso/vdso32.so.dbg 2024-04-21T08:54:18.561 INFO:tasks.workunit.client.0.smithi057.stdout: OBJCOPY arch/x86/entry/vdso/vdso64.so 2024-04-21T08:54:18.569 INFO:tasks.workunit.client.0.smithi057.stdout: OBJCOPY arch/x86/entry/vdso/vdso32.so 2024-04-21T08:54:18.577 INFO:tasks.workunit.client.0.smithi057.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-64.c 2024-04-21T08:54:18.588 INFO:tasks.workunit.client.0.smithi057.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-32.c 2024-04-21T08:54:18.597 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vdso-image-64.o 2024-04-21T08:54:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:18 smithi136 ceph-mon[31791]: pgmap v276: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 636 KiB/s wr, 1.07k op/s 2024-04-21T08:54:18.717 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/keyctl.o 2024-04-21T08:54:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:18 smithi129 ceph-mon[31272]: pgmap v276: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 636 KiB/s wr, 1.07k op/s 2024-04-21T08:54:18.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:18 smithi057 ceph-mon[27968]: pgmap v276: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 636 KiB/s wr, 1.07k op/s 2024-04-21T08:54:18.894 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/calibrate.o 2024-04-21T08:54:18.897 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vdso/vdso-image-32.o 2024-04-21T08:54:19.002 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/amd/iommu.o 2024-04-21T08:54:19.030 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/mark.o 2024-04-21T08:54:19.195 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/entry/vdso/built-in.a 2024-04-21T08:54:19.263 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/init_task.o 2024-04-21T08:54:19.372 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/fadvise.o 2024-04-21T08:54:19.397 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/vsyscall/vsyscall_64.o 2024-04-21T08:54:19.409 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sched/fair.o 2024-04-21T08:54:19.447 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/shm.o 2024-04-21T08:54:19.840 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/permission.o 2024-04-21T08:54:19.881 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/events/amd/built-in.a 2024-04-21T08:54:19.953 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/core.o 2024-04-21T08:54:19.990 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/notify/fdinfo.o 2024-04-21T08:54:20.024 INFO:tasks.workunit.client.0.smithi057.stdout: CC init/version.o 2024-04-21T08:54:20.191 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/maccess.o 2024-04-21T08:54:20.243 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/vsyscall/vsyscall_emu_64.o 2024-04-21T08:54:20.305 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/entry/vsyscall/built-in.a 2024-04-21T08:54:20.329 INFO:tasks.workunit.client.0.smithi057.stdout: AR init/built-in.a 2024-04-21T08:54:20.360 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/entry.o 2024-04-21T08:54:20.365 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/entry_64.o 2024-04-21T08:54:20.397 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/syscall.o 2024-04-21T08:54:20.459 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/syscall_64.o 2024-04-21T08:54:20.497 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/process_keys.o 2024-04-21T08:54:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:20 smithi136 ceph-mon[31791]: pgmap v277: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 730 KiB/s wr, 1.96k op/s 2024-04-21T08:54:20.685 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/notify/built-in.a 2024-04-21T08:54:20.699 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/bts.o 2024-04-21T08:54:20.739 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs_common/nfsacl.o 2024-04-21T08:54:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:20 smithi129 ceph-mon[31272]: pgmap v277: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 730 KiB/s wr, 1.96k op/s 2024-04-21T08:54:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:20 smithi057 ceph-mon[27968]: pgmap v277: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 730 KiB/s wr, 1.96k op/s 2024-04-21T08:54:20.958 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/page-writeback.o 2024-04-21T08:54:21.156 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/ipc_sysctl.o 2024-04-21T08:54:21.190 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/common.o 2024-04-21T08:54:21.409 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs_common/grace.o 2024-04-21T08:54:21.524 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/request_key.o 2024-04-21T08:54:21.582 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/folio-compat.o 2024-04-21T08:54:21.845 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/mqueue.o 2024-04-21T08:54:21.938 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/thunk_64.o 2024-04-21T08:54:21.989 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/entry/entry_64_compat.o 2024-04-21T08:54:22.065 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/entry/syscall_32.o 2024-04-21T08:54:22.180 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/nfs_common/built-in.a 2024-04-21T08:54:22.237 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/trace.o 2024-04-21T08:54:22.395 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/ds.o 2024-04-21T08:54:22.409 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/request_key_auth.o 2024-04-21T08:54:22.488 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/knc.o 2024-04-21T08:54:22.610 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/readahead.o 2024-04-21T08:54:22.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:22 smithi129 ceph-mon[31272]: pgmap v278: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 642 KiB/s wr, 2.51k op/s 2024-04-21T08:54:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:22 smithi057 ceph-mon[27968]: pgmap v278: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 642 KiB/s wr, 2.51k op/s 2024-04-21T08:54:22.848 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/entry/built-in.a 2024-04-21T08:54:22.884 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sched/build_policy.o 2024-04-21T08:54:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:22 smithi136 ceph-mon[31791]: pgmap v278: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 642 KiB/s wr, 2.51k op/s 2024-04-21T08:54:22.903 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/swap.o 2024-04-21T08:54:23.069 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/user_defined.o 2024-04-21T08:54:23.283 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/compat.o 2024-04-21T08:54:23.328 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/iter.o 2024-04-21T08:54:23.537 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/namespace.o 2024-04-21T08:54:23.631 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/buffered-io.o 2024-04-21T08:54:23.737 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/proc.o 2024-04-21T08:54:24.075 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/sysctl.o 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 -- 172.21.15.57:0/2483756809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 msgr2=0x7feb30108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 --2- 172.21.15.57:0/2483756809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 0x7feb30108f50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feb24009900 tx=0x7feb2402bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 -- 172.21.15.57:0/2483756809 shutdown_connections 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 --2- 172.21.15.57:0/2483756809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 0x7feb30108f50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 --2- 172.21.15.57:0/2483756809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb3010e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 --2- 172.21.15.57:0/2483756809 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 0x7feb301108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 -- 172.21.15.57:0/2483756809 >> 172.21.15.57:0/2483756809 conn(0x7feb3006cf00 msgr2=0x7feb3006d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.108+0000 7feb35a11640 1 -- 172.21.15.57:0/2483756809 shutdown_connections 2024-04-21T08:54:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.109+0000 7feb35a11640 1 -- 172.21.15.57:0/2483756809 wait complete. 2024-04-21T08:54:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.109+0000 7feb35a11640 1 Processor -- start 2024-04-21T08:54:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 -- start start 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb301a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 0x7feb301a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 0x7feb301abb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feb30120e60 con 0x7feb301104c0 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb30120ce0 con 0x7feb3010ecc0 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.110+0000 7feb35a11640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feb30120fe0 con 0x7feb3010e220 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb301a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb301a7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:32926/0 (socket says 172.21.15.57:32926) 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 -- 172.21.15.57:0/3450778719 learned_addr learned my addr 172.21.15.57:0/3450778719 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 0x7feb301a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 0x7feb301abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 -- 172.21.15.57:0/3450778719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 msgr2=0x7feb301a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 0x7feb301a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 -- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 msgr2=0x7feb301abb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 0x7feb301abb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2effd640 1 -- 172.21.15.57:0/3450778719 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb301ac270 con 0x7feb3010e220 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.111+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 0x7feb301abb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.112+0000 7feb2effd640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb301a7280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb2800eab0 tx=0x7feb2800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.112+0000 7feb34a0f640 1 -- 172.21.15.57:0/3450778719 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2800ce40 con 0x7feb3010e220 2024-04-21T08:54:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.112+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb301ac560 con 0x7feb3010e220 2024-04-21T08:54:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.112+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb301b3df0 con 0x7feb3010e220 2024-04-21T08:54:24.115 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.115+0000 7feb34a0f640 1 -- 172.21.15.57:0/3450778719 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb28011920 con 0x7feb3010e220 2024-04-21T08:54:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.116+0000 7feb34a0f640 1 -- 172.21.15.57:0/3450778719 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28010a30 con 0x7feb3010e220 2024-04-21T08:54:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.116+0000 7feb34a0f640 1 -- 172.21.15.57:0/3450778719 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feb28010c50 con 0x7feb3010e220 2024-04-21T08:54:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 Processor -- start 2024-04-21T08:54:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 -- start start 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a28810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a28d50 0x55e014a291d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a322d0 0x55e014a346c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e01498d8d0 con 0x55e014a29ac0 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e01492ae80 con 0x55e014a322d0 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb38600740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e014989890 con 0x55e014a28d50 2024-04-21T08:54:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb2f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a28810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb2f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a28810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54156/0 (socket says 172.21.15.57:54156) 2024-04-21T08:54:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb2f7fe640 1 -- 172.21.15.57:0/456725341 learned_addr learned my addr 172.21.15.57:0/456725341 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.117+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a322d0 0x55e014a346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2effd640 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a28d50 0x55e014a291d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2e7fc640 1 -- 172.21.15.57:0/456725341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a28d50 msgr2=0x55e014a291d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a28d50 0x55e014a291d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2e7fc640 1 -- 172.21.15.57:0/456725341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 msgr2=0x55e014a28810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a28810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2e7fc640 1 -- 172.21.15.57:0/456725341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01494c320 con 0x55e014a322d0 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.118+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a322d0 0x55e014a346c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb3006dbd0 tx=0x7feb200117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec03f3153e8bc74c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.119+0000 7feb34a0f640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7feb180a3db0 0x7feb180a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.119+0000 7feb34a0f640 1 -- 172.21.15.57:0/3450778719 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7feb280d2140 con 0x7feb3010e220 2024-04-21T08:54:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.120+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7feb180a3db0 0x7feb180a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.124 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.124+0000 7feb15ffb640 1 -- 172.21.15.57:0/456725341 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20004030 con 0x55e014a322d0 2024-04-21T08:54:24.124 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.124+0000 7feb15ffb640 1 -- 172.21.15.57:0/456725341 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb200041d0 con 0x55e014a322d0 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 -- 172.21.15.57:0/456725341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a322d0 msgr2=0x55e014a346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7feb180a3db0 0x7feb180a6270 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb24006010 tx=0x7feb2402c150 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a322d0 0x55e014a346c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7feb3006dbd0 tx=0x7feb200117b0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 -- 172.21.15.57:0/456725341 shutdown_connections 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a322d0 0x55e014a346c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a28d50 0x55e014a291d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 --2- 172.21.15.57:0/456725341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a28810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 -- 172.21.15.57:0/456725341 >> 172.21.15.57:0/456725341 conn(0x55e0149efc90 msgr2=0x55e014a23a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:24.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 -- 172.21.15.57:0/456725341 shutdown_connections 2024-04-21T08:54:24.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.125+0000 7feb38600740 1 -- 172.21.15.57:0/456725341 wait complete. 2024-04-21T08:54:24.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 -- start start 2024-04-21T08:54:24.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 Processor -- start 2024-04-21T08:54:24.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 0x55e014a31b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a29ac0 0x55e014a2ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a2f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55e014a35050 con 0x55e014a28d50 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e014915eb0 con 0x55e014a29ac0 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.126+0000 7feb38600740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55e014a351d0 con 0x55e014a322d0 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.127+0000 7feb38600740 10 client.0 _ll_register_callbacks cb 0x55e014942b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.127+0000 7feb2effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 0x55e014a31b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.127+0000 7feb2effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 0x55e014a31b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54166/0 (socket says 172.21.15.57:54166) 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.127+0000 7feb2effd640 1 -- 172.21.15.57:0/3619231715 learned_addr learned my addr 172.21.15.57:0/3619231715 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.127+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a29ac0 0x55e014a2ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.127+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a2f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb2effd640 1 -- 172.21.15.57:0/3619231715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 msgr2=0x55e014a2f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb2effd640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a2f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb2effd640 1 -- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a29ac0 msgr2=0x55e014a2ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb2effd640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a29ac0 0x55e014a2ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb2effd640 1 -- 172.21.15.57:0/3619231715 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0146a0210 con 0x55e014a28d50 2024-04-21T08:54:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb2effd640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 0x55e014a31b80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7feb280cf500 tx=0x7feb28015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad9a435e1f73bf server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb137f6640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb137f6640 1 -- 172.21.15.57:0/3619231715 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb280a66d0 con 0x55e014a28d50 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb200041d0 con 0x55e014a28d50 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.128+0000 7feb137f6640 1 -- 172.21.15.57:0/3619231715 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7feb280114c0 con 0x55e014a28d50 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.129+0000 7feb137f6640 1 -- 172.21.15.57:0/3619231715 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feb2809dcf0 con 0x55e014a28d50 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.129+0000 7feb38600740 10 client.17013 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.129+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55e01476c620 con 0x55e014a28d50 2024-04-21T08:54:24.134 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.129+0000 7feb137f6640 1 -- 172.21.15.57:0/3619231715 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb200041d0 con 0x55e014a28d50 2024-04-21T08:54:24.134 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.134+0000 7feb137f6640 1 -- 172.21.15.57:0/3619231715 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7feb2800f040 con 0x55e014a28d50 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb38600740 10 client.17013 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb38600740 10 client.17013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb38600740 20 client.17013 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb38600740 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014a44770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb38600740 4 client.17013 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0149a1880 con 0x55e014a42340 2024-04-21T08:54:24.137 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.137+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014a44770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.138 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.138+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014a44770 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f60223ea367476 server_cookie=e8dfdcfd5ecf5ac9 in_seq=0 out_seq=0 2024-04-21T08:54:24.138 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.138+0000 7feb137f6640 10 client.17013 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:24.138 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb137f6640 1 -- 172.21.15.57:0/3619231715 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0149a1880 con 0x55e014a42340 2024-04-21T08:54:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb137f6640 10 client.17013 handle_command_reply: tid=0 2024-04-21T08:54:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb38600740 1 client.17013 shutdown 2024-04-21T08:54:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 msgr2=0x55e014a31b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:24.139 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb38600740 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 0x55e014a31b80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7feb280cf500 tx=0x7feb28015000 comp rx=0 tx=0).stop 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 shutdown_connections 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.139+0000 7feb38600740 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014a44770 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a2f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb137f6640 0 client.17013 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a29ac0 0x55e014a2ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 1 --2- 172.21.15.57:0/3619231715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a28d50 0x55e014a31b80 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 >> 172.21.15.57:0/3619231715 conn(0x55e0149efc90 msgr2=0x55e014a25790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 shutdown_connections 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 1 -- 172.21.15.57:0/3619231715 wait complete. 2024-04-21T08:54:24.140 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.140+0000 7feb38600740 20 client.17013 trim_cache size 0 max 16384 2024-04-21T08:54:24.165 INFO:tasks.workunit.client.0.smithi057.stdout: CC ipc/mq_sysctl.o 2024-04-21T08:54:24.169 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/lbr.o 2024-04-21T08:54:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.173+0000 7feb38600740 1 Processor -- start 2024-04-21T08:54:24.177 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/p4.o 2024-04-21T08:54:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.180+0000 7feb38600740 1 -- start start 2024-04-21T08:54:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.180+0000 7feb38600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a22960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.180+0000 7feb38600740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a31640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.180+0000 7feb38600740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a31b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a31640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a31640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54182/0 (socket says 172.21.15.57:54182) 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb2e7fc640 1 -- 172.21.15.57:0/1343651976 learned_addr learned my addr 172.21.15.57:0/1343651976 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb2effd640 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a22960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c00e090 con 0x55e014a29ac0 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a31b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c0108f0 con 0x55e014a28d50 2024-04-21T08:54:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c00e6e0 con 0x55e014a322d0 2024-04-21T08:54:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.181+0000 7feb2e7fc640 1 -- 172.21.15.57:0/1343651976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 msgr2=0x55e014a31b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.182+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a31b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.182+0000 7feb2e7fc640 1 -- 172.21.15.57:0/1343651976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 msgr2=0x55e014a22960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.182+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a22960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.182+0000 7feb2e7fc640 1 -- 172.21.15.57:0/1343651976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e014a28020 con 0x55e014a29ac0 2024-04-21T08:54:24.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.182+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a31640 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7feb3006dbd0 tx=0x7feb20009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70031dddf957b04 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:24.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.183+0000 7feb157fa640 1 -- 172.21.15.57:0/1343651976 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20031070 con 0x55e014a29ac0 2024-04-21T08:54:24.189 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.183+0000 7feb157fa640 1 -- 172.21.15.57:0/1343651976 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb2002a030 con 0x55e014a29ac0 2024-04-21T08:54:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.183+0000 7feb157fa640 1 -- 172.21.15.57:0/1343651976 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2001f040 con 0x55e014a29ac0 2024-04-21T08:54:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 msgr2=0x55e014a31640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a31640 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7feb3006dbd0 tx=0x7feb20009540 comp rx=0 tx=0).stop 2024-04-21T08:54:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 shutdown_connections 2024-04-21T08:54:24.190 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a31b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a31640 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 --2- 172.21.15.57:0/1343651976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a22960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 >> 172.21.15.57:0/1343651976 conn(0x55e0149efc90 msgr2=0x55e014a22250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 shutdown_connections 2024-04-21T08:54:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.184+0000 7feb38600740 1 -- 172.21.15.57:0/1343651976 wait complete. 2024-04-21T08:54:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.185+0000 7feb38600740 1 -- start start 2024-04-21T08:54:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.185+0000 7feb38600740 1 Processor -- start 2024-04-21T08:54:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a33040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a33580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb2effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a33040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a33ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb2effd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a33040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:56000/0 (socket says 172.21.15.57:56000) 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a33580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb2e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a33580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:54194/0 (socket says 172.21.15.57:54194) 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb2effd640 1 -- 172.21.15.57:0/1572521559 learned_addr learned my addr 172.21.15.57:0/1572521559 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c00edb0 con 0x55e014a29ac0 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c010220 con 0x55e014a28d50 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c00f480 con 0x55e014a322d0 2024-04-21T08:54:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb2f7fe640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a33ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.186+0000 7feb38600740 10 client.0 _ll_register_callbacks cb 0x7feb2809dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2effd640 1 -- 172.21.15.57:0/1572521559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 msgr2=0x55e014a33ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2effd640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a33ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2effd640 1 -- 172.21.15.57:0/1572521559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 msgr2=0x55e014a33580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2effd640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a33580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2effd640 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e014a38b40 con 0x55e014a28d50 2024-04-21T08:54:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a33580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:24.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb2002a030 con 0x55e014a28d50 2024-04-21T08:54:24.195 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.187+0000 7feb2effd640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a33040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb2800ea80 tx=0x7feb2809dbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6053dc72c246e9a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.200+0000 7feb137f6640 10 client.26483 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.200+0000 7feb137f6640 1 -- 172.21.15.57:0/1572521559 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2809d650 con 0x55e014a28d50 2024-04-21T08:54:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.200+0000 7feb137f6640 1 -- 172.21.15.57:0/1572521559 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9563+0+0 (secure 0 0 0) 0x7feb28011470 con 0x55e014a28d50 2024-04-21T08:54:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.200+0000 7feb137f6640 1 -- 172.21.15.57:0/1572521559 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7feb280116b0 con 0x55e014a28d50 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 10 client.26483 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55e014aa8ce0 con 0x55e014a28d50 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb137f6640 1 -- 172.21.15.57:0/1572521559 <== mon.2 v2:172.21.15.129:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7feb2809d140 con 0x55e014a28d50 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 10 client.26483 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 10 client.26483 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 20 client.26483 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014aab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 4 client.26483 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e01492d3d0 con 0x55e014a42340 2024-04-21T08:54:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.201+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014aab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.203+0000 7feb2e7fc640 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014aab1a0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4508e0217f433b98 server_cookie=1b5b6bd3309ac35a in_seq=0 out_seq=0 2024-04-21T08:54:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:24.203+0000 7feb137f6640 10 client.26483 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:24.543 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/keys/keyctl_pkey.o 2024-04-21T08:54:24.772 INFO:tasks.workunit.client.0.smithi057.stdout: AR ipc/built-in.a 2024-04-21T08:54:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:24 smithi129 ceph-mon[31272]: pgmap v279: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 941 KiB/s wr, 3.48k op/s 2024-04-21T08:54:24.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:24 smithi057 ceph-mon[27968]: pgmap v279: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 941 KiB/s wr, 3.48k op/s 2024-04-21T08:54:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:24 smithi136 ceph-mon[31791]: pgmap v279: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 941 KiB/s wr, 3.48k op/s 2024-04-21T08:54:25.026 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/init.o 2024-04-21T08:54:25.105 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/truncate.o 2024-04-21T08:54:25.252 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/p6.o 2024-04-21T08:54:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.357+0000 7feb137f6640 1 -- 172.21.15.57:0/1572521559 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e01492d3d0 con 0x55e014a42340 2024-04-21T08:54:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.357+0000 7feb137f6640 10 client.26483 handle_command_reply: tid=0 2024-04-21T08:54:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.357+0000 7feb38600740 1 client.26483 shutdown 2024-04-21T08:54:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 msgr2=0x55e014a33040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:25.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a33040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7feb2800ea80 tx=0x7feb2809dbf0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 shutdown_connections 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55e014a42340 0x55e014aab1a0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55e014a322d0 0x55e014a33ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55e014a29ac0 0x55e014a33580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 --2- 172.21.15.57:0/1572521559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e014a28d50 0x55e014a33040 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.358+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 >> 172.21.15.57:0/1572521559 conn(0x55e0149efc90 msgr2=0x55e014a22ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.359+0000 7feb137f6640 0 client.26483 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.359+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 shutdown_connections 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.359+0000 7feb38600740 1 -- 172.21.15.57:0/1572521559 wait complete. 2024-04-21T08:54:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.359+0000 7feb38600740 20 client.26483 trim_cache size 0 max 16384 2024-04-21T08:54:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:54:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:54:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:54:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:54:25.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.365+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7feb180a3db0 msgr2=0x7feb180a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:25.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.365+0000 7feb35a11640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7feb180a3db0 0x7feb180a6270 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb24006010 tx=0x7feb2402c150 comp rx=0 tx=0).stop 2024-04-21T08:54:25.366 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.365+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 msgr2=0x7feb301a7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.365+0000 7feb35a11640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb301a7280 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7feb2800eab0 tx=0x7feb2800ef70 comp rx=0 tx=0).stop 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.366+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 shutdown_connections 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.366+0000 7feb35a11640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7feb180a3db0 0x7feb180a6270 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.366+0000 7feb35a11640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feb301104c0 0x7feb301abb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.366+0000 7feb35a11640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb3010ecc0 0x7feb301a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.366+0000 7feb35a11640 1 --2- 172.21.15.57:0/3450778719 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb3010e220 0x7feb301a7280 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:25.367 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.366+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 >> 172.21.15.57:0/3450778719 conn(0x7feb3006cf00 msgr2=0x7feb3010a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:25.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.370+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 shutdown_connections 2024-04-21T08:54:25.370 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:25.371+0000 7feb35a11640 1 -- 172.21.15.57:0/3450778719 wait complete. 2024-04-21T08:54:25.402 INFO:tasks.workunit.client.0.smithi057.stdout: AR security/keys/built-in.a 2024-04-21T08:54:25.412 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/asymmetric_type.o 2024-04-21T08:54:25.468 INFO:tasks.workunit.client.0.smithi057.stdout: GEN security/selinux/flask.h security/selinux/av_permissions.h 2024-04-21T08:54:25.547 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/avc.o 2024-04-21T08:54:25.582 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/restrict.o 2024-04-21T08:54:25.668 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/direct-io.o 2024-04-21T08:54:25.777 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/header.o 2024-04-21T08:54:25.921 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/trampoline_64.o 2024-04-21T08:54:25.969 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/pt.o 2024-04-21T08:54:25.974 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/stack.o 2024-04-21T08:54:26.013 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/signature.o 2024-04-21T08:54:26.017 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/reboot.o 2024-04-21T08:54:26.077 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/wakeup_asm.o 2024-04-21T08:54:26.113 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/vmscan.o 2024-04-21T08:54:26.137 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/rm/wakemain.o 2024-04-21T08:54:26.148 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/rm/video-mode.o 2024-04-21T08:54:26.260 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/public_key.o 2024-04-21T08:54:26.319 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/copy.o 2024-04-21T08:54:26.352 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rm/bioscall.o 2024-04-21T08:54:26.385 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sched/build_utility.o 2024-04-21T08:54:26.391 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/rm/regs.o 2024-04-21T08:54:26.463 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/rm/video-vga.o 2024-04-21T08:54:26.466 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/rm/video-vesa.o 2024-04-21T08:54:26.554 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/fiemap.o 2024-04-21T08:54:26.559 INFO:tasks.workunit.client.0.smithi057.stdout: ASN.1 crypto/asymmetric_keys/x509.asn1.[ch] 2024-04-21T08:54:26.559 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/realmode/rm/video-bios.o 2024-04-21T08:54:26.573 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/shmem.o 2024-04-21T08:54:26.648 INFO:tasks.workunit.client.0.smithi057.stdout: PASYMS arch/x86/realmode/rm/pasyms.h 2024-04-21T08:54:26.686 INFO:tasks.workunit.client.0.smithi057.stdout: LDS arch/x86/realmode/rm/realmode.lds 2024-04-21T08:54:26.730 INFO:tasks.workunit.client.0.smithi057.stdout: LD arch/x86/realmode/rm/realmode.elf 2024-04-21T08:54:26.751 INFO:tasks.workunit.client.0.smithi057.stdout: RELOCS arch/x86/realmode/rm/realmode.relocs 2024-04-21T08:54:26.762 INFO:tasks.workunit.client.0.smithi057.stdout: OBJCOPY arch/x86/realmode/rm/realmode.bin 2024-04-21T08:54:26.774 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/realmode/rmpiggy.o 2024-04-21T08:54:26.821 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/realmode/built-in.a 2024-04-21T08:54:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:26 smithi129 ceph-mon[31272]: pgmap v280: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 632 KiB/s wr, 4.13k op/s 2024-04-21T08:54:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:26 smithi129 ceph-mon[31272]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:54:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:26 smithi057 ceph-mon[27968]: pgmap v280: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 632 KiB/s wr, 4.13k op/s 2024-04-21T08:54:26.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:26 smithi057 ceph-mon[27968]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:54:26.881 INFO:tasks.workunit.client.0.smithi057.stdout: ASN.1 crypto/asymmetric_keys/x509_akid.asn1.[ch] 2024-04-21T08:54:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:26 smithi136 ceph-mon[31791]: pgmap v280: 129 pgs: 129 active+clean; 1.4 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 632 KiB/s wr, 4.13k op/s 2024-04-21T08:54:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:26 smithi136 ceph-mon[31791]: osdmap e102: 12 total, 12 up, 12 in 2024-04-21T08:54:26.917 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/fpu/init.o 2024-04-21T08:54:26.934 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/x509_loader.o 2024-04-21T08:54:27.134 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/uncore.o 2024-04-21T08:54:27.159 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/hooks.o 2024-04-21T08:54:27.179 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/x509_public_key.o 2024-04-21T08:54:27.196 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/seek.o 2024-04-21T08:54:27.336 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/fpu/bugs.o 2024-04-21T08:54:27.596 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/fpu/core.o 2024-04-21T08:54:27.750 INFO:tasks.workunit.client.0.smithi057.stdout: ASN.1 crypto/asymmetric_keys/pkcs7.asn1.[ch] 2024-04-21T08:54:27.761 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/pkcs7_trust.o 2024-04-21T08:54:27.975 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/iomap/swapfile.o 2024-04-21T08:54:28.168 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/pkcs7_verify.o 2024-04-21T08:54:28.787 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/x509.asn1.o 2024-04-21T08:54:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:28 smithi129 ceph-mon[31272]: pgmap v282: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 793 KiB/s wr, 4.81k op/s 2024-04-21T08:54:28.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:28 smithi057 ceph-mon[27968]: pgmap v282: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 793 KiB/s wr, 4.81k op/s 2024-04-21T08:54:28.879 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/uncore_nhmex.o 2024-04-21T08:54:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:28 smithi136 ceph-mon[31791]: pgmap v282: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 793 KiB/s wr, 4.81k op/s 2024-04-21T08:54:28.901 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/fpu/regset.o 2024-04-21T08:54:28.901 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/x509_akid.asn1.o 2024-04-21T08:54:29.031 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/x509_cert_parser.o 2024-04-21T08:54:29.122 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pat/set_memory.o 2024-04-21T08:54:29.131 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/iomap/built-in.a 2024-04-21T08:54:29.320 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/quota/dquot.o 2024-04-21T08:54:29.328 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/util.o 2024-04-21T08:54:29.706 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/pkcs7.asn1.o 2024-04-21T08:54:29.753 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/asymmetric_keys/pkcs7_parser.o 2024-04-21T08:54:29.756 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/fpu/signal.o 2024-04-21T08:54:30.161 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/sched/built-in.a 2024-04-21T08:54:30.238 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/mutex.o 2024-04-21T08:54:30.274 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/uncore_snb.o 2024-04-21T08:54:30.403 INFO:tasks.workunit.client.0.smithi057.stdout: AR crypto/asymmetric_keys/built-in.a 2024-04-21T08:54:30.456 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/api.o 2024-04-21T08:54:30.490 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mmzone.o 2024-04-21T08:54:30.768 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/fpu/xstate.o 2024-04-21T08:54:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:30 smithi129 ceph-mon[31272]: pgmap v283: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 783 KiB/s wr, 4.96k op/s 2024-04-21T08:54:30.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:30 smithi057 ceph-mon[27968]: pgmap v283: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 783 KiB/s wr, 4.96k op/s 2024-04-21T08:54:30.848 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pat/memtype.o 2024-04-21T08:54:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:30 smithi136 ceph-mon[31791]: pgmap v283: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 783 KiB/s wr, 4.96k op/s 2024-04-21T08:54:31.117 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/vmstat.o 2024-04-21T08:54:31.154 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/selinuxfs.o 2024-04-21T08:54:31.210 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/cipher.o 2024-04-21T08:54:31.246 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/uncore_snbep.o 2024-04-21T08:54:31.282 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/quota/quota_v2.o 2024-04-21T08:54:31.510 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/semaphore.o 2024-04-21T08:54:31.692 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pat/memtype_interval.o 2024-04-21T08:54:31.757 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/compress.o 2024-04-21T08:54:31.851 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/quota/quota_tree.o 2024-04-21T08:54:31.919 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/kernel/fpu/built-in.a 2024-04-21T08:54:31.969 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mce/core.o 2024-04-21T08:54:32.098 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/rwsem.o 2024-04-21T08:54:32.252 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/mm/pat/built-in.a 2024-04-21T08:54:32.277 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/algapi.o 2024-04-21T08:54:32.287 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/init.o 2024-04-21T08:54:32.358 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/backing-dev.o 2024-04-21T08:54:32.611 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/netlink.o 2024-04-21T08:54:32.636 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/quota/quota.o 2024-04-21T08:54:32.822 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/percpu-rwsem.o 2024-04-21T08:54:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:32 smithi129 ceph-mon[31272]: pgmap v284: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 891 KiB/s wr, 4.90k op/s 2024-04-21T08:54:32.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:32 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:32.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:32 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:54:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:32 smithi057 ceph-mon[27968]: pgmap v284: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 891 KiB/s wr, 4.90k op/s 2024-04-21T08:54:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:32 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:32 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:54:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:32 smithi136 ceph-mon[31791]: pgmap v284: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 891 KiB/s wr, 4.90k op/s 2024-04-21T08:54:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:32 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:32 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:54:32.985 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/uncore_discovery.o 2024-04-21T08:54:33.155 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/init_64.o 2024-04-21T08:54:33.241 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mm_init.o 2024-04-21T08:54:33.259 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/nlmsgtab.o 2024-04-21T08:54:33.288 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/scatterwalk.o 2024-04-21T08:54:33.358 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/spinlock.o 2024-04-21T08:54:33.379 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mce/severity.o 2024-04-21T08:54:33.642 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/quota/kqid.o 2024-04-21T08:54:33.712 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/osq_lock.o 2024-04-21T08:54:33.807 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/intel/cstate.o 2024-04-21T08:54:33.838 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/proc.o 2024-04-21T08:54:33.985 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/qspinlock.o 2024-04-21T08:54:34.031 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mce/genpool.o 2024-04-21T08:54:34.071 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/netif.o 2024-04-21T08:54:34.096 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/quota/netlink.o 2024-04-21T08:54:34.347 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/percpu.o 2024-04-21T08:54:34.359 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/aead.o 2024-04-21T08:54:34.371 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/fault.o 2024-04-21T08:54:34.421 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/rtmutex_api.o 2024-04-21T08:54:34.510 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/events/intel/built-in.a 2024-04-21T08:54:34.562 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mce/intel.o 2024-04-21T08:54:34.644 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/zhaoxin/core.o 2024-04-21T08:54:34.763 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/quota/built-in.a 2024-04-21T08:54:34.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:34 smithi129 ceph-mon[31272]: pgmap v285: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 891 KiB/s wr, 4.58k op/s 2024-04-21T08:54:34.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:34 smithi057 ceph-mon[27968]: pgmap v285: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 891 KiB/s wr, 4.58k op/s 2024-04-21T08:54:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:34 smithi136 ceph-mon[31791]: pgmap v285: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 891 KiB/s wr, 4.58k op/s 2024-04-21T08:54:35.069 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/netnode.o 2024-04-21T08:54:35.073 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/geniv.o 2024-04-21T08:54:35.169 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mce/amd.o 2024-04-21T08:54:35.211 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/task_mmu.o 2024-04-21T08:54:35.269 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/locking/qrwlock.o 2024-04-21T08:54:35.375 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/events/zhaoxin/built-in.a 2024-04-21T08:54:35.410 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/core.o 2024-04-21T08:54:35.433 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/ioremap.o 2024-04-21T08:54:35.673 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/locking/built-in.a 2024-04-21T08:54:35.713 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/qos.o 2024-04-21T08:54:35.821 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/skcipher.o 2024-04-21T08:54:35.964 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/netport.o 2024-04-21T08:54:36.201 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/slab_common.o 2024-04-21T08:54:36.211 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/extable.o 2024-04-21T08:54:36.333 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mce/threshold.o 2024-04-21T08:54:36.511 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/inode.o 2024-04-21T08:54:36.535 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/main.o 2024-04-21T08:54:36.728 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/seqiv.o 2024-04-21T08:54:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:36 smithi129 ceph-mon[31272]: pgmap v286: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.2 MiB/s wr, 3.91k op/s 2024-04-21T08:54:36.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:36 smithi057 ceph-mon[27968]: pgmap v286: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.2 MiB/s wr, 3.91k op/s 2024-04-21T08:54:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:36 smithi136 ceph-mon[31791]: pgmap v286: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.2 MiB/s wr, 3.91k op/s 2024-04-21T08:54:36.909 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/mmap.o 2024-04-21T08:54:36.922 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/status.o 2024-04-21T08:54:36.984 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/probe.o 2024-04-21T08:54:37.082 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/kernel/cpu/mce/built-in.a 2024-04-21T08:54:37.131 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mtrr/mtrr.o 2024-04-21T08:54:37.217 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/root.o 2024-04-21T08:54:37.301 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/echainiv.o 2024-04-21T08:54:37.313 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/utils.o 2024-04-21T08:54:37.391 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/console.o 2024-04-21T08:54:37.493 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pgtable.o 2024-04-21T08:54:37.594 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/compaction.o 2024-04-21T08:54:37.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:37.600 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:37.820 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/base.o 2024-04-21T08:54:37.823 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/ebitmap.o 2024-04-21T08:54:37.826 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mtrr/if.o 2024-04-21T08:54:37.906 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/ahash.o 2024-04-21T08:54:37.967 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/rapl.o 2024-04-21T08:54:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 -- 172.21.15.57:0/1245017946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10eb40 msgr2=0x7f088c11d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 --2- 172.21.15.57:0/1245017946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10eb40 0x7f088c11d740 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0884009ca0 tx=0x7f088402c1b0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 -- 172.21.15.57:0/1245017946 shutdown_connections 2024-04-21T08:54:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 --2- 172.21.15.57:0/1245017946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10eb40 0x7f088c11d740 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 --2- 172.21.15.57:0/1245017946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10e0c0 0x7f088c10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 --2- 172.21.15.57:0/1245017946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f088c110550 0x7f088c110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 -- 172.21.15.57:0/1245017946 >> 172.21.15.57:0/1245017946 conn(0x7f088c06ce60 msgr2=0x7f088c06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 -- 172.21.15.57:0/1245017946 shutdown_connections 2024-04-21T08:54:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.973+0000 7f0892567640 1 -- 172.21.15.57:0/1245017946 wait complete. 2024-04-21T08:54:37.975 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.974+0000 7f0892567640 1 Processor -- start 2024-04-21T08:54:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 -- start start 2024-04-21T08:54:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10e0c0 0x7f088c1a7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 0x7f088c1a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f088c110550 0x7f088c1abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f088c120de0 con 0x7f088c10e0c0 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f088c120c60 con 0x7f088c10eb40 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.975+0000 7f0892567640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f088c120f60 con 0x7f088c110550 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 0x7f088c1a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 0x7f088c1a78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57598/0 (socket says 172.21.15.57:57598) 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 -- 172.21.15.57:0/3336027392 learned_addr learned my addr 172.21.15.57:0/3336027392 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088bfff640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10e0c0 0x7f088c1a7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.977 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f0890b04640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f088c110550 0x7f088c1abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 -- 172.21.15.57:0/3336027392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f088c110550 msgr2=0x7f088c1abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f088c110550 0x7f088c1abc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 -- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10e0c0 msgr2=0x7f088c1a7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.976+0000 7f088b7fe640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10e0c0 0x7f088c1a7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.977+0000 7f088b7fe640 1 -- 172.21.15.57:0/3336027392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088c1ac320 con 0x7f088c10eb40 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.977+0000 7f088bfff640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10e0c0 0x7f088c1a7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.977+0000 7f088b7fe640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 0x7f088c1a78a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f088000d900 tx=0x7f088000ddc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.977+0000 7f08897fa640 1 -- 172.21.15.57:0/3336027392 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880017070 con 0x7f088c10eb40 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.977+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088c1ac610 con 0x7f088c10eb40 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.977+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c110f10 con 0x7f088c10eb40 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.978+0000 7f08897fa640 1 -- 172.21.15.57:0/3336027392 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0880009c30 con 0x7f088c10eb40 2024-04-21T08:54:37.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.978+0000 7f08897fa640 1 -- 172.21.15.57:0/3336027392 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088000f620 con 0x7f088c10eb40 2024-04-21T08:54:37.981 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.981+0000 7f08897fa640 1 -- 172.21.15.57:0/3336027392 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f088000f7c0 con 0x7f088c10eb40 2024-04-21T08:54:37.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.981+0000 7f0895236740 1 Processor -- start 2024-04-21T08:54:37.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.981+0000 7f0895236740 1 -- start start 2024-04-21T08:54:37.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0895236740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd75ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0895236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd7e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0895236740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd80ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0895236740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561b4ccda1e0 con 0x561b4cd75700 2024-04-21T08:54:37.983 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0895236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561b4cc77390 con 0x561b4cd76020 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0895236740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561b4ccd61a0 con 0x561b4cd7ec00 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd7e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f0890b04640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd75ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd7e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57614/0 (socket says 172.21.15.57:57614) 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088bfff640 1 -- 172.21.15.57:0/26819472 learned_addr learned my addr 172.21.15.57:0/26819472 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088b7fe640 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd80ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f08897fa640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08740a3e60 0x7f08740a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088bfff640 1 -- 172.21.15.57:0/26819472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 msgr2=0x561b4cd80ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088bfff640 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd80ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.982+0000 7f088bfff640 1 -- 172.21.15.57:0/26819472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 msgr2=0x561b4cd75ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.983+0000 7f088bfff640 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd75ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.984 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.983+0000 7f08897fa640 1 -- 172.21.15.57:0/3336027392 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f08800d1620 con 0x7f088c10eb40 2024-04-21T08:54:37.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.984+0000 7f088bfff640 1 -- 172.21.15.57:0/26819472 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b4cab67d0 con 0x561b4cd76020 2024-04-21T08:54:37.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.984+0000 7f088b7fe640 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd80ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:37.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.984+0000 7f0890b04640 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd75ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:37.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.984+0000 7f088bfff640 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd7e6c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f087c010450 tx=0x7f087c010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7388e96278e61c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.984+0000 7f0890b04640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08740a3e60 0x7f08740a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.985+0000 7f0890b04640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08740a3e60 0x7f08740a6320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0884006010 tx=0x7f08840095c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.985+0000 7f08727fc640 1 -- 172.21.15.57:0/26819472 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c011760 con 0x561b4cd76020 2024-04-21T08:54:37.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.985+0000 7f08727fc640 1 -- 172.21.15.57:0/26819472 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f087c011db0 con 0x561b4cd76020 2024-04-21T08:54:37.986 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.985+0000 7f08727fc640 1 -- 172.21.15.57:0/26819472 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c01a380 con 0x561b4cd76020 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 -- 172.21.15.57:0/26819472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 msgr2=0x561b4cd7e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd7e6c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f087c010450 tx=0x7f087c010910 comp rx=0 tx=0).stop 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 -- 172.21.15.57:0/26819472 shutdown_connections 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd80ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd7e6c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 --2- 172.21.15.57:0/26819472 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd75ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.987 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.986+0000 7f0895236740 1 -- 172.21.15.57:0/26819472 >> 172.21.15.57:0/26819472 conn(0x561b4cd6fe30 msgr2=0x561b4cd70210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.987+0000 7f0895236740 1 -- 172.21.15.57:0/26819472 shutdown_connections 2024-04-21T08:54:37.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.987+0000 7f0895236740 1 -- 172.21.15.57:0/26819472 wait complete. 2024-04-21T08:54:37.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.987+0000 7f0895236740 1 -- start start 2024-04-21T08:54:37.988 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.987+0000 7f0895236740 1 Processor -- start 2024-04-21T08:54:37.990 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd75700 0x561b4cd7bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd7c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd7ec00 0x561b4cd7c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561b4cd819f0 con 0x561b4cd7ec00 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561b4cc932f0 con 0x561b4cd75700 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561b4cd81b70 con 0x561b4cd76020 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.988+0000 7f0895236740 10 client.0 _ll_register_callbacks cb 0x561b4cca0580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd75700 0x561b4cd7bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd75700 0x561b4cd7bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57616/0 (socket says 172.21.15.57:57616) 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088bfff640 1 -- 172.21.15.57:0/2631971968 learned_addr learned my addr 172.21.15.57:0/2631971968 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f0890b04640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd7ec00 0x561b4cd7c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088b7fe640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd7c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088b7fe640 1 -- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd75700 msgr2=0x561b4cd7bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088b7fe640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd75700 0x561b4cd7bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.991 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.989+0000 7f088b7fe640 1 -- 172.21.15.57:0/2631971968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd7ec00 msgr2=0x561b4cd7c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:37.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.990+0000 7f088b7fe640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd7ec00 0x561b4cd7c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.990+0000 7f088b7fe640 1 -- 172.21.15.57:0/2631971968 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b4c9ec3f0 con 0x561b4cd76020 2024-04-21T08:54:37.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.990+0000 7f0890b04640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd7ec00 0x561b4cd7c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:37.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.990+0000 7f088b7fe640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd7c140 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f08800ce950 tx=0x7f08800995b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f001d5b850e8f928 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:37.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.990+0000 7f086b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.990+0000 7f086b7fe640 1 -- 172.21.15.57:0/2631971968 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880020070 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.991+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f087c011db0 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.991+0000 7f086b7fe640 1 -- 172.21.15.57:0/2631971968 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f08800d11a0 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.991+0000 7f086b7fe640 1 -- 172.21.15.57:0/2631971968 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0880009de0 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.991+0000 7f0895236740 10 client.26383 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.991+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561b4c9ac0f0 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f086b7fe640 1 -- 172.21.15.57:0/2631971968 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f087c011db0 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f086b7fe640 1 -- 172.21.15.57:0/2631971968 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f08800a1100 con 0x561b4cd76020 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f0895236740 10 client.26383 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f0895236740 10 client.26383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f0895236740 20 client.26383 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:37.994 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f0895236740 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cd8ec10 0x561b4cd91030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f0895236740 4 client.26383 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b4cce7c00 con 0x561b4cd8ec10 2024-04-21T08:54:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.992+0000 7f088bfff640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cd8ec10 0x561b4cd91030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.993+0000 7f088bfff640 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cd8ec10 0x561b4cd91030 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35f7594a2c31a9b1 server_cookie=8628a9341658da30 in_seq=0 out_seq=0 2024-04-21T08:54:37.995 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.993+0000 7f086b7fe640 10 client.26383 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.994+0000 7f086b7fe640 1 -- 172.21.15.57:0/2631971968 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b4cce7c00 con 0x561b4cd8ec10 2024-04-21T08:54:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.994+0000 7f086b7fe640 10 client.26383 handle_command_reply: tid=0 2024-04-21T08:54:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.994+0000 7f0895236740 1 client.26383 shutdown 2024-04-21T08:54:37.996 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 msgr2=0x561b4cd7c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd7c140 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f08800ce950 tx=0x7f08800995b0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 shutdown_connections 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cd8ec10 0x561b4cd91030 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd7ec00 0x561b4cd7c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd7c140 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 --2- 172.21.15.57:0/2631971968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd75700 0x561b4cd7bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 >> 172.21.15.57:0/2631971968 conn(0x561b4cd6fe30 msgr2=0x561b4cd71b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f086b7fe640 0 client.26383 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 shutdown_connections 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 1 -- 172.21.15.57:0/2631971968 wait complete. 2024-04-21T08:54:37.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:37.995+0000 7f0895236740 20 client.26383 trim_cache size 0 max 16384 2024-04-21T08:54:38.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.034+0000 7f0895236740 1 Processor -- start 2024-04-21T08:54:38.034 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.034+0000 7f0895236740 1 -- start start 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.034+0000 7f0895236740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd7e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.034+0000 7f0895236740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd8a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f088bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd7e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f0895236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd7ec00 0x561b4cd6ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f0895236740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f087800dfe0 con 0x561b4cd75700 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f088bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd7e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36570/0 (socket says 172.21.15.57:36570) 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f088bfff640 1 -- 172.21.15.57:0/1239693220 learned_addr learned my addr 172.21.15.57:0/1239693220 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f088b7fe640 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd8a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0878010840 con 0x561b4cd7ec00 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f087800e630 con 0x561b4cd76020 2024-04-21T08:54:38.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.035+0000 7f0890b04640 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd7ec00 0x561b4cd6ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.036+0000 7f088bfff640 1 -- 172.21.15.57:0/1239693220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 msgr2=0x561b4cd8a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.036+0000 7f088bfff640 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd8a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.036+0000 7f088bfff640 1 -- 172.21.15.57:0/1239693220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd7ec00 msgr2=0x561b4cd6ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.036+0000 7f088bfff640 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd7ec00 0x561b4cd6ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.036+0000 7f088bfff640 1 -- 172.21.15.57:0/1239693220 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b4cd74ab0 con 0x561b4cd75700 2024-04-21T08:54:38.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.036+0000 7f088bfff640 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd7e330 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f087c010450 tx=0x7f087c011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fba81231bfee0f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.040+0000 7f0871ffb640 1 -- 172.21.15.57:0/1239693220 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c030020 con 0x561b4cd75700 2024-04-21T08:54:38.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.040+0000 7f0871ffb640 1 -- 172.21.15.57:0/1239693220 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f087c00ba40 con 0x561b4cd75700 2024-04-21T08:54:38.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.040+0000 7f0871ffb640 1 -- 172.21.15.57:0/1239693220 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c0055f0 con 0x561b4cd75700 2024-04-21T08:54:38.042 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.041+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 msgr2=0x561b4cd7e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.042 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.041+0000 7f0895236740 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd7e330 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f087c010450 tx=0x7f087c011f60 comp rx=0 tx=0).stop 2024-04-21T08:54:38.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.043+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 shutdown_connections 2024-04-21T08:54:38.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.043+0000 7f0895236740 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd7ec00 0x561b4cd6ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.043+0000 7f0895236740 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd76020 0x561b4cd8a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.043+0000 7f0895236740 1 --2- 172.21.15.57:0/1239693220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd7e330 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.044 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.043+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 >> 172.21.15.57:0/1239693220 conn(0x561b4cd6fe30 msgr2=0x561b4cd6e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.044+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 shutdown_connections 2024-04-21T08:54:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.044+0000 7f0895236740 1 -- 172.21.15.57:0/1239693220 wait complete. 2024-04-21T08:54:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.044+0000 7f0895236740 1 -- start start 2024-04-21T08:54:38.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.044+0000 7f0895236740 1 Processor -- start 2024-04-21T08:54:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd86d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd87280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd822c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f087800ed00 con 0x561b4cd75700 2024-04-21T08:54:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0878010170 con 0x561b4cd76020 2024-04-21T08:54:38.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f087800f3d0 con 0x561b4cd7ec00 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f0895236740 10 client.0 _ll_register_callbacks cb 0x7f08800d4dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f088b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd87280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f088b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd87280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57626/0 (socket says 172.21.15.57:57626) 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.045+0000 7f088b7fe640 1 -- 172.21.15.57:0/307243920 learned_addr learned my addr 172.21.15.57:0/307243920 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.046+0000 7f0890b04640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.046+0000 7f088bfff640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd86d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.047+0000 7f088bfff640 1 -- 172.21.15.57:0/307243920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 msgr2=0x561b4cd822c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.047 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.047+0000 7f088bfff640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd822c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.047+0000 7f088bfff640 1 -- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 msgr2=0x561b4cd87280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.047+0000 7f088bfff640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd87280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.047+0000 7f088bfff640 1 -- 172.21.15.57:0/307243920 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b4cd82ac0 con 0x561b4cd75700 2024-04-21T08:54:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.047+0000 7f088bfff640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd86d40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x561b4cd70e80 tx=0x7f087c01c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19780b584a81ecdc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.048+0000 7f086b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:38.048 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.048+0000 7f086b7fe640 1 -- 172.21.15.57:0/307243920 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c030050 con 0x561b4cd75700 2024-04-21T08:54:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.048+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f087c00ba40 con 0x561b4cd75700 2024-04-21T08:54:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.048+0000 7f086b7fe640 1 -- 172.21.15.57:0/307243920 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f087c004020 con 0x561b4cd75700 2024-04-21T08:54:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.049+0000 7f086b7fe640 1 -- 172.21.15.57:0/307243920 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f087c004220 con 0x561b4cd75700 2024-04-21T08:54:38.049 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.049+0000 7f0895236740 10 client.17037 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:38.050 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.049+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561b4cdf4cc0 con 0x561b4cd75700 2024-04-21T08:54:38.050 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.049+0000 7f086b7fe640 1 -- 172.21.15.57:0/307243920 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f087c00ba40 con 0x561b4cd75700 2024-04-21T08:54:38.050 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f086b7fe640 1 -- 172.21.15.57:0/307243920 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f087c004920 con 0x561b4cd75700 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f0895236740 10 client.17037 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f0895236740 10 client.17037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f0895236740 20 client.17037 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f0895236740 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cdf5340 0x561b4cdf7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f0895236740 4 client.17037 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561b4cce1a70 con 0x561b4cdf5340 2024-04-21T08:54:38.051 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.050+0000 7f088b7fe640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cdf5340 0x561b4cdf7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.051+0000 7f088b7fe640 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cdf5340 0x561b4cdf7760 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5074097fa268567a server_cookie=36343c4065ef270b in_seq=0 out_seq=0 2024-04-21T08:54:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.051+0000 7f086b7fe640 10 client.17037 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.051+0000 7f086b7fe640 1 -- 172.21.15.57:0/307243920 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561b4cce1a70 con 0x561b4cdf5340 2024-04-21T08:54:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.051+0000 7f086b7fe640 10 client.17037 handle_command_reply: tid=0 2024-04-21T08:54:38.052 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.051+0000 7f0895236740 1 client.17037 shutdown 2024-04-21T08:54:38.056 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.055+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 msgr2=0x561b4cd86d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.055+0000 7f0895236740 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd86d40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x561b4cd70e80 tx=0x7f087c01c770 comp rx=0 tx=0).stop 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 shutdown_connections 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561b4cdf5340 0x561b4cdf7760 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561b4cd7ec00 0x561b4cd822c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561b4cd76020 0x561b4cd87280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 --2- 172.21.15.57:0/307243920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561b4cd75700 0x561b4cd86d40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 >> 172.21.15.57:0/307243920 conn(0x561b4cd6fe30 msgr2=0x561b4cd71af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.057 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f086b7fe640 0 client.17037 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 shutdown_connections 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 1 -- 172.21.15.57:0/307243920 wait complete. 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.056+0000 7f0895236740 20 client.17037 trim_cache size 0 max 16384 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:54:38.058 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08740a3e60 msgr2=0x7f08740a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08740a3e60 0x7f08740a6320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0884006010 tx=0x7f08840095c0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 msgr2=0x7f088c1a78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 0x7f088c1a78a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f088000d900 tx=0x7f088000ddc0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 shutdown_connections 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f08740a3e60 0x7f08740a6320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f088c110550 0x7f088c1abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f088c10eb40 0x7f088c1a78a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 --2- 172.21.15.57:0/3336027392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f088c10e0c0 0x7f088c1a7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 >> 172.21.15.57:0/3336027392 conn(0x7f088c06ce60 msgr2=0x7f088c071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 shutdown_connections 2024-04-21T08:54:38.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.058+0000 7f0892567640 1 -- 172.21.15.57:0/3336027392 wait complete. 2024-04-21T08:54:38.063 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/process.o 2024-04-21T08:54:38.080 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:54:38.080 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:54:38.325 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/physaddr.o 2024-04-21T08:54:38.497 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mtrr/generic.o 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 -- 172.21.15.57:0/3508313738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 msgr2=0x7f5b6810eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 --2- 172.21.15.57:0/3508313738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b6810eec0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009900 tx=0x7f5b5802bc30 comp rx=0 tx=0).stop 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 -- 172.21.15.57:0/3508313738 shutdown_connections 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 --2- 172.21.15.57:0/3508313738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b6810f400 0x7f5b68115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 --2- 172.21.15.57:0/3508313738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b6810eec0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 --2- 172.21.15.57:0/3508313738 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b68109de0 0x7f5b6810a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 -- 172.21.15.57:0/3508313738 >> 172.21.15.57:0/3508313738 conn(0x7f5b6806e410 msgr2=0x7f5b680719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 -- 172.21.15.57:0/3508313738 shutdown_connections 2024-04-21T08:54:38.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.512+0000 7f5b701bd640 1 -- 172.21.15.57:0/3508313738 wait complete. 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.513+0000 7f5b701bd640 1 Processor -- start 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.513+0000 7f5b701bd640 1 -- start start 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.513+0000 7f5b701bd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b68109de0 0x7f5b681a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b701bd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b681a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b701bd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b6810f400 0x7f5b681a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b701bd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6811aa90 con 0x7f5b6810f400 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b701bd640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6811a910 con 0x7f5b68109de0 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b701bd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6811ac10 con 0x7f5b6810ea40 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b681a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b681a13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59522/0 (socket says 172.21.15.57:59522) 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 -- 172.21.15.57:0/3814390513 learned_addr learned my addr 172.21.15.57:0/3814390513 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6df59640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b68109de0 0x7f5b681a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6e75a640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b6810f400 0x7f5b681a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 -- 172.21.15.57:0/3814390513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b68109de0 msgr2=0x7f5b681a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b68109de0 0x7f5b681a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 -- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b6810f400 msgr2=0x7f5b681a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b6810f400 0x7f5b681a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6d758640 1 -- 172.21.15.57:0/3814390513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b681a5e50 con 0x7f5b6810ea40 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6df59640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b68109de0 0x7f5b681a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.514+0000 7f5b6e75a640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b6810f400 0x7f5b681a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:38.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.515+0000 7f5b6d758640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b681a13f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5b58000c00 tx=0x7f5b5802c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.532+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58043020 con 0x7f5b6810ea40 2024-04-21T08:54:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.532+0000 7f5b701bd640 1 -- 172.21.15.57:0/3814390513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b681a60e0 con 0x7f5b6810ea40 2024-04-21T08:54:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.532+0000 7f5b701bd640 1 -- 172.21.15.57:0/3814390513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b681ad9d0 con 0x7f5b6810ea40 2024-04-21T08:54:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.532+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b5802cb60 con 0x7f5b6810ea40 2024-04-21T08:54:38.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.532+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58003af0 con 0x7f5b6810ea40 2024-04-21T08:54:38.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.535+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5b58046020 con 0x7f5b6810ea40 2024-04-21T08:54:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.536+0000 7f5b5effd640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b540a3ea0 0x7f5b540a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.536+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f5b580f2b20 con 0x7f5b6810ea40 2024-04-21T08:54:38.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.537+0000 7f5b6df59640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b540a3ea0 0x7f5b540a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.538+0000 7f5b6df59640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b540a3ea0 0x7f5b540a6360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5b60008e00 tx=0x7f5b60008350 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.543+0000 7f5b701bd640 1 -- 172.21.15.57:0/3814390513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b4c005290 con 0x7f5b6810ea40 2024-04-21T08:54:38.550 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:38.550+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b5804d020 con 0x7f5b6810ea40 2024-04-21T08:54:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 -- 172.21.15.57:0/206076212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d410ecc0 msgr2=0x7fd6d4108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 --2- 172.21.15.57:0/206076212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d410ecc0 0x7fd6d4108f50 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009950 tx=0x7fd6c402bcd0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 -- 172.21.15.57:0/206076212 shutdown_connections 2024-04-21T08:54:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 --2- 172.21.15.57:0/206076212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d410ecc0 0x7fd6d4108f50 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 --2- 172.21.15.57:0/206076212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d410e220 0x7fd6d410e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 --2- 172.21.15.57:0/206076212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d41104c0 0x7fd6d41108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.551+0000 7fd6d968c640 1 -- 172.21.15.57:0/206076212 >> 172.21.15.57:0/206076212 conn(0x7fd6d406cf00 msgr2=0x7fd6d406d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.552+0000 7fd6d968c640 1 -- 172.21.15.57:0/206076212 shutdown_connections 2024-04-21T08:54:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.552+0000 7fd6d968c640 1 -- 172.21.15.57:0/206076212 wait complete. 2024-04-21T08:54:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.552+0000 7fd6d968c640 1 Processor -- start 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 -- start start 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d410e220 0x7fd6d411e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 0x7fd6d411ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d4117a40 0x7fd6d4117ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4120ec0 con 0x7fd6d41104c0 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4120d40 con 0x7fd6d410e220 2024-04-21T08:54:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.553+0000 7fd6d968c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4121040 con 0x7fd6d4117a40 2024-04-21T08:54:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 0x7fd6d411ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 0x7fd6d411ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36600/0 (socket says 172.21.15.57:36600) 2024-04-21T08:54:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 -- 172.21.15.57:0/2214828827 learned_addr learned my addr 172.21.15.57:0/2214828827 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 -- 172.21.15.57:0/2214828827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d4117a40 msgr2=0x7fd6d4117ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:54:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d4117a40 0x7fd6d4117ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 -- 172.21.15.57:0/2214828827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d410e220 msgr2=0x7fd6d411e9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:54:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d410e220 0x7fd6d411e9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.554+0000 7fd6d27fc640 1 -- 172.21.15.57:0/2214828827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d4118640 con 0x7fd6d41104c0 2024-04-21T08:54:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.555+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 0x7fd6d411ef00 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800b750 tx=0x7fd6c800bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.563 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.563+0000 7fd6bbfff640 1 -- 172.21.15.57:0/2214828827 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8019070 con 0x7fd6d41104c0 2024-04-21T08:54:38.564 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.563+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d411c4e0 con 0x7fd6d41104c0 2024-04-21T08:54:38.564 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.563+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d411ca30 con 0x7fd6d41104c0 2024-04-21T08:54:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.568+0000 7fd6dc280740 1 Processor -- start 2024-04-21T08:54:38.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 -- start start 2024-04-21T08:54:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7d9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7da1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557b7c73e8d0 con 0x557b7c7d9d50 2024-04-21T08:54:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557b7c6dbe80 con 0x557b7c7daac0 2024-04-21T08:54:38.570 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6dc280740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557b7c73a890 con 0x557b7c7e32d0 2024-04-21T08:54:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6d2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7da1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6d2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7da1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36612/0 (socket says 172.21.15.57:36612) 2024-04-21T08:54:38.571 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6d2ffd640 1 -- 172.21.15.57:0/2136330582 learned_addr learned my addr 172.21.15.57:0/2136330582 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.569+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7d9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d37fe640 1 -- 172.21.15.57:0/2136330582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 msgr2=0x557b7c7e56c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e56c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d37fe640 1 -- 172.21.15.57:0/2136330582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 msgr2=0x557b7c7da1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7da1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d37fe640 1 -- 172.21.15.57:0/2136330582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b7c6fd320 con 0x557b7c7daac0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7d9810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009230 tx=0x7fd6c4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=254ec86219534c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6b97fa640 1 -- 172.21.15.57:0/2136330582 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4041070 con 0x557b7c7daac0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6b97fa640 1 -- 172.21.15.57:0/2136330582 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c4009460 con 0x557b7c7daac0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.570+0000 7fd6b97fa640 1 -- 172.21.15.57:0/2136330582 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40052b0 con 0x557b7c7daac0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.571+0000 7fd6bbfff640 1 -- 172.21.15.57:0/2214828827 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c800cbd0 con 0x7fd6d41104c0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.571+0000 7fd6bbfff640 1 -- 172.21.15.57:0/2214828827 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8014dc0 con 0x7fd6d41104c0 2024-04-21T08:54:38.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.571+0000 7fd6bbfff640 1 -- 172.21.15.57:0/2214828827 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd6c801c070 con 0x7fd6d41104c0 2024-04-21T08:54:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.571+0000 7fd6dc280740 1 -- 172.21.15.57:0/2136330582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 msgr2=0x557b7c7d9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.571+0000 7fd6dc280740 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7d9810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009230 tx=0x7fd6c4005be0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.572+0000 7fd6bbfff640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd6bc0a3ed0 0x7fd6bc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.573 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.572+0000 7fd6bbfff640 1 -- 172.21.15.57:0/2214828827 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fd6c80d07d0 con 0x7fd6d41104c0 2024-04-21T08:54:38.574 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.572+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd6bc0a3ed0 0x7fd6bc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.574 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.573+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd6bc0a3ed0 0x7fd6bc0a6390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009230 tx=0x7fd6c402cf90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.575+0000 7fd6dc280740 1 -- 172.21.15.57:0/2136330582 shutdown_connections 2024-04-21T08:54:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.575+0000 7fd6dc280740 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.575+0000 7fd6dc280740 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.575+0000 7fd6dc280740 1 --2- 172.21.15.57:0/2136330582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7d9810 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.576 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.575+0000 7fd6dc280740 1 -- 172.21.15.57:0/2136330582 >> 172.21.15.57:0/2136330582 conn(0x557b7c7a0c90 msgr2=0x557b7c7d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.585+0000 7fd6dc280740 1 -- 172.21.15.57:0/2136330582 shutdown_connections 2024-04-21T08:54:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.585+0000 7fd6dc280740 1 -- 172.21.15.57:0/2136330582 wait complete. 2024-04-21T08:54:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.585+0000 7fd6dc280740 1 -- start start 2024-04-21T08:54:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.585+0000 7fd6dc280740 1 Processor -- start 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7e07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x557b7c7e5fa0 con 0x557b7c7d9d50 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557b7c6c6eb0 con 0x557b7c7daac0 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557b7c7e6120 con 0x557b7c7e32d0 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6dc280740 10 client.0 _ll_register_callbacks cb 0x557b7c6f3b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59552/0 (socket says 172.21.15.57:59552) 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6d37fe640 1 -- 172.21.15.57:0/3039879734 learned_addr learned my addr 172.21.15.57:0/3039879734 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7e07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.586+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d37fe640 1 -- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 msgr2=0x557b7c7e07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7e07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d37fe640 1 -- 172.21.15.57:0/3039879734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 msgr2=0x557b7c7e0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d37fe640 1 -- 172.21.15.57:0/3039879734 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b7c451210 con 0x557b7c7e32d0 2024-04-21T08:54:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7e07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.587+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e0cf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40057b0 tx=0x7fd6c4005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a56d05c39b796be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3039879734 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4041050 con 0x557b7c7e32d0 2024-04-21T08:54:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3039879734 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fd6c4034650 con 0x557b7c7e32d0 2024-04-21T08:54:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c4009460 con 0x557b7c7e32d0 2024-04-21T08:54:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3039879734 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd6c405d050 con 0x557b7c7e32d0 2024-04-21T08:54:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6dc280740 10 client.26401 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.606+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557b7c51d620 con 0x557b7c7e32d0 2024-04-21T08:54:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3039879734 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c4009460 con 0x557b7c7e32d0 2024-04-21T08:54:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3039879734 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd6c404f620 con 0x557b7c7e32d0 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6dc280740 10 client.26401 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6dc280740 10 client.26401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6dc280740 20 client.26401 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c7f3360 0x557b7c7f5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6dc280740 4 client.26401 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.607+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b7c752880 con 0x557b7c7f3360 2024-04-21T08:54:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.608+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c7f3360 0x557b7c7f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.609+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c7f3360 0x557b7c7f5750 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c6b5bfe271e3f0f server_cookie=968a8a9268f76bcf in_seq=0 out_seq=0 2024-04-21T08:54:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.609+0000 7fd6b6ff5640 10 client.26401 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3039879734 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b7c752880 con 0x557b7c7f3360 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6b6ff5640 10 client.26401 handle_command_reply: tid=0 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 client.26401 shutdown 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 msgr2=0x557b7c7e0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e0cf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40057b0 tx=0x7fd6c4005be0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 shutdown_connections 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c7f3360 0x557b7c7f5750 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7e32d0 0x557b7c7e0cf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7daac0 0x557b7c7e07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6b6ff5640 0 client.26401 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3039879734 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 >> 172.21.15.57:0/3039879734 conn(0x557b7c7a0c90 msgr2=0x557b7c7d6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 shutdown_connections 2024-04-21T08:54:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 1 -- 172.21.15.57:0/3039879734 wait complete. 2024-04-21T08:54:38.612 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.610+0000 7fd6dc280740 20 client.26401 trim_cache size 0 max 16384 2024-04-21T08:54:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.657+0000 7fd6dc280740 1 Processor -- start 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 -- start start 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7dd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c7dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41c49b0 con 0x557b7c7d9d50 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41c7210 con 0x557b7c7e32d0 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6dc280740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41c5000 con 0x557b7c7daac0 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c7dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6d27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7dd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c7dda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57668/0 (socket says 172.21.15.57:57668) 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6d37fe640 1 -- 172.21.15.57:0/1189672976 learned_addr learned my addr 172.21.15.57:0/1189672976 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.670+0000 7fd6d2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d27fc640 1 -- 172.21.15.57:0/1189672976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 msgr2=0x557b7c7dda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c7dda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d27fc640 1 -- 172.21.15.57:0/1189672976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 msgr2=0x557b7c7e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d27fc640 1 -- 172.21.15.57:0/1189672976 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b7c7d4f80 con 0x557b7c7daac0 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d37fe640 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c7dda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.671+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7dd4c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80cdc10 tx=0x7fd6c8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2557cfbabae8d7f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.684+0000 7fd6b8ff9640 1 -- 172.21.15.57:0/1189672976 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80b5070 con 0x557b7c7daac0 2024-04-21T08:54:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.684+0000 7fd6b8ff9640 1 -- 172.21.15.57:0/1189672976 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c80a5190 con 0x557b7c7daac0 2024-04-21T08:54:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.684+0000 7fd6b8ff9640 1 -- 172.21.15.57:0/1189672976 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80b5be0 con 0x557b7c7daac0 2024-04-21T08:54:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 -- 172.21.15.57:0/1189672976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 msgr2=0x557b7c7dd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7dd4c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80cdc10 tx=0x7fd6c8000bc0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 -- 172.21.15.57:0/1189672976 shutdown_connections 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c7dda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7dd4c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 --2- 172.21.15.57:0/1189672976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 -- 172.21.15.57:0/1189672976 >> 172.21.15.57:0/1189672976 conn(0x557b7c7a0c90 msgr2=0x557b7c7d6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 -- 172.21.15.57:0/1189672976 shutdown_connections 2024-04-21T08:54:38.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 -- 172.21.15.57:0/1189672976 wait complete. 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 -- start start 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.685+0000 7fd6dc280740 1 Processor -- start 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7f4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c858530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41c56d0 con 0x557b7c7d9d50 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41c6b40 con 0x557b7c7e32d0 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41c5da0 con 0x557b7c7daac0 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6dc280740 10 client.0 _ll_register_callbacks cb 0x7fd6c403f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6d27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6d2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c858530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6d27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7f4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59574/0 (socket says 172.21.15.57:59574) 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6d37fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c858530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57672/0 (socket says 172.21.15.57:57672) 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.686+0000 7fd6d27fc640 1 -- 172.21.15.57:0/3348308022 learned_addr learned my addr 172.21.15.57:0/3348308022 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d27fc640 1 -- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 msgr2=0x557b7c858530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c858530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d27fc640 1 -- 172.21.15.57:0/3348308022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 msgr2=0x557b7c7f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d27fc640 1 -- 172.21.15.57:0/3348308022 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b7c7d9020 con 0x557b7c7daac0 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.687+0000 7fd6d27fc640 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7f4a50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8098980 tx=0x7fd6c8098850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93c27e93bd00b5e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.693+0000 7fd6b6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.693+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3348308022 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80b9070 con 0x557b7c7daac0 2024-04-21T08:54:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.693+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3348308022 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fd6c80b5be0 con 0x557b7c7daac0 2024-04-21T08:54:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.693+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c80a5190 con 0x557b7c7daac0 2024-04-21T08:54:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3348308022 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd6c80ac050 con 0x557b7c7daac0 2024-04-21T08:54:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 10 client.26407 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x557b7c8599f0 con 0x557b7c7daac0 2024-04-21T08:54:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3348308022 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6c80a5190 con 0x557b7c7daac0 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3348308022 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fd6c80d0330 con 0x557b7c7daac0 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 10 client.26407 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 10 client.26407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 20 client.26407 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c85a040 0x557b7c85c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 4 client.26407 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.706+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b7c6de3d0 con 0x557b7c85a040 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.707+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c85a040 0x557b7c85c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.707+0000 7fd6b6ff5640 10 client.26407 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:38.707+0000 7fd6d2ffd640 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c85a040 0x557b7c85c460 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a285720f43f3593b server_cookie=f3862e6b8e8e3981 in_seq=0 out_seq=0 2024-04-21T08:54:38.805 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/shash.o 2024-04-21T08:54:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:38 smithi129 ceph-mon[31272]: pgmap v287: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 3.21k op/s 2024-04-21T08:54:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:38 smithi057 ceph-mon[27968]: pgmap v287: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 3.21k op/s 2024-04-21T08:54:38.842 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/events/msr.o 2024-04-21T08:54:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:38 smithi136 ceph-mon[31791]: pgmap v287: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.0 MiB/s wr, 3.21k op/s 2024-04-21T08:54:38.935 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/tlb.o 2024-04-21T08:54:38.951 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/hashtab.o 2024-04-21T08:54:38.956 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/suspend.o 2024-04-21T08:54:39.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.083+0000 7f5b701bd640 1 -- 172.21.15.57:0/3814390513 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b4c005850 con 0x7f5b6810ea40 2024-04-21T08:54:39.085 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.084+0000 7f5b5effd640 1 -- 172.21.15.57:0/3814390513 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f5b580bef10 con 0x7f5b6810ea40 2024-04-21T08:54:39.087 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:54:39.087 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:54:39.087 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:54:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 -- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b540a3ea0 msgr2=0x7f5b540a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b540a3ea0 0x7f5b540a6360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5b60008e00 tx=0x7f5b60008350 comp rx=0 tx=0).stop 2024-04-21T08:54:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 -- 172.21.15.57:0/3814390513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 msgr2=0x7f5b681a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b681a13f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5b58000c00 tx=0x7f5b5802c7e0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 -- 172.21.15.57:0/3814390513 shutdown_connections 2024-04-21T08:54:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f5b540a3ea0 0x7f5b540a6360 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5b6810f400 0x7f5b681a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b6810ea40 0x7f5b681a13f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 --2- 172.21.15.57:0/3814390513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b68109de0 0x7f5b681a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 -- 172.21.15.57:0/3814390513 >> 172.21.15.57:0/3814390513 conn(0x7f5b6806e410 msgr2=0x7f5b68071ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 -- 172.21.15.57:0/3814390513 shutdown_connections 2024-04-21T08:54:39.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.108+0000 7f5b5cff9640 1 -- 172.21.15.57:0/3814390513 wait complete. 2024-04-21T08:54:39.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:54:39.406 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/symtab.o 2024-04-21T08:54:39.481 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/mtrr/cleanup.o 2024-04-21T08:54:39.482 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/generic.o 2024-04-21T08:54:39.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.513+0000 7f4a78bd5640 1 -- 172.21.15.57:0/3880235577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74106050 msgr2=0x7f4a7410dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.513+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/3880235577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74106050 0x7f4a7410dc20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c009be0 tx=0x7f4a6c02bef0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 -- 172.21.15.57:0/3880235577 shutdown_connections 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/3880235577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74106050 0x7f4a7410dc20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/3880235577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a74101930 0x7f4a74105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/3880235577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74100f80 0x7f4a74101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 -- 172.21.15.57:0/3880235577 >> 172.21.15.57:0/3880235577 conn(0x7f4a740fc820 msgr2=0x7f4a740fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 -- 172.21.15.57:0/3880235577 shutdown_connections 2024-04-21T08:54:39.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.514+0000 7f4a78bd5640 1 -- 172.21.15.57:0/3880235577 wait complete. 2024-04-21T08:54:39.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.515+0000 7f4a78bd5640 1 Processor -- start 2024-04-21T08:54:39.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.515+0000 7f4a78bd5640 1 -- start start 2024-04-21T08:54:39.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a78bd5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74100f80 0x7f4a741985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a78bd5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 0x7f4a74198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a78bd5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7419ce60 0x7f4a7419d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a78bd5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74110ff0 con 0x7f4a74106050 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a78bd5640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74110e70 con 0x7f4a7419ce60 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a78bd5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a74111170 con 0x7f4a74100f80 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 0x7f4a74198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 0x7f4a74198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36650/0 (socket says 172.21.15.57:36650) 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a71d9b640 1 -- 172.21.15.57:0/86022013 learned_addr learned my addr 172.21.15.57:0/86022013 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:39.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7419ce60 0x7f4a7419d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a7259c640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74100f80 0x7f4a741985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a72d9d640 1 -- 172.21.15.57:0/86022013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74100f80 msgr2=0x7f4a741985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74100f80 0x7f4a741985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.516+0000 7f4a72d9d640 1 -- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 msgr2=0x7f4a74198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.517+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 0x7f4a74198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.517+0000 7f4a72d9d640 1 -- 172.21.15.57:0/86022013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7419d950 con 0x7f4a7419ce60 2024-04-21T08:54:39.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.517+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 0x7f4a74198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:39.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.517+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7419ce60 0x7f4a7419d310 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c02c430 tx=0x7f4a6c033c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:39.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.518+0000 7f4a637fe640 1 -- 172.21.15.57:0/86022013 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c043020 con 0x7f4a7419ce60 2024-04-21T08:54:39.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.518+0000 7f4a637fe640 1 -- 172.21.15.57:0/86022013 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a6c02ca80 con 0x7f4a7419ce60 2024-04-21T08:54:39.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.518+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7419dbe0 con 0x7f4a7419ce60 2024-04-21T08:54:39.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.518+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a740fd390 con 0x7f4a7419ce60 2024-04-21T08:54:39.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.518+0000 7f4a637fe640 1 -- 172.21.15.57:0/86022013 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c034430 con 0x7f4a7419ce60 2024-04-21T08:54:39.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.525+0000 7f4a637fe640 1 -- 172.21.15.57:0/86022013 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a6c046020 con 0x7f4a7419ce60 2024-04-21T08:54:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.526+0000 7f4a637fe640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a5c0a3e70 0x7f4a5c0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.526+0000 7f4a637fe640 1 -- 172.21.15.57:0/86022013 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f4a6c0f3e50 con 0x7f4a7419ce60 2024-04-21T08:54:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.526+0000 7f4a7259c640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a5c0a3e70 0x7f4a5c0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 Processor -- start 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 -- start start 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9b98ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9ba1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9ba4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9afd1f0 con 0x55ebb9ba1c00 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9a9a4b0 con 0x55ebb9b986d0 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.527+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9af91b0 con 0x55ebb9b99080 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a7259c640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a5c0a3e70 0x7f4a5c0a6330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a680095f0 tx=0x7f4a68005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9ba1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9ba1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59602/0 (socket says 172.21.15.57:59602) 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 -- 172.21.15.57:0/703201912 learned_addr learned my addr 172.21.15.57:0/703201912 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9ba4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a7259c640 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9b98ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 -- 172.21.15.57:0/703201912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 msgr2=0x55ebb9b98ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9b98ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 -- 172.21.15.57:0/703201912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 msgr2=0x55ebb9ba4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9ba4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.528+0000 7f4a71d9b640 1 -- 172.21.15.57:0/703201912 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb9a8c1b0 con 0x55ebb9b99080 2024-04-21T08:54:39.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.529+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9ba1630 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4a640114f0 tx=0x7f4a640119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d937ad9047ff44f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:39.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.529+0000 7f4a60ff9640 1 -- 172.21.15.57:0/703201912 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6401e070 con 0x55ebb9b99080 2024-04-21T08:54:39.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.529+0000 7f4a60ff9640 1 -- 172.21.15.57:0/703201912 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a6400fa70 con 0x55ebb9b99080 2024-04-21T08:54:39.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/703201912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 msgr2=0x55ebb9ba1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:39.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9ba1630 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4a640114f0 tx=0x7f4a640119b0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/703201912 shutdown_connections 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9ba4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9ba1630 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/703201912 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9b98ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.530+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/703201912 >> 172.21.15.57:0/703201912 conn(0x55ebb9b92da0 msgr2=0x55ebb9b93180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.531+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/703201912 shutdown_connections 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.531+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/703201912 wait complete. 2024-04-21T08:54:39.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.531+0000 7f4a7b7c6740 1 -- start start 2024-04-21T08:54:39.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 Processor -- start 2024-04-21T08:54:39.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9ba1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b99080 0x55ebb9b9e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b9ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9ba4bc0 con 0x55ebb9b99080 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9ab6500 con 0x55ebb9b986d0 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ebb9ba4d40 con 0x55ebb9ba1c00 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.532+0000 7f4a7b7c6740 10 client.0 _ll_register_callbacks cb 0x55ebb9a84b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9ba1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b99080 0x55ebb9b9e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9ba1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57706/0 (socket says 172.21.15.57:57706) 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 -- 172.21.15.57:0/3308099799 learned_addr learned my addr 172.21.15.57:0/3308099799 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:39.534 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b9ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 -- 172.21.15.57:0/3308099799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9ba1c00 msgr2=0x55ebb9b9ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:39.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b9ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 -- 172.21.15.57:0/3308099799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b99080 msgr2=0x55ebb9b9e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:39.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b99080 0x55ebb9b9e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:39.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a7259c640 1 -- 172.21.15.57:0/3308099799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb980f3f0 con 0x55ebb9b986d0 2024-04-21T08:54:39.535 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.533+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b99080 0x55ebb9b9e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:39.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.534+0000 7f4a7259c640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9ba1570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4a68010f80 tx=0x7f4a680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e3ca33d0537246 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:39.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.534+0000 7f4a59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:39.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.535+0000 7f4a59ffb640 1 -- 172.21.15.57:0/3308099799 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68020840 con 0x55ebb9b986d0 2024-04-21T08:54:39.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.535+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ebb9a9c930 con 0x55ebb9b986d0 2024-04-21T08:54:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a59ffb640 1 -- 172.21.15.57:0/3308099799 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f4a680209e0 con 0x55ebb9b986d0 2024-04-21T08:54:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a59ffb640 1 -- 172.21.15.57:0/3308099799 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4a68031050 con 0x55ebb9b986d0 2024-04-21T08:54:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a7b7c6740 10 client.26528 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ebb98d97d0 con 0x55ebb9b986d0 2024-04-21T08:54:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a59ffb640 1 -- 172.21.15.57:0/3308099799 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ebb9a9c930 con 0x55ebb9b986d0 2024-04-21T08:54:39.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a59ffb640 1 -- 172.21.15.57:0/3308099799 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4a6800f040 con 0x55ebb9b986d0 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a7b7c6740 10 client.26528 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a7b7c6740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.536+0000 7f4a7b7c6740 20 client.26528 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.537+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9bc30d0 0x55ebb9bc54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.537+0000 7f4a7b7c6740 4 client.26528 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.537+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebb9b0ac10 con 0x55ebb9bc30d0 2024-04-21T08:54:39.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.537+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9bc30d0 0x55ebb9bc54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:39.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.537+0000 7f4a59ffb640 10 client.26528 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:39.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:39.538+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9bc30d0 0x55ebb9bc54f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b75bed030af3e2f server_cookie=deba5f47c0a4d7c4 in_seq=0 out_seq=0 2024-04-21T08:54:39.545 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/events/built-in.a 2024-04-21T08:54:39.814 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/sidtab.o 2024-04-21T08:54:39.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:39 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3814390513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:39.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:39 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3814390513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:39.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:39 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3814390513' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:54:39.903 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/cpu_entry_area.o 2024-04-21T08:54:39.953 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/akcipher.o 2024-04-21T08:54:40.079 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/hibernate.o 2024-04-21T08:54:40.338 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/array.o 2024-04-21T08:54:40.370 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/printk/printk.o 2024-04-21T08:54:40.527 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/kernel/cpu/mtrr/built-in.a 2024-04-21T08:54:40.768 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/microcode/core.o 2024-04-21T08:54:40.770 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/maccess.o 2024-04-21T08:54:40.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a59ffb640 1 -- 172.21.15.57:0/3308099799 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebb9b0ac10 con 0x55ebb9bc30d0 2024-04-21T08:54:40.788 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a59ffb640 10 client.26528 handle_command_reply: tid=0 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 client.26528 shutdown 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 msgr2=0x55ebb9ba1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9ba1570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4a68010f80 tx=0x7f4a680046c0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 shutdown_connections 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9bc30d0 0x55ebb9bc54f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b9ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b99080 0x55ebb9b9e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/3308099799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b986d0 0x55ebb9ba1570 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 >> 172.21.15.57:0/3308099799 conn(0x55ebb9b92da0 msgr2=0x55ebb9b94af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.787+0000 7f4a59ffb640 0 client.26528 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.788+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 shutdown_connections 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.788+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/3308099799 wait complete. 2024-04-21T08:54:40.789 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.788+0000 7f4a7b7c6740 20 client.26528 trim_cache size 0 max 16384 2024-04-21T08:54:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.789+0000 7fd6b6ff5640 1 -- 172.21.15.57:0/3348308022 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b7c6de3d0 con 0x557b7c85a040 2024-04-21T08:54:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.789+0000 7fd6b6ff5640 10 client.26407 handle_command_reply: tid=0 2024-04-21T08:54:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.790+0000 7fd6dc280740 1 client.26407 shutdown 2024-04-21T08:54:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 msgr2=0x557b7c7f4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7f4a50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8098980 tx=0x7fd6c8098850 comp rx=0 tx=0).stop 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 shutdown_connections 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x557b7c85a040 0x557b7c85c460 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557b7c7e32d0 0x557b7c858530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557b7c7daac0 0x557b7c7f4a50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 --2- 172.21.15.57:0/3348308022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x557b7c7d9d50 0x557b7c7f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 >> 172.21.15.57:0/3348308022 conn(0x557b7c7a0c90 msgr2=0x557b7c7e0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6b6ff5640 0 client.26407 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 shutdown_connections 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.792+0000 7fd6dc280740 1 -- 172.21.15.57:0/3348308022 wait complete. 2024-04-21T08:54:40.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.793+0000 7fd6dc280740 20 client.26407 trim_cache size 0 max 16384 2024-04-21T08:54:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:54:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:54:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:54:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:54:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd6bc0a3ed0 msgr2=0x7fd6bc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd6bc0a3ed0 0x7fd6bc0a6390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009230 tx=0x7fd6c402cf90 comp rx=0 tx=0).stop 2024-04-21T08:54:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 msgr2=0x7fd6d411ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 0x7fd6d411ef00 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800b750 tx=0x7fd6c800bc10 comp rx=0 tx=0).stop 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 shutdown_connections 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd6bc0a3ed0 0x7fd6bc0a6390 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6d4117a40 0x7fd6d4117ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6d41104c0 0x7fd6d411ef00 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 --2- 172.21.15.57:0/2214828827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6d410e220 0x7fd6d411e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.810+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 >> 172.21.15.57:0/2214828827 conn(0x7fd6d406cf00 msgr2=0x7fd6d410c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.811+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 shutdown_connections 2024-04-21T08:54:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:40.811+0000 7fd6d968c640 1 -- 172.21.15.57:0/2214828827 wait complete. 2024-04-21T08:54:40.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:40 smithi129 ceph-mon[31272]: pgmap v288: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 995 KiB/s wr, 2.74k op/s 2024-04-21T08:54:40.833 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/show_mem.o 2024-04-21T08:54:40.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.835+0000 7f4a7b7c6740 1 Processor -- start 2024-04-21T08:54:40.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.835+0000 7f4a7b7c6740 1 -- start start 2024-04-21T08:54:40.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.836+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b986d0 0x55ebb9badb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.836+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9b8dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.836+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b91ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9b8dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9b8dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59618/0 (socket says 172.21.15.57:59618) 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a71d9b640 1 -- 172.21.15.57:0/2978633922 learned_addr learned my addr 172.21.15.57:0/2978633922 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5400e020 con 0x55ebb9b986d0 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a54010880 con 0x55ebb9ba1c00 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5400e670 con 0x55ebb9b99080 2024-04-21T08:54:40.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.837+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b91ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a7259c640 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b986d0 0x55ebb9badb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:40 smithi057 ceph-mon[27968]: pgmap v288: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 995 KiB/s wr, 2.74k op/s 2024-04-21T08:54:40.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a71d9b640 1 -- 172.21.15.57:0/2978633922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9ba1c00 msgr2=0x55ebb9b91ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:40.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b91ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a71d9b640 1 -- 172.21.15.57:0/2978633922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b986d0 msgr2=0x55ebb9badb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:40.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b986d0 0x55ebb9badb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a71d9b640 1 -- 172.21.15.57:0/2978633922 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb9b8e670 con 0x55ebb9b99080 2024-04-21T08:54:40.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.838+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9b8dcb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a64011eb0 tx=0x7f4a64019da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9ebff55a1e3eed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:40.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.851+0000 7f4a5bfff640 1 -- 172.21.15.57:0/2978633922 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6402e070 con 0x55ebb9b99080 2024-04-21T08:54:40.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.851+0000 7f4a5bfff640 1 -- 172.21.15.57:0/2978633922 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a64029e50 con 0x55ebb9b99080 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 msgr2=0x55ebb9b8dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9b8dcb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a64011eb0 tx=0x7f4a64019da0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 shutdown_connections 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9ba1c00 0x55ebb9b91ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b99080 0x55ebb9b8dcb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/2978633922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9b986d0 0x55ebb9badb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 >> 172.21.15.57:0/2978633922 conn(0x55ebb9b92da0 msgr2=0x55ebb9b91590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:40.853 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 shutdown_connections 2024-04-21T08:54:40.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.852+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/2978633922 wait complete. 2024-04-21T08:54:40.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 -- start start 2024-04-21T08:54:40.854 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 Processor -- start 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b986d0 0x55ebb9baa070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9baa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b9ab90 0x55ebb9b9b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5400ed40 con 0x55ebb9ba1c00 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a540101b0 con 0x55ebb9b9ab90 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5400f410 con 0x55ebb9b986d0 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.853+0000 7f4a7b7c6740 10 client.0 _ll_register_callbacks cb 0x7f4a68010c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9baa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9baa5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:36688/0 (socket says 172.21.15.57:36688) 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 -- 172.21.15.57:0/1670339099 learned_addr learned my addr 172.21.15.57:0/1670339099 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:40.855 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a7259c640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b986d0 0x55ebb9baa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a72d9d640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b9ab90 0x55ebb9b9b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 -- 172.21.15.57:0/1670339099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b986d0 msgr2=0x55ebb9baa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b986d0 0x55ebb9baa070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 -- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b9ab90 msgr2=0x55ebb9b9b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b9ab90 0x55ebb9b9b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a71d9b640 1 -- 172.21.15.57:0/1670339099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb9b9b550 con 0x55ebb9ba1c00 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.854+0000 7f4a7259c640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b986d0 0x55ebb9baa070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:40.856 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.855+0000 7f4a71d9b640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9baa5b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4a6402c040 tx=0x7f4a640093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4627ffbf3f6ff8c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:40.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.869+0000 7f4a59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:40.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.869+0000 7f4a59ffb640 1 -- 172.21.15.57:0/1670339099 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6402e070 con 0x55ebb9ba1c00 2024-04-21T08:54:40.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.869+0000 7f4a59ffb640 1 -- 172.21.15.57:0/1670339099 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f4a64029210 con 0x55ebb9ba1c00 2024-04-21T08:54:40.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.869+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ebb9b04a80 con 0x55ebb9ba1c00 2024-04-21T08:54:40.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.872+0000 7f4a59ffb640 1 -- 172.21.15.57:0/1670339099 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4a64019e60 con 0x55ebb9ba1c00 2024-04-21T08:54:40.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.872+0000 7f4a7b7c6740 10 client.17076 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:40.873 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.872+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ebb9c22280 con 0x55ebb9ba1c00 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a59ffb640 1 -- 172.21.15.57:0/1670339099 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ebb9b04a80 con 0x55ebb9ba1c00 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a59ffb640 1 -- 172.21.15.57:0/1670339099 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4a64005370 con 0x55ebb9ba1c00 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7b7c6740 10 client.17076 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7b7c6740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7b7c6740 20 client.17076 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9c22980 0x55ebb9c24da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7b7c6740 4 client.17076 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebb9a806e0 con 0x55ebb9c22980 2024-04-21T08:54:40.874 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.873+0000 7f4a7259c640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9c22980 0x55ebb9c24da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:40.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.874+0000 7f4a7259c640 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9c22980 0x55ebb9c24da0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3d7fe0583700a59 server_cookie=3d2af56329fd9b4e in_seq=0 out_seq=0 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a59ffb640 10 client.17076 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a59ffb640 1 -- 172.21.15.57:0/1670339099 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebb9a806e0 con 0x55ebb9c22980 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a59ffb640 10 client.17076 handle_command_reply: tid=0 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 client.17076 shutdown 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 msgr2=0x55ebb9baa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9baa5b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4a6402c040 tx=0x7f4a640093b0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 shutdown_connections 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ebb9c22980 0x55ebb9c24da0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ebb9b9ab90 0x55ebb9b9b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ebb9ba1c00 0x55ebb9baa5b0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 --2- 172.21.15.57:0/1670339099 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ebb9b986d0 0x55ebb9baa070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 >> 172.21.15.57:0/1670339099 conn(0x55ebb9b92da0 msgr2=0x55ebb9b91e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a59ffb640 0 client.17076 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 shutdown_connections 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 1 -- 172.21.15.57:0/1670339099 wait complete. 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.884+0000 7f4a7b7c6740 20 client.17076 trim_cache size 0 max 16384 2024-04-21T08:54:40.886 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:54:40.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:40 smithi136 ceph-mon[31791]: pgmap v288: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 995 KiB/s wr, 2.74k op/s 2024-04-21T08:54:40.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.901+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a5c0a3e70 msgr2=0x7f4a5c0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.901+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a5c0a3e70 0x7f4a5c0a6330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a680095f0 tx=0x7f4a68005f90 comp rx=0 tx=0).stop 2024-04-21T08:54:40.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.901+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7419ce60 msgr2=0x7f4a7419d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:40.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.901+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7419ce60 0x7f4a7419d310 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c02c430 tx=0x7f4a6c033c40 comp rx=0 tx=0).stop 2024-04-21T08:54:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.902+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 shutdown_connections 2024-04-21T08:54:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.902+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a5c0a3e70 0x7f4a5c0a6330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.902+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7419ce60 0x7f4a7419d310 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.902+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a74106050 0x7f4a74198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.902+0000 7f4a78bd5640 1 --2- 172.21.15.57:0/86022013 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a74100f80 0x7f4a741985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:40.904 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.902+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 >> 172.21.15.57:0/86022013 conn(0x7f4a740fc820 msgr2=0x7f4a74069420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.904+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 shutdown_connections 2024-04-21T08:54:40.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:40.904+0000 7f4a78bd5640 1 -- 172.21.15.57:0/86022013 wait complete. 2024-04-21T08:54:40.942 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/sig.o 2024-04-21T08:54:41.092 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pgprot.o 2024-04-21T08:54:41.217 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/avtab.o 2024-04-21T08:54:41.349 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/fd.o 2024-04-21T08:54:41.370 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/snapshot.o 2024-04-21T08:54:41.408 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/microcode/intel.o 2024-04-21T08:54:41.579 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/kpp.o 2024-04-21T08:54:41.606 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/hugetlbpage.o 2024-04-21T08:54:41.714 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/interval_tree.o 2024-04-21T08:54:41.925 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:54:41.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:54:41.925 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:54:42.003 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/proc_tty.o 2024-04-21T08:54:42.037 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/printk/printk_safe.o 2024-04-21T08:54:42.144 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/microcode/amd.o 2024-04-21T08:54:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.155+0000 7fbfb77fe640 1 -- 172.21.15.57:0/4094069053 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0032910 con 0x7fbfb8109910 2024-04-21T08:54:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 -- 172.21.15.57:0/4094069053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 msgr2=0x7fbfb807a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 --2- 172.21.15.57:0/4094069053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb807a1b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009ca0 tx=0x7fbfb002c1c0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 -- 172.21.15.57:0/4094069053 shutdown_connections 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 --2- 172.21.15.57:0/4094069053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb807a1b0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 --2- 172.21.15.57:0/4094069053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb81093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 --2- 172.21.15.57:0/4094069053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfb810e660 0x7fbfb810ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 -- 172.21.15.57:0/4094069053 >> 172.21.15.57:0/4094069053 conn(0x7fbfb806d060 msgr2=0x7fbfb806d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 -- 172.21.15.57:0/4094069053 shutdown_connections 2024-04-21T08:54:42.158 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.156+0000 7fbfbf159640 1 -- 172.21.15.57:0/4094069053 wait complete. 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.157+0000 7fbfbf159640 1 Processor -- start 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.157+0000 7fbfbf159640 1 -- start start 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbf159640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb807a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbf159640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb8074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbf159640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfb810e660 0x7fbfb8074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbf159640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb807ce60 con 0x7fbfb8109910 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbf159640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb807cce0 con 0x7fbfb8108f50 2024-04-21T08:54:42.159 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbf159640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb807cfe0 con 0x7fbfb810e660 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbcef5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb807a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbcef5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb807a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57756/0 (socket says 172.21.15.57:57756) 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbcef5640 1 -- 172.21.15.57:0/3369661742 learned_addr learned my addr 172.21.15.57:0/3369661742 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb8074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.158+0000 7fbfbcef5640 1 -- 172.21.15.57:0/3369661742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfb810e660 msgr2=0x7fbfb8074710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfb810e660 0x7fbfb8074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfb810e660 0x7fbfb8074710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.160 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfbcef5640 1 -- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 msgr2=0x7fbfb8074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb8074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfbcef5640 1 -- 172.21.15.57:0/3369661742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb81bc120 con 0x7fbfb8108f50 2024-04-21T08:54:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb8074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:42.161 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.159+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb807a040 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e9f0 tx=0x7fbfa800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.166 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.165+0000 7fbfb5ffb640 1 -- 172.21.15.57:0/3369661742 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa800cd80 con 0x7fbfb8108f50 2024-04-21T08:54:42.166 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.165+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb81bc2f0 con 0x7fbfb8108f50 2024-04-21T08:54:42.166 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.166+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb81bc7b0 con 0x7fbfb8108f50 2024-04-21T08:54:42.166 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.166+0000 7fbfb5ffb640 1 -- 172.21.15.57:0/3369661742 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfa80108e0 con 0x7fbfb8108f50 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfb5ffb640 1 -- 172.21.15.57:0/3369661742 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8018980 con 0x7fbfb8108f50 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 Processor -- start 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfb5ffb640 1 -- 172.21.15.57:0/3369661742 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbfa8018be0 con 0x7fbfb8108f50 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 -- start start 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe3fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe48840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe4b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x560e0fda3f00 con 0x560e0fe3f8e0 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e0fd41590 con 0x560e0fe48e10 2024-04-21T08:54:42.172 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.171+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560e0fd9fec0 con 0x560e0fe40290 2024-04-21T08:54:42.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe4b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.173 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe4b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57770/0 (socket says 172.21.15.57:57770) 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 -- 172.21.15.57:0/3557605946 learned_addr learned my addr 172.21.15.57:0/3557605946 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe3fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 -- 172.21.15.57:0/3557605946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 msgr2=0x560e0fe48840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe48840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 -- 172.21.15.57:0/3557605946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 msgr2=0x560e0fe3fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe3fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 -- 172.21.15.57:0/3557605946 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e0fb7e7e0 con 0x560e0fe48e10 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe3fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.172+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe4b260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbfac010480 tx=0x7fbfac010940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b63609dfd2af42d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.174 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.173+0000 7fbfb5ffb640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbfa00b4e50 0x7fbfa00b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.173+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbfa00b4e50 0x7fbfa00b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.174+0000 7fbfb5ffb640 1 -- 172.21.15.57:0/3369661742 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fbfa80d1e40 con 0x7fbfb8108f50 2024-04-21T08:54:42.175 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.174+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbfa00b4e50 0x7fbfa00b7310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002c7d0 tx=0x7fbfb002e040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.182+0000 7fbf9effd640 1 -- 172.21.15.57:0/3557605946 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac011760 con 0x560e0fe48e10 2024-04-21T08:54:42.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.182+0000 7fbf9effd640 1 -- 172.21.15.57:0/3557605946 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac011db0 con 0x560e0fe48e10 2024-04-21T08:54:42.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.182+0000 7fbf9effd640 1 -- 172.21.15.57:0/3557605946 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac00e430 con 0x560e0fe48e10 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/3557605946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 msgr2=0x560e0fe4b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe4b260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbfac010480 tx=0x7fbfac010940 comp rx=0 tx=0).stop 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/3557605946 shutdown_connections 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe4b260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe48840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/3557605946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe3fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.183+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/3557605946 >> 172.21.15.57:0/3557605946 conn(0x560e0fe39fb0 msgr2=0x560e0fe3a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.184+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/3557605946 shutdown_connections 2024-04-21T08:54:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.184+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/3557605946 wait complete. 2024-04-21T08:54:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.184+0000 7fbfc1e2b740 1 -- start start 2024-04-21T08:54:42.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.184+0000 7fbfc1e2b740 1 Processor -- start 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe45d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe40290 0x560e0fe462b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe48e10 0x560e0fe467f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x560e0fe4bdd0 con 0x560e0fe40290 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560e0fd5d580 con 0x560e0fe3f8e0 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x560e0fe4bf50 con 0x560e0fe48e10 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfc1e2b740 10 client.0 _ll_register_callbacks cb 0x560e0fd63650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe40290 0x560e0fe462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfbcef5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe45d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfbcef5640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe45d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57776/0 (socket says 172.21.15.57:57776) 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfbcef5640 1 -- 172.21.15.57:0/1581012147 learned_addr learned my addr 172.21.15.57:0/1581012147 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.185+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe48e10 0x560e0fe467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfbcef5640 1 -- 172.21.15.57:0/1581012147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe48e10 msgr2=0x560e0fe467f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe48e10 0x560e0fe467f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfbcef5640 1 -- 172.21.15.57:0/1581012147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe40290 msgr2=0x560e0fe462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe40290 0x560e0fe462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfbcef5640 1 -- 172.21.15.57:0/1581012147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e0fbdef70 con 0x560e0fe3f8e0 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe40290 0x560e0fe462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe45d70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c6a0 tx=0x7fbfa809e140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdfbe5a35155dbd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.186+0000 7fbf9c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:42.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/1581012147 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80a1df0 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/1581012147 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fbfa80a6220 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfac011db0 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/1581012147 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbfa80a66b0 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbfc1e2b740 10 client.26546 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e0fa76170 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.187+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/1581012147 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac011db0 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/1581012147 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbfa809c900 con 0x560e0fe3f8e0 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfc1e2b740 10 client.26546 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfc1e2b740 10 client.26546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfc1e2b740 20 client.26546 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fe59290 0x560e0fe5b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfc1e2b740 4 client.26546 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:42.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e0fdb1920 con 0x560e0fe59290 2024-04-21T08:54:42.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.188+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fe59290 0x560e0fe5b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.189+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fe59290 0x560e0fe5b700 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e1d90fad5bec749 server_cookie=3f59063eddeecf7d in_seq=0 out_seq=0 2024-04-21T08:54:42.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.189+0000 7fbf9c7f8640 10 client.26546 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.189+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/1581012147 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e0fdb1920 con 0x560e0fe59290 2024-04-21T08:54:42.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.189+0000 7fbf9c7f8640 10 client.26546 handle_command_reply: tid=0 2024-04-21T08:54:42.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.190+0000 7fbfc1e2b740 1 client.26546 shutdown 2024-04-21T08:54:42.192 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/policydb.o 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 msgr2=0x560e0fe45d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe45d70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800c6a0 tx=0x7fbfa809e140 comp rx=0 tx=0).stop 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 shutdown_connections 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fe59290 0x560e0fe5b700 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe48e10 0x560e0fe467f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe40290 0x560e0fe462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/1581012147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe45d70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbf9c7f8640 0 client.26546 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 >> 172.21.15.57:0/1581012147 conn(0x560e0fe39fb0 msgr2=0x560e0fe3bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 shutdown_connections 2024-04-21T08:54:42.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/1581012147 wait complete. 2024-04-21T08:54:42.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.198+0000 7fbfc1e2b740 20 client.26546 trim_cache size 0 max 16384 2024-04-21T08:54:42.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.236+0000 7fbfc1e2b740 1 Processor -- start 2024-04-21T08:54:42.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.236+0000 7fbfc1e2b740 1 -- start start 2024-04-21T08:54:42.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.236+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe484a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.236+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe54cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe60fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400de20 con 0x560e0fe3f8e0 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4010680 con 0x560e0fe48e10 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400e470 con 0x560e0fe40290 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe54cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfbcef5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe54cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59678/0 (socket says 172.21.15.57:59678) 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 -- 172.21.15.57:0/4139808038 learned_addr learned my addr 172.21.15.57:0/4139808038 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe60fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 -- 172.21.15.57:0/4139808038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 msgr2=0x560e0fe60fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe60fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 -- 172.21.15.57:0/4139808038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 msgr2=0x560e0fe484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfb7fff640 1 -- 172.21.15.57:0/4139808038 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e0fe4b200 con 0x560e0fe40290 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.237+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:42.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.238+0000 7fbfb7fff640 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe54cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbfac01cae0 tx=0x7fbfac00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=896e19a790f6796e server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.238+0000 7fbf9e7fc640 1 -- 172.21.15.57:0/4139808038 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac01c820 con 0x560e0fe40290 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.239+0000 7fbf9e7fc640 1 -- 172.21.15.57:0/4139808038 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac005040 con 0x560e0fe40290 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/4139808038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 msgr2=0x560e0fe54cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe54cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbfac01cae0 tx=0x7fbfac00bd60 comp rx=0 tx=0).stop 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/4139808038 shutdown_connections 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe48e10 0x560e0fe60fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe54cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/4139808038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe3f8e0 0x560e0fe484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/4139808038 >> 172.21.15.57:0/4139808038 conn(0x560e0fe39fb0 msgr2=0x560e0fe39010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/4139808038 shutdown_connections 2024-04-21T08:54:42.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/4139808038 wait complete. 2024-04-21T08:54:42.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 -- start start 2024-04-21T08:54:42.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.240+0000 7fbfc1e2b740 1 Processor -- start 2024-04-21T08:54:42.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe4cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe4d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe48e10 0x560e0fe4d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400eb40 con 0x560e0fe48e10 2024-04-21T08:54:42.244 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400ffb0 con 0x560e0fe3f8e0 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400f210 con 0x560e0fe40290 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfc1e2b740 10 client.0 _ll_register_callbacks cb 0x7fbfa80995c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe4d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe4d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59694/0 (socket says 172.21.15.57:59694) 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfb7fff640 1 -- 172.21.15.57:0/920724199 learned_addr learned my addr 172.21.15.57:0/920724199 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.243+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe48e10 0x560e0fe4d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe4cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbd6f6640 1 -- 172.21.15.57:0/920724199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 msgr2=0x560e0fe4d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe4d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbd6f6640 1 -- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 msgr2=0x560e0fe4cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe4cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbd6f6640 1 -- 172.21.15.57:0/920724199 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e0febe940 con 0x560e0fe48e10 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbfbd6f6640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe48e10 0x560e0fe4d7c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0037040 tx=0x7fbfb002c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf38740869dea2d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.244+0000 7fbf9c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:42.246 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.245+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/920724199 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0050030 con 0x560e0fe48e10 2024-04-21T08:54:42.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.245+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e0fdab790 con 0x560e0fe48e10 2024-04-21T08:54:42.247 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.245+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/920724199 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fbfb003f880 con 0x560e0fe48e10 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.246+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/920724199 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fbfb005d050 con 0x560e0fe48e10 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.246+0000 7fbfc1e2b740 10 client.17094 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.246+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x560e0febfd10 con 0x560e0fe48e10 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.246+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/920724199 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560e0fdab790 con 0x560e0fe48e10 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/920724199 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fbfb0034cc0 con 0x560e0fe48e10 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfc1e2b740 10 client.17094 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfc1e2b740 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfc1e2b740 20 client.17094 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fec0510 0x560e0fec2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfc1e2b740 4 client.17094 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560e0fd27c50 con 0x560e0fec0510 2024-04-21T08:54:42.248 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.247+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fec0510 0x560e0fec2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.248+0000 7fbfbcef5640 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fec0510 0x560e0fec2930 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed46ca93d024104e server_cookie=6a46f7bf094c0f78 in_seq=0 out_seq=0 2024-04-21T08:54:42.249 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.248+0000 7fbf9c7f8640 10 client.17094 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.249+0000 7fbf9c7f8640 1 -- 172.21.15.57:0/920724199 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560e0fd27c50 con 0x560e0fec0510 2024-04-21T08:54:42.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.249+0000 7fbf9c7f8640 10 client.17094 handle_command_reply: tid=0 2024-04-21T08:54:42.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.249+0000 7fbfc1e2b740 1 client.17094 shutdown 2024-04-21T08:54:42.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.249+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe48e10 msgr2=0x560e0fe4d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.249+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe48e10 0x560e0fe4d7c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0037040 tx=0x7fbfb002c1c0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 shutdown_connections 2024-04-21T08:54:42.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x560e0fec0510 0x560e0fec2930 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x560e0fe48e10 0x560e0fe4d7c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x560e0fe40290 0x560e0fe4d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 --2- 172.21.15.57:0/920724199 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560e0fe3f8e0 0x560e0fe4cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.251 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 >> 172.21.15.57:0/920724199 conn(0x560e0fe39fb0 msgr2=0x560e0fe45d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbf9c7f8640 0 client.17094 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 shutdown_connections 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 1 -- 172.21.15.57:0/920724199 wait complete. 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.250+0000 7fbfc1e2b740 20 client.17094 trim_cache size 0 max 16384 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "2f79e585-3b65-4439-ac68-97fc759446ab", 2024-04-21T08:54:42.252 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:54:42.253 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:54:42.253 INFO:tasks.workunit.client.0.smithi057.stdout: ASN.1 crypto/rsapubkey.asn1.[ch] 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbfa00b4e50 msgr2=0x7fbfa00b7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbfa00b4e50 0x7fbfa00b7310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbfb002c7d0 tx=0x7fbfb002e040 comp rx=0 tx=0).stop 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 msgr2=0x7fbfb807a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb807a040 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbfa800e9f0 tx=0x7fbfa800eeb0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 shutdown_connections 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fbfa00b4e50 0x7fbfa00b7310 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfb810e660 0x7fbfb8074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbfb8109910 0x7fbfb8074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.254 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 --2- 172.21.15.57:0/3369661742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfb8108f50 0x7fbfb807a040 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.253+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 >> 172.21.15.57:0/3369661742 conn(0x7fbfb806d060 msgr2=0x7fbfb8070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.254+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 shutdown_connections 2024-04-21T08:54:42.255 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.254+0000 7fbfbf159640 1 -- 172.21.15.57:0/3369661742 wait complete. 2024-04-21T08:54:42.260 INFO:tasks.workunit.client.0.smithi057.stdout: ASN.1 crypto/rsaprivkey.asn1.[ch] 2024-04-21T08:54:42.268 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/dump_pagetables.o 2024-04-21T08:54:42.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:54:42.274 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:54:42.280 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/rsa.o 2024-04-21T08:54:42.520 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/list_lru.o 2024-04-21T08:54:42.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:42 smithi057 ceph-mon[27968]: pgmap v289: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 914 KiB/s wr, 1.73k op/s 2024-04-21T08:54:42.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 -- 172.21.15.57:0/3322485539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac108f50 msgr2=0x7f8dac1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 --2- 172.21.15.57:0/3322485539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac108f50 0x7f8dac1093b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009ca0 tx=0x7f8da402c180 comp rx=0 tx=0).stop 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 -- 172.21.15.57:0/3322485539 shutdown_connections 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 --2- 172.21.15.57:0/3322485539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1099d0 0x7f8dac11d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 --2- 172.21.15.57:0/3322485539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac108f50 0x7f8dac1093b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 --2- 172.21.15.57:0/3322485539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1105f0 0x7f8dac1109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 -- 172.21.15.57:0/3322485539 >> 172.21.15.57:0/3322485539 conn(0x7f8dac06ced0 msgr2=0x7f8dac06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 -- 172.21.15.57:0/3322485539 shutdown_connections 2024-04-21T08:54:42.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.621+0000 7f8db140b640 1 -- 172.21.15.57:0/3322485539 wait complete. 2024-04-21T08:54:42.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.622+0000 7f8db140b640 1 Processor -- start 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.622+0000 7f8db140b640 1 -- start start 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db140b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1099d0 0x7f8dac1a7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db140b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac1105f0 0x7f8dac1a7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db140b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 0x7f8dac1ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db140b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8dac120ea0 con 0x7f8dac1105f0 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db140b640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8dac120d20 con 0x7f8dac1abee0 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db140b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8dac121020 con 0x7f8dac1099d0 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8dabfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1099d0 0x7f8dac1a7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db0c0a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 0x7f8dac1ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db0c0a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 0x7f8dac1ac390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57816/0 (socket says 172.21.15.57:57816) 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8db0c0a640 1 -- 172.21.15.57:0/924784253 learned_addr learned my addr 172.21.15.57:0/924784253 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.623+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac1105f0 0x7f8dac1a7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8db0c0a640 1 -- 172.21.15.57:0/924784253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1099d0 msgr2=0x7f8dac1a7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1099d0 0x7f8dac1a7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8db0c0a640 1 -- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac1105f0 msgr2=0x7f8dac1a7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac1105f0 0x7f8dac1a7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8db0c0a640 1 -- 172.21.15.57:0/924784253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dac1ac9d0 con 0x7f8dac1abee0 2024-04-21T08:54:42.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8dabfff640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1099d0 0x7f8dac1a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:42.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac1105f0 0x7f8dac1a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:42.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.624+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 0x7f8dac1ac390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c970 tx=0x7f8d9c00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.625+0000 7f8da97fa640 1 -- 172.21.15.57:0/924784253 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c007bb0 con 0x7f8dac1abee0 2024-04-21T08:54:42.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.625+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dac1accc0 con 0x7f8dac1abee0 2024-04-21T08:54:42.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.625+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dac111200 con 0x7f8dac1abee0 2024-04-21T08:54:42.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.626+0000 7f8da97fa640 1 -- 172.21.15.57:0/924784253 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d9c007d50 con 0x7f8dac1abee0 2024-04-21T08:54:42.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.627+0000 7f8da97fa640 1 -- 172.21.15.57:0/924784253 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c004e20 con 0x7f8dac1abee0 2024-04-21T08:54:42.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.629+0000 7f8db5264740 1 Processor -- start 2024-04-21T08:54:42.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 -- start start 2024-04-21T08:54:42.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27ecae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5654f27511e0 con 0x5654f27ec700 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5654f26ee390 con 0x5654f27ed020 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8db5264740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5654f274d1a0 con 0x5654f27f5c00 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8da97fa640 1 -- 172.21.15.57:0/924784253 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8d9c005020 con 0x7f8dac1abee0 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.631+0000 7f8dabfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:57824/0 (socket says 172.21.15.57:57824) 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 -- 172.21.15.57:0/691086064 learned_addr learned my addr 172.21.15.57:0/691086064 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27ecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 -- 172.21.15.57:0/691086064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 msgr2=0x5654f27f7ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f7ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 -- 172.21.15.57:0/691086064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 msgr2=0x5654f27ecae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27ecae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 -- 172.21.15.57:0/691086064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f252d7d0 con 0x5654f27ed020 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27ecae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8dabfff640 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f56c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8da000dc40 tx=0x7f8da000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98799cc64e8d5b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.632+0000 7f8da97fa640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8d940a3d80 0x7f8d940a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.633+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8d940a3d80 0x7f8d940a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.633+0000 7f8da97fa640 1 -- 172.21.15.57:0/924784253 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f8d9c0d08a0 con 0x7f8dac1abee0 2024-04-21T08:54:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.633+0000 7f8d9a7fc640 1 -- 172.21.15.57:0/691086064 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0005020 con 0x5654f27ed020 2024-04-21T08:54:42.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.633+0000 7f8d9a7fc640 1 -- 172.21.15.57:0/691086064 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da00047a0 con 0x5654f27ed020 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.633+0000 7f8d9a7fc640 1 -- 172.21.15.57:0/691086064 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da00116a0 con 0x5654f27ed020 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.633+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8d940a3d80 0x7f8d940a6240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009bf0 tx=0x7f8da4000f40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 -- 172.21.15.57:0/691086064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 msgr2=0x5654f27f56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f56c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8da000dc40 tx=0x7f8da000f730 comp rx=0 tx=0).stop 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 -- 172.21.15.57:0/691086064 shutdown_connections 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f56c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 --2- 172.21.15.57:0/691086064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27ecae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.634+0000 7f8db5264740 1 -- 172.21.15.57:0/691086064 >> 172.21.15.57:0/691086064 conn(0x5654f27e6e30 msgr2=0x5654f27e7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.635+0000 7f8db5264740 1 -- 172.21.15.57:0/691086064 shutdown_connections 2024-04-21T08:54:42.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.635+0000 7f8db5264740 1 -- 172.21.15.57:0/691086064 wait complete. 2024-04-21T08:54:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.635+0000 7f8db5264740 1 -- start start 2024-04-21T08:54:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.635+0000 7f8db5264740 1 Processor -- start 2024-04-21T08:54:42.638 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 0x5654f27f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27f5c00 0x5654f27f3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5654f27f89f0 con 0x5654f27f5c00 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5654f270a2f0 con 0x5654f27ed020 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5654f27f8b70 con 0x5654f27ec700 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db5264740 10 client.0 _ll_register_callbacks cb 0x5654f2717580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 0x5654f27f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 0x5654f27f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:59724/0 (socket says 172.21.15.57:59724) 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 -- 172.21.15.57:0/3873324353 learned_addr learned my addr 172.21.15.57:0/3873324353 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27f5c00 0x5654f27f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 -- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 msgr2=0x5654f27f3140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f3140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 -- 172.21.15.57:0/3873324353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27f5c00 msgr2=0x5654f27f3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27f5c00 0x5654f27f3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8dabfff640 1 -- 172.21.15.57:0/3873324353 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f24633f0 con 0x5654f27ec700 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.636+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27f5c00 0x5654f27f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.637+0000 7f8dabfff640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 0x5654f27f2c00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8da0005cb0 tx=0x7f8da000d9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ed75a177d976a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.637+0000 7f8d8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.637+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3873324353 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da002a060 con 0x5654f27ec700 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.637+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da00047a0 con 0x5654f27ec700 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.637+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3873324353 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f8da0004c20 con 0x5654f27ec700 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.637+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3873324353 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8da0031050 con 0x5654f27ec700 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 10 client.26443 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5654f24230f0 con 0x5654f27ec700 2024-04-21T08:54:42.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3873324353 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da00047a0 con 0x5654f27ec700 2024-04-21T08:54:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3873324353 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8da00119b0 con 0x5654f27ec700 2024-04-21T08:54:42.641 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 10 client.26443 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 10 client.26443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 20 client.26443 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f2805b50 0x5654f2807f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 4 client.26443 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.638+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654f275ec00 con 0x5654f2805b50 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.639+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f2805b50 0x5654f2807f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.639+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f2805b50 0x5654f2807f40 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee8e481e440bf981 server_cookie=803e3fcceca665b3 in_seq=0 out_seq=0 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.639+0000 7f8d8f7fe640 10 client.26443 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3873324353 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654f275ec00 con 0x5654f2805b50 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8d8f7fe640 10 client.26443 handle_command_reply: tid=0 2024-04-21T08:54:42.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 client.26443 shutdown 2024-04-21T08:54:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 msgr2=0x5654f27f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 0x5654f27f2c00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8da0005cb0 tx=0x7f8da000d9b0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 shutdown_connections 2024-04-21T08:54:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f2805b50 0x5654f2807f40 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27f5c00 0x5654f27f3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ed020 0x5654f27f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 --2- 172.21.15.57:0/3873324353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ec700 0x5654f27f2c00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.641+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 >> 172.21.15.57:0/3873324353 conn(0x5654f27e6e30 msgr2=0x5654f27e8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.642+0000 7f8d8f7fe640 0 client.26443 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.642+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 shutdown_connections 2024-04-21T08:54:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.642+0000 7f8db5264740 1 -- 172.21.15.57:0/3873324353 wait complete. 2024-04-21T08:54:42.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.642+0000 7f8db5264740 20 client.26443 trim_cache size 0 max 16384 2024-04-21T08:54:42.649 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/printk/printk_ringbuffer.o 2024-04-21T08:54:42.654 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/cmdline.o 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.683+0000 7f8db5264740 1 Processor -- start 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.683+0000 7f8db5264740 1 -- start start 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db5264740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db5264740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 0x5654f2801910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db5264740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27f5c00 0x5654f280d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db5264740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9000df10 con 0x5654f27ec700 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db5264740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d90010770 con 0x5654f27f5c00 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db5264740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9000e560 con 0x5654f27ed020 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dab7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 0x5654f2801910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8db0c0a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27f5c00 0x5654f280d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dab7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 0x5654f2801910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42022/0 (socket says 172.21.15.57:42022) 2024-04-21T08:54:42.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dab7fe640 1 -- 172.21.15.57:0/2964287175 learned_addr learned my addr 172.21.15.57:0/2964287175 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.686 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dab7fe640 1 -- 172.21.15.57:0/2964287175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27f5c00 msgr2=0x5654f280d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27f5c00 0x5654f280d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dab7fe640 1 -- 172.21.15.57:0/2964287175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 msgr2=0x5654f27f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.684+0000 7f8dabfff640 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.685+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.685+0000 7f8dab7fe640 1 -- 172.21.15.57:0/2964287175 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f27f8b70 con 0x5654f27ed020 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.685+0000 7f8dabfff640 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.685+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 0x5654f2801910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8da4005e30 tx=0x7f8da403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5024360e322222d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.686+0000 7f8d99ffb640 1 -- 172.21.15.57:0/2964287175 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da4039040 con 0x5654f27ed020 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.686+0000 7f8d99ffb640 1 -- 172.21.15.57:0/2964287175 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da4004040 con 0x5654f27ed020 2024-04-21T08:54:42.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.687+0000 7f8db5264740 1 -- 172.21.15.57:0/2964287175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 msgr2=0x5654f2801910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.687+0000 7f8db5264740 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 0x5654f2801910 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8da4005e30 tx=0x7f8da403f000 comp rx=0 tx=0).stop 2024-04-21T08:54:42.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.693+0000 7f8db5264740 1 -- 172.21.15.57:0/2964287175 shutdown_connections 2024-04-21T08:54:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.693+0000 7f8db5264740 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27f5c00 0x5654f280d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.693+0000 7f8db5264740 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27ed020 0x5654f2801910 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.693+0000 7f8db5264740 1 --2- 172.21.15.57:0/2964287175 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ec700 0x5654f27f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.693+0000 7f8db5264740 1 -- 172.21.15.57:0/2964287175 >> 172.21.15.57:0/2964287175 conn(0x5654f27e6e30 msgr2=0x5654f27f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.695 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.695+0000 7f8db5264740 1 -- 172.21.15.57:0/2964287175 shutdown_connections 2024-04-21T08:54:42.696 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.695+0000 7f8db5264740 1 -- 172.21.15.57:0/2964287175 wait complete. 2024-04-21T08:54:42.696 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.695+0000 7f8db5264740 1 -- start start 2024-04-21T08:54:42.696 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 Processor -- start 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ec700 0x5654f27fde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ed020 0x5654f27fe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9000ec30 con 0x5654f27ed020 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d900100a0 con 0x5654f27ec700 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9000f300 con 0x5654f27f5c00 2024-04-21T08:54:42.697 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.696+0000 7f8db5264740 10 client.0 _ll_register_callbacks cb 0x7f8da0015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:42.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8dabfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ec700 0x5654f27fde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.698 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8dab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ed020 0x5654f27fe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:42028/0 (socket says 172.21.15.57:42028) 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 -- 172.21.15.57:0/3077922171 learned_addr learned my addr 172.21.15.57:0/3077922171 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 -- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ec700 msgr2=0x5654f27fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ec700 0x5654f27fde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 -- 172.21.15.57:0/3077922171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ed020 msgr2=0x5654f27fe350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ed020 0x5654f27fe350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.697+0000 7f8db0c0a640 1 -- 172.21.15.57:0/3077922171 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f27f9d80 con 0x5654f27f5c00 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.698+0000 7f8dab7fe640 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ed020 0x5654f27fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:54:42.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.698+0000 7f8db0c0a640 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f9550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c09c120 tx=0x7f8d9c007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b7e6649611c65f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:42.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.705+0000 7f8d8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:54:42.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.705+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3077922171 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0b4020 con 0x5654f27f5c00 2024-04-21T08:54:42.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.714+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da4004040 con 0x5654f27f5c00 2024-04-21T08:54:42.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.715+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3077922171 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7f8d9c0d0600 con 0x5654f27f5c00 2024-04-21T08:54:42.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.715+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3077922171 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f8d9c01d040 con 0x5654f27f5c00 2024-04-21T08:54:42.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.715+0000 7f8db5264740 10 client.26449 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:42.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.715+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5654f286ba60 con 0x5654f27f5c00 2024-04-21T08:54:42.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.715+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3077922171 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da4004040 con 0x5654f27f5c00 2024-04-21T08:54:42.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3077922171 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f8d9c0b5c30 con 0x5654f27f5c00 2024-04-21T08:54:42.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8db5264740 10 client.26449 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:42.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8db5264740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:42.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8db5264740 20 client.26449 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:42.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8db5264740 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f286c1e0 0x5654f286e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:42.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8db5264740 4 client.26449 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:54:42.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.716+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5654f2758a70 con 0x5654f286c1e0 2024-04-21T08:54:42.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.717+0000 7f8dabfff640 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f286c1e0 0x5654f286e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:42.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.717+0000 7f8d8f7fe640 10 client.26449 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.717+0000 7f8dabfff640 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f286c1e0 0x5654f286e600 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1f66cc52b013c4 server_cookie=da5cf92fb267085a in_seq=0 out_seq=0 2024-04-21T08:54:42.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.719+0000 7f8d8f7fe640 1 -- 172.21.15.57:0/3077922171 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x5654f2758a70 con 0x5654f286c1e0 2024-04-21T08:54:42.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.719+0000 7f8d8f7fe640 10 client.26449 handle_command_reply: tid=0 2024-04-21T08:54:42.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.719+0000 7f8db5264740 1 client.26449 shutdown 2024-04-21T08:54:42.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 msgr2=0x5654f27f9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f9550 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c09c120 tx=0x7f8d9c007f00 comp rx=0 tx=0).stop 2024-04-21T08:54:42.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 shutdown_connections 2024-04-21T08:54:42.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5654f286c1e0 0x5654f286e600 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.721 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5654f27f5c00 0x5654f27f9550 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5654f27ed020 0x5654f27fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 --2- 172.21.15.57:0/3077922171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5654f27ec700 0x5654f27fde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 >> 172.21.15.57:0/3077922171 conn(0x5654f27e6e30 msgr2=0x5654f27e8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8d8f7fe640 0 client.26449 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 shutdown_connections 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.720+0000 7f8db5264740 1 -- 172.21.15.57:0/3077922171 wait complete. 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.721+0000 7f8db5264740 20 client.26449 trim_cache size 0 max 16384 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (15434 inodes in the stack)", 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:54:42.722 INFO:teuthology.orchestra.run.smithi057.stdout: "2f79e585-3b65-4439-ac68-97fc759446ab": { 2024-04-21T08:54:42.723 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:54:42.723 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "2f79e585-3b65-4439-ac68-97fc759446ab", 2024-04-21T08:54:42.723 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:54:42.723 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:54:42.723 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:54:42.723 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:54:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8d940a3d80 msgr2=0x7f8d940a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8d940a3d80 0x7f8d940a6240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8da4009bf0 tx=0x7f8da4000f40 comp rx=0 tx=0).stop 2024-04-21T08:54:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 msgr2=0x7f8dac1ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:42.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 0x7f8dac1ac390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c970 tx=0x7f8d9c00ce30 comp rx=0 tx=0).stop 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 shutdown_connections 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f8d940a3d80 0x7f8d940a6240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dac1abee0 0x7f8dac1ac390 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dac1105f0 0x7f8dac1a7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 --2- 172.21.15.57:0/924784253 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dac1099d0 0x7f8dac1a7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 >> 172.21.15.57:0/924784253 conn(0x7f8dac06ced0 msgr2=0x7f8dac072230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 shutdown_connections 2024-04-21T08:54:42.731 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:54:42.729+0000 7f8db140b640 1 -- 172.21.15.57:0/924784253 wait complete. 2024-04-21T08:54:42.750 INFO:tasks.cephfs.filesystem:scrub status for tag:2f79e585-3b65-4439-ac68-97fc759446ab - {'path': '/', 'tag': '2f79e585-3b65-4439-ac68-97fc759446ab', 'options': 'recursive,force'} 2024-04-21T08:54:42.752 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/swap.o 2024-04-21T08:54:42.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:42 smithi129 ceph-mon[31272]: pgmap v289: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 914 KiB/s wr, 1.73k op/s 2024-04-21T08:54:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:42 smithi136 ceph-mon[31791]: pgmap v289: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 914 KiB/s wr, 1.73k op/s 2024-04-21T08:54:42.949 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/rsa_helper.o 2024-04-21T08:54:43.019 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/numa.o 2024-04-21T08:54:43.042 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/kernel/cpu/microcode/built-in.a 2024-04-21T08:54:43.057 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/rsa-pkcs1pad.o 2024-04-21T08:54:43.115 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/cacheinfo.o 2024-04-21T08:54:43.158 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/consoles.o 2024-04-21T08:54:43.267 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/printk/sysctl.o 2024-04-21T08:54:43.448 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/workingset.o 2024-04-21T08:54:43.584 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/printk/built-in.a 2024-04-21T08:54:43.626 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/debug.o 2024-04-21T08:54:43.713 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/cpuinfo.o 2024-04-21T08:54:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:43 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:54:43.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:43 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:43.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:43 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:54:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:43 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:54:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:43 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:43.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:43 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:54:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:43 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:54:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:43 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:54:43.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:43 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:54:43.898 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/acompress.o 2024-04-21T08:54:43.904 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/user.o 2024-04-21T08:54:44.063 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/numa_64.o 2024-04-21T08:54:44.119 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/scattered.o 2024-04-21T08:54:44.375 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/devices.o 2024-04-21T08:54:44.419 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/gup.o 2024-04-21T08:54:44.805 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/scompress.o 2024-04-21T08:54:44.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:44 smithi129 ceph-mon[31272]: pgmap v290: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 869 KiB/s wr, 1.29k op/s 2024-04-21T08:54:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:44 smithi057 ceph-mon[27968]: pgmap v290: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 869 KiB/s wr, 1.29k op/s 2024-04-21T08:54:44.861 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/amdtopology.o 2024-04-21T08:54:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:44 smithi136 ceph-mon[31791]: pgmap v290: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 869 KiB/s wr, 1.29k op/s 2024-04-21T08:54:44.934 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/topology.o 2024-04-21T08:54:45.029 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/services.o 2024-04-21T08:54:45.082 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/power/poweroff.o 2024-04-21T08:54:45.381 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/interrupts.o 2024-04-21T08:54:45.442 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/power/built-in.a 2024-04-21T08:54:45.462 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/common.o 2024-04-21T08:54:45.524 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mmap_lock.o 2024-04-21T08:54:45.666 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/irqdesc.o 2024-04-21T08:54:45.667 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/algboss.o 2024-04-21T08:54:45.815 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/srat.o 2024-04-21T08:54:45.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:45 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:54:45.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:45 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:54:45.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:45 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:54:45.971 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/loadavg.o 2024-04-21T08:54:46.611 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/highmem.o 2024-04-21T08:54:46.658 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/handle.o 2024-04-21T08:54:46.722 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/meminfo.o 2024-04-21T08:54:46.790 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pkeys.o 2024-04-21T08:54:46.823 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/testmgr.o 2024-04-21T08:54:46.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:46 smithi129 ceph-mon[31272]: pgmap v291: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 870 op/s 2024-04-21T08:54:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:46 smithi057 ceph-mon[27968]: pgmap v291: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 870 op/s 2024-04-21T08:54:46.869 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/kaslr.o 2024-04-21T08:54:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:46 smithi136 ceph-mon[31791]: pgmap v291: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 870 op/s 2024-04-21T08:54:47.197 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/rdrand.o 2024-04-21T08:54:47.250 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/manage.o 2024-04-21T08:54:47.357 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/memory.o 2024-04-21T08:54:47.372 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/match.o 2024-04-21T08:54:47.452 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/cmac.o 2024-04-21T08:54:47.506 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/spurious.o 2024-04-21T08:54:47.550 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/stat.o 2024-04-21T08:54:47.607 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/mm/pti.o 2024-04-21T08:54:47.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:47 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:47.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:47 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:47.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:47 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:54:47.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:47 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:47.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:47 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:47.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:47 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:54:47.857 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/conditional.o 2024-04-21T08:54:47.861 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/bugs.o 2024-04-21T08:54:47.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:47 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:54:47.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:47 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:54:47.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:47 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:54:48.040 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/hmac.o 2024-04-21T08:54:48.116 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/resend.o 2024-04-21T08:54:48.237 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/uptime.o 2024-04-21T08:54:48.559 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/mm/built-in.a 2024-04-21T08:54:48.614 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/chip.o 2024-04-21T08:54:48.616 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/dummychip.o 2024-04-21T08:54:48.682 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/crypto/built-in.a 2024-04-21T08:54:48.705 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/crypto_null.o 2024-04-21T08:54:48.708 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/util.o 2024-04-21T08:54:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:48 smithi129 ceph-mon[31272]: pgmap v292: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 909 KiB/s wr, 781 op/s 2024-04-21T08:54:48.825 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/version.o 2024-04-21T08:54:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:48 smithi057 ceph-mon[27968]: pgmap v292: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 909 KiB/s wr, 781 op/s 2024-04-21T08:54:48.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:48 smithi136 ceph-mon[31791]: pgmap v292: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 909 KiB/s wr, 781 op/s 2024-04-21T08:54:49.020 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/mls.o 2024-04-21T08:54:49.110 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/devres.o 2024-04-21T08:54:49.231 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/softirqs.o 2024-04-21T08:54:49.361 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/namespaces.o 2024-04-21T08:54:49.397 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/md5.o 2024-04-21T08:54:49.418 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/aperfmperf.o 2024-04-21T08:54:49.676 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/sha256_generic.o 2024-04-21T08:54:49.698 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/autoprobe.o 2024-04-21T08:54:49.812 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/self.o 2024-04-21T08:54:50.002 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/sha512_generic.o 2024-04-21T08:54:50.037 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/cpuid-deps.o 2024-04-21T08:54:50.182 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/thread_self.o 2024-04-21T08:54:50.217 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/ss/context.o 2024-04-21T08:54:50.310 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/irqdomain.o 2024-04-21T08:54:50.351 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/sha3_generic.o 2024-04-21T08:54:50.480 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/proc_sysctl.o 2024-04-21T08:54:50.483 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/umwait.o 2024-04-21T08:54:50.571 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mincore.o 2024-04-21T08:54:50.731 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/cbc.o 2024-04-21T08:54:50.750 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mlock.o 2024-04-21T08:54:50.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:50 smithi129 ceph-mon[31272]: pgmap v293: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 1.45k op/s 2024-04-21T08:54:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:50 smithi057 ceph-mon[27968]: pgmap v293: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 1.45k op/s 2024-04-21T08:54:50.866 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/proc_net.o 2024-04-21T08:54:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:50 smithi136 ceph-mon[31791]: pgmap v293: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 1.45k op/s 2024-04-21T08:54:51.127 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/selinux/netlabel.o 2024-04-21T08:54:51.213 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/ctr.o 2024-04-21T08:54:51.220 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/proc.o 2024-04-21T08:54:51.402 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/gcm.o 2024-04-21T08:54:51.478 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/proc.o 2024-04-21T08:54:51.700 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/kcore.o 2024-04-21T08:54:51.817 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/vmcore.o 2024-04-21T08:54:51.872 INFO:tasks.workunit.client.0.smithi057.stdout: MKCAP arch/x86/kernel/cpu/capflags.c 2024-04-21T08:54:51.975 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mmap.o 2024-04-21T08:54:52.076 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/ia32/audit.o 2024-04-21T08:54:52.137 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/ia32/built-in.a 2024-04-21T08:54:52.169 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/migration.o 2024-04-21T08:54:52.201 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/kmsg.o 2024-04-21T08:54:52.415 INFO:tasks.workunit.client.0.smithi057.stdout: AR security/selinux/built-in.a 2024-04-21T08:54:52.460 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/ccm.o 2024-04-21T08:54:52.465 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/integrity/iint.o 2024-04-21T08:54:52.604 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/cpuhotplug.o 2024-04-21T08:54:52.662 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/cpu/powerflags.o 2024-04-21T08:54:52.707 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc/page.o 2024-04-21T08:54:52.799 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/atom/built-in.a 2024-04-21T08:54:52.819 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/net/built-in.a 2024-04-21T08:54:52.821 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/ce4100/built-in.a 2024-04-21T08:54:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:52 smithi129 ceph-mon[31272]: pgmap v294: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 1.98k op/s 2024-04-21T08:54:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:52 smithi057 ceph-mon[27968]: pgmap v294: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 1.98k op/s 2024-04-21T08:54:52.844 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/pm.o 2024-04-21T08:54:52.853 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/platform/efi/memmap.o 2024-04-21T08:54:52.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:52 smithi136 ceph-mon[31791]: pgmap v294: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 1.98k op/s 2024-04-21T08:54:53.064 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/integrity/integrity_audit.o 2024-04-21T08:54:53.079 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/platform/efi/quirks.o 2024-04-21T08:54:53.470 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/aes_generic.o 2024-04-21T08:54:53.571 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/platform/efi/efi.o 2024-04-21T08:54:53.590 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/proc/built-in.a 2024-04-21T08:54:53.638 INFO:tasks.workunit.client.0.smithi057.stdout: AR security/integrity/built-in.a 2024-04-21T08:54:53.682 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/kernfs/mount.o 2024-04-21T08:54:53.684 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/commoncap.o 2024-04-21T08:54:53.696 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/msi.o 2024-04-21T08:54:54.017 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/kernfs/inode.o 2024-04-21T08:54:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.033+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2353793186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0810ece0 msgr2=0x7fde0810d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.033+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2353793186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0810ece0 0x7fde0810d000 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009950 tx=0x7fddf802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2353793186 shutdown_connections 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2353793186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2353793186 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0810ece0 0x7fde0810d000 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2353793186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 0x7fde0810e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2353793186 >> 172.21.15.57:0/2353793186 conn(0x7fde0806cf00 msgr2=0x7fde0806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2353793186 shutdown_connections 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.034+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2353793186 wait complete. 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.035+0000 7fde0cd7e640 1 Processor -- start 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.035+0000 7fde0cd7e640 1 -- start start 2024-04-21T08:54:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0cd7e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0cd7e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 0x7fde0807f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0cd7e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0807f820 0x7fde08137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0cd7e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fde0807cbb0 con 0x7fde0807f820 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0cd7e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fde0807ca30 con 0x7fde0810d730 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0cd7e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fde0807cd30 con 0x7fde0810e330 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:37578/0 (socket says 172.21.15.57:37578) 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 -- 172.21.15.57:0/2382695953 learned_addr learned my addr 172.21.15.57:0/2382695953 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde05d9b640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 0x7fde0807f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde06d9d640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0807f820 0x7fde08137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 -- 172.21.15.57:0/2382695953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 msgr2=0x7fde0807f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 0x7fde0807f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 -- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0807f820 msgr2=0x7fde08137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0807f820 0x7fde08137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.036+0000 7fde0659c640 1 -- 172.21.15.57:0/2382695953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde081377b0 con 0x7fde0810d730 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.037+0000 7fde05d9b640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 0x7fde0807f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.037+0000 7fde06d9d640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0807f820 0x7fde08137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.037+0000 7fde0659c640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807eda0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00b450 tx=0x7fddfc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.038+0000 7fddf77fe640 1 -- 172.21.15.57:0/2382695953 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc015020 con 0x7fde0810d730 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.038+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde08137aa0 con 0x7fde0810d730 2024-04-21T08:54:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.038+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde08137ff0 con 0x7fde0810d730 2024-04-21T08:54:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.044+0000 7fde0f96c740 1 Processor -- start 2024-04-21T08:54:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.044+0000 7fde0f96c740 1 -- start start 2024-04-21T08:54:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.044+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 0x55dbaf513810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.044+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf5141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.045+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 0x55dbaf513810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.045+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 0x55dbaf513810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47620/0 (socket says 172.21.15.57:47620) 2024-04-21T08:54:54.045 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.045+0000 7fde06d9d640 1 -- 172.21.15.57:0/3653196539 learned_addr learned my addr 172.21.15.57:0/3653196539 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:54.045 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.045+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.045 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55dbaf4788d0 con 0x55dbaf513d50 2024-04-21T08:54:54.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dbaf415e80 con 0x55dbaf51d2d0 2024-04-21T08:54:54.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55dbaf474890 con 0x55dbaf514ac0 2024-04-21T08:54:54.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fddf77fe640 1 -- 172.21.15.57:0/2382695953 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddfc011d40 con 0x7fde0810d730 2024-04-21T08:54:54.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fde05d9b640 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fde0659c640 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf5141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.046+0000 7fddf77fe640 1 -- 172.21.15.57:0/2382695953 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc00f340 con 0x7fde0810d730 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.047+0000 7fde06d9d640 1 -- 172.21.15.57:0/3653196539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf51d2d0 msgr2=0x55dbaf51f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.047+0000 7fde06d9d640 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.047+0000 7fde06d9d640 1 -- 172.21.15.57:0/3653196539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 msgr2=0x55dbaf5141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.047+0000 7fde06d9d640 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf5141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.047 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.047+0000 7fddf77fe640 1 -- 172.21.15.57:0/2382695953 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fddfc018020 con 0x7fde0810d730 2024-04-21T08:54:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.048+0000 7fddf77fe640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fddf00a3f50 0x7fddf00a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.048+0000 7fde06d9d640 1 -- 172.21.15.57:0/3653196539 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbaf437320 con 0x55dbaf514ac0 2024-04-21T08:54:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.048+0000 7fde0659c640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fddf00a3f50 0x7fddf00a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.048 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.048+0000 7fde06d9d640 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 0x55dbaf513810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fde00009fd0 tx=0x7fde000108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f02dfcbcf62dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.049+0000 7fde0659c640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fddf00a3f50 0x7fddf00a6410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55dbaf515220 tx=0x7fddfc0d6040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.050+0000 7fddf77fe640 1 -- 172.21.15.57:0/2382695953 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fddfc0d15d0 con 0x7fde0810d730 2024-04-21T08:54:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.050+0000 7fddf4ff9640 1 -- 172.21.15.57:0/3653196539 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00011760 con 0x55dbaf514ac0 2024-04-21T08:54:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.050+0000 7fddf4ff9640 1 -- 172.21.15.57:0/3653196539 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde00011db0 con 0x55dbaf514ac0 2024-04-21T08:54:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.051+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 msgr2=0x55dbaf513810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.051+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 0x55dbaf513810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fde00009fd0 tx=0x7fde000108e0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.052+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 shutdown_connections 2024-04-21T08:54:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.052+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.052+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf5141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.052+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3653196539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf514ac0 0x55dbaf513810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.052+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 >> 172.21.15.57:0/3653196539 conn(0x55dbaf4dac90 msgr2=0x55dbaf50ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.056+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 shutdown_connections 2024-04-21T08:54:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.056+0000 7fde0f96c740 1 -- 172.21.15.57:0/3653196539 wait complete. 2024-04-21T08:54:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.056+0000 7fde0f96c740 1 -- start start 2024-04-21T08:54:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.056+0000 7fde0f96c740 1 Processor -- start 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf51cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf519f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55dbaf520050 con 0x55dbaf513d50 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dbaf400eb0 con 0x55dbaf514ac0 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55dbaf5201d0 con 0x55dbaf51d2d0 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde05d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf519f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.058 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde05d9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf519f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:37594/0 (socket says 172.21.15.57:37594) 2024-04-21T08:54:54.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde05d9b640 1 -- 172.21.15.57:0/3491935775 learned_addr learned my addr 172.21.15.57:0/3491935775 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:54.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf51cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde06d9d640 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.058+0000 7fde0f96c740 10 client.0 _ll_register_callbacks cb 0x55dbaf42db40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:54.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.059+0000 7fde05d9b640 1 -- 172.21.15.57:0/3491935775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 msgr2=0x55dbaf51a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.059 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.059+0000 7fde05d9b640 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.059+0000 7fde05d9b640 1 -- 172.21.15.57:0/3491935775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 msgr2=0x55dbaf51cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.059+0000 7fde05d9b640 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf51cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.059+0000 7fde05d9b640 1 -- 172.21.15.57:0/3491935775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbaf18b210 con 0x55dbaf514ac0 2024-04-21T08:54:54.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.059+0000 7fde05d9b640 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf519f40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036040 tx=0x7fddf802c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8f9f3fe801f21b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.069+0000 7fddedffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:54:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.069+0000 7fddedffb640 1 -- 172.21.15.57:0/3491935775 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8043030 con 0x55dbaf514ac0 2024-04-21T08:54:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.069+0000 7fddedffb640 1 -- 172.21.15.57:0/3491935775 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fddf802cc60 con 0x55dbaf514ac0 2024-04-21T08:54:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.069+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde00011db0 con 0x55dbaf514ac0 2024-04-21T08:54:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.070+0000 7fddedffb640 1 -- 172.21.15.57:0/3491935775 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fddf8050050 con 0x55dbaf514ac0 2024-04-21T08:54:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.070+0000 7fde0f96c740 10 client.26576 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.070+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dbaf257620 con 0x55dbaf514ac0 2024-04-21T08:54:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fddedffb640 1 -- 172.21.15.57:0/3491935775 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde00011db0 con 0x55dbaf514ac0 2024-04-21T08:54:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fddedffb640 1 -- 172.21.15.57:0/3491935775 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fddf8004c30 con 0x55dbaf514ac0 2024-04-21T08:54:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fde0f96c740 10 client.26576 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fde0f96c740 10 client.26576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fde0f96c740 20 client.26576 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf52d6b0 0x55dbaf52fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fde0f96c740 4 client.26576 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:54:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.071+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbaf48c880 con 0x55dbaf52d6b0 2024-04-21T08:54:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.072+0000 7fde06d9d640 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf52d6b0 0x55dbaf52fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.072+0000 7fde06d9d640 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf52d6b0 0x55dbaf52fb50 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b879aaca9593c873 server_cookie=ba20ec9ba69b0535 in_seq=0 out_seq=0 2024-04-21T08:54:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.072+0000 7fddedffb640 10 client.26576 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.073+0000 7fddedffb640 1 -- 172.21.15.57:0/3491935775 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbaf48c880 con 0x55dbaf52d6b0 2024-04-21T08:54:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.073+0000 7fddedffb640 10 client.26576 handle_command_reply: tid=0 2024-04-21T08:54:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.073+0000 7fde0f96c740 1 client.26576 shutdown 2024-04-21T08:54:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 msgr2=0x55dbaf519f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf519f40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036040 tx=0x7fddf802c820 comp rx=0 tx=0).stop 2024-04-21T08:54:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 shutdown_connections 2024-04-21T08:54:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf52d6b0 0x55dbaf52fb50 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf519f40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 --2- 172.21.15.57:0/3491935775 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf513d50 0x55dbaf51cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 >> 172.21.15.57:0/3491935775 conn(0x55dbaf4dac90 msgr2=0x55dbaf510790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fddedffb640 0 client.26576 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 shutdown_connections 2024-04-21T08:54:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 1 -- 172.21.15.57:0/3491935775 wait complete. 2024-04-21T08:54:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.074+0000 7fde0f96c740 20 client.26576 trim_cache size 0 max 16384 2024-04-21T08:54:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.114+0000 7fde0f96c740 1 Processor -- start 2024-04-21T08:54:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.114+0000 7fde0f96c740 1 -- start start 2024-04-21T08:54:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf513d50 0x55dbaf528fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf51c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf513d50 0x55dbaf528fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf513d50 0x55dbaf528fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47646/0 (socket says 172.21.15.57:47646) 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde06d9d640 1 -- 172.21.15.57:0/41141974 learned_addr learned my addr 172.21.15.57:0/41141974 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdde800df80 con 0x55dbaf51d2d0 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdde80107e0 con 0x55dbaf514ac0 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdde800e5d0 con 0x55dbaf513d50 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.115+0000 7fde05d9b640 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf51c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde06d9d640 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde05d9b640 1 -- 172.21.15.57:0/41141974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf513d50 msgr2=0x55dbaf528fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde05d9b640 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf513d50 0x55dbaf528fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde05d9b640 1 -- 172.21.15.57:0/41141974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf51d2d0 msgr2=0x55dbaf51cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde05d9b640 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde05d9b640 1 -- 172.21.15.57:0/41141974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbaf513020 con 0x55dbaf514ac0 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde06d9d640 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:54:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.116+0000 7fde05d9b640 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf51c670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fddf802e010 tx=0x7fddf802bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23280e5728eb5182 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.118+0000 7fddeffff640 1 -- 172.21.15.57:0/41141974 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8043040 con 0x55dbaf514ac0 2024-04-21T08:54:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.118+0000 7fddeffff640 1 -- 172.21.15.57:0/41141974 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf80037b0 con 0x55dbaf514ac0 2024-04-21T08:54:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.119+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 msgr2=0x55dbaf51c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.119+0000 7fde0f96c740 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf51c670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fddf802e010 tx=0x7fddf802bcd0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 shutdown_connections 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf51d2d0 0x55dbaf51cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf514ac0 0x55dbaf51c670 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 --2- 172.21.15.57:0/41141974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf513d50 0x55dbaf528fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 >> 172.21.15.57:0/41141974 conn(0x55dbaf4dac90 msgr2=0x55dbaf518810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 shutdown_connections 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.120+0000 7fde0f96c740 1 -- 172.21.15.57:0/41141974 wait complete. 2024-04-21T08:54:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 -- start start 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 Processor -- start 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf513d50 0x55dbaf525520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf514ac0 0x55dbaf525a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf520bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdde800eca0 con 0x55dbaf514ac0 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdde8010110 con 0x55dbaf513d50 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdde800f370 con 0x55dbaf51d2d0 2024-04-21T08:54:54.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.121+0000 7fde0f96c740 10 client.0 _ll_register_callbacks cb 0x7fddf8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:54:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf513d50 0x55dbaf525520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde06d9d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf513d50 0x55dbaf525520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:37618/0 (socket says 172.21.15.57:37618) 2024-04-21T08:54:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde06d9d640 1 -- 172.21.15.57:0/2222126226 learned_addr learned my addr 172.21.15.57:0/2222126226 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:54:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde06d9d640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf520bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.122 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde05d9b640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf514ac0 0x55dbaf525a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde05d9b640 1 -- 172.21.15.57:0/2222126226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 msgr2=0x55dbaf520bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.122+0000 7fde05d9b640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf520bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.123+0000 7fde05d9b640 1 -- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf513d50 msgr2=0x55dbaf525520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:54:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.123+0000 7fde05d9b640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf513d50 0x55dbaf525520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:54.123 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.123+0000 7fde05d9b640 1 -- 172.21.15.57:0/2222126226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbaf5213c0 con 0x55dbaf514ac0 2024-04-21T08:54:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.123+0000 7fde05d9b640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf514ac0 0x55dbaf525a60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x55dbaf51afc0 tx=0x7fddf8004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d144c9d4b12d7c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:54:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.123+0000 7fddedffb640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:54:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.124+0000 7fddedffb640 1 -- 172.21.15.57:0/2222126226 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8043040 con 0x55dbaf514ac0 2024-04-21T08:54:54.124 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.124+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddf80037b0 con 0x55dbaf514ac0 2024-04-21T08:54:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.124+0000 7fddedffb640 1 -- 172.21.15.57:0/2222126226 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9475+0+0 (secure 0 0 0) 0x7fddf802cc60 con 0x55dbaf514ac0 2024-04-21T08:54:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.124+0000 7fddedffb640 1 -- 172.21.15.57:0/2222126226 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fddf8003b60 con 0x55dbaf514ac0 2024-04-21T08:54:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.124+0000 7fde0f96c740 10 client.17124 fetch_fsmap learned FSMap version 21 2024-04-21T08:54:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.124+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55dbaf593910 con 0x55dbaf514ac0 2024-04-21T08:54:54.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.125+0000 7fddedffb640 1 -- 172.21.15.57:0/2222126226 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddf80037b0 con 0x55dbaf514ac0 2024-04-21T08:54:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fddedffb640 1 -- 172.21.15.57:0/2222126226 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fddf803f440 con 0x55dbaf514ac0 2024-04-21T08:54:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fde0f96c740 10 client.17124 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:54:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fde0f96c740 10 client.17124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:54:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fde0f96c740 20 client.17124 populate_metadata read hostname 'smithi057' 2024-04-21T08:54:54.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fde0f96c740 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf593f00 0x55dbaf596320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:54:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fde0f96c740 4 client.17124 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:54:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.126+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbaf4183d0 con 0x55dbaf593f00 2024-04-21T08:54:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.127+0000 7fde06d9d640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf593f00 0x55dbaf596320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:54:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.127+0000 7fde06d9d640 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf593f00 0x55dbaf596320 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97db527e209efb61 server_cookie=57e7db808e599569 in_seq=0 out_seq=0 2024-04-21T08:54:54.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:54.127+0000 7fddedffb640 10 client.17124 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:54.150 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mmu_gather.o 2024-04-21T08:54:54.303 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/crc32c_generic.o 2024-04-21T08:54:54.403 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/kernfs/dir.o 2024-04-21T08:54:54.655 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/platform/efi/efi_64.o 2024-04-21T08:54:54.725 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/authenc.o 2024-04-21T08:54:54.793 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/authencesn.o 2024-04-21T08:54:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:54 smithi057 ceph-mon[27968]: pgmap v295: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 3.45k op/s 2024-04-21T08:54:54.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:54 smithi136 ceph-mon[31791]: pgmap v295: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 3.45k op/s 2024-04-21T08:54:54.891 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/min_addr.o 2024-04-21T08:54:54.931 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/affinity.o 2024-04-21T08:54:54.941 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mprotect.o 2024-04-21T08:54:55.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:54 smithi129 ceph-mon[31272]: pgmap v295: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 3.45k op/s 2024-04-21T08:54:55.420 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/security.o 2024-04-21T08:54:55.425 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/kernfs/file.o 2024-04-21T08:54:55.446 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq/matrix.o 2024-04-21T08:54:55.647 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/rng.o 2024-04-21T08:54:55.695 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/drbg.o 2024-04-21T08:54:55.718 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/platform/efi/efi_stub_64.o 2024-04-21T08:54:55.777 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/platform/efi/efi_thunk_64.o 2024-04-21T08:54:55.819 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/platform/efi/runtime-map.o 2024-04-21T08:54:56.076 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mremap.o 2024-04-21T08:54:56.346 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/kernfs/symlink.o 2024-04-21T08:54:56.440 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/msync.o 2024-04-21T08:54:56.490 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/irq/built-in.a 2024-04-21T08:54:56.502 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/jitterentropy.o 2024-04-21T08:54:56.602 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/efi/built-in.a 2024-04-21T08:54:56.657 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/jitterentropy-kcapi.o 2024-04-21T08:54:56.843 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:56 smithi057 ceph-mon[27968]: pgmap v296: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.4 MiB/s wr, 4.11k op/s 2024-04-21T08:54:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:56 smithi136 ceph-mon[31791]: pgmap v296: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.4 MiB/s wr, 4.11k op/s 2024-04-21T08:54:57.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:56 smithi129 ceph-mon[31272]: pgmap v296: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.4 MiB/s wr, 4.11k op/s 2024-04-21T08:54:57.202 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/kernfs/built-in.a 2024-04-21T08:54:57.270 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/page_vma_mapped.o 2024-04-21T08:54:57.326 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/ghash-generic.o 2024-04-21T08:54:57.347 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/pagewalk.o 2024-04-21T08:54:57.594 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/rcu/update.o 2024-04-21T08:54:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.618+0000 7fddedffb640 1 -- 172.21.15.57:0/2222126226 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbaf4183d0 con 0x55dbaf593f00 2024-04-21T08:54:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.619+0000 7fddedffb640 10 client.17124 handle_command_reply: tid=0 2024-04-21T08:54:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.619+0000 7fde0f96c740 1 client.17124 shutdown 2024-04-21T08:54:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.619+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf514ac0 msgr2=0x55dbaf525a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.619+0000 7fde0f96c740 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf514ac0 0x55dbaf525a60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x55dbaf51afc0 tx=0x7fddf8004460 comp rx=0 tx=0).stop 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 shutdown_connections 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55dbaf593f00 0x55dbaf596320 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55dbaf51d2d0 0x55dbaf520bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55dbaf514ac0 0x55dbaf525a60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 --2- 172.21.15.57:0/2222126226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dbaf513d50 0x55dbaf525520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 >> 172.21.15.57:0/2222126226 conn(0x55dbaf4dac90 msgr2=0x55dbaf50d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fddedffb640 0 client.17124 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 shutdown_connections 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.621+0000 7fde0f96c740 1 -- 172.21.15.57:0/2222126226 wait complete. 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.622+0000 7fde0f96c740 20 client.17124 trim_cache size 0 max 16384 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:54:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:54:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.624+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fddf00a3f50 msgr2=0x7fddf00a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.624+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fddf00a3f50 0x7fddf00a6410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55dbaf515220 tx=0x7fddfc0d6040 comp rx=0 tx=0).stop 2024-04-21T08:54:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.624+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 msgr2=0x7fde0807eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:54:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.624+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807eda0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fddfc00b450 tx=0x7fddfc00b910 comp rx=0 tx=0).stop 2024-04-21T08:54:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.625+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 shutdown_connections 2024-04-21T08:54:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.625+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fddf00a3f50 0x7fddf00a6410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.625+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fde0807f820 0x7fde08137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.625+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde0810e330 0x7fde0807f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.625+0000 7fde0cd7e640 1 --2- 172.21.15.57:0/2382695953 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde0810d730 0x7fde0807eda0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:54:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.625+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 >> 172.21.15.57:0/2382695953 conn(0x7fde0806cf00 msgr2=0x7fde0810b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:54:57.628 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.628+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 shutdown_connections 2024-04-21T08:54:57.629 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:54:57.629+0000 7fde0cd7e640 1 -- 172.21.15.57:0/2382695953 wait complete. 2024-04-21T08:54:57.633 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/geode/built-in.a 2024-04-21T08:54:57.635 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sysfs/file.o 2024-04-21T08:54:57.671 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/iris/built-in.a 2024-04-21T08:54:57.718 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/platform/intel/iosf_mbi.o 2024-04-21T08:54:57.785 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/hash_info.o 2024-04-21T08:54:57.840 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/lsm_audit.o 2024-04-21T08:54:57.846 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/rsapubkey.asn1.o 2024-04-21T08:54:57.886 INFO:tasks.workunit.client.0.smithi057.stdout: CC crypto/rsaprivkey.asn1.o 2024-04-21T08:54:57.938 INFO:tasks.workunit.client.0.smithi057.stdout: AR crypto/built-in.a 2024-04-21T08:54:58.018 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/devpts/inode.o 2024-04-21T08:54:58.085 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/acpi/boot.o 2024-04-21T08:54:58.214 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/pgtable-generic.o 2024-04-21T08:54:58.419 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/intel/built-in.a 2024-04-21T08:54:58.431 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sysfs/dir.o 2024-04-21T08:54:58.481 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/intel-mid/built-in.a 2024-04-21T08:54:58.503 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/intel-quark/built-in.a 2024-04-21T08:54:58.529 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/olpc/built-in.a 2024-04-21T08:54:58.555 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/scx200/built-in.a 2024-04-21T08:54:58.584 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/ts5500/built-in.a 2024-04-21T08:54:58.612 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/uv/built-in.a 2024-04-21T08:54:58.626 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/platform/built-in.a 2024-04-21T08:54:58.704 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sysfs/symlink.o 2024-04-21T08:54:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:54:58 smithi057 ceph-mon[27968]: pgmap v297: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.4 MiB/s wr, 4.54k op/s 2024-04-21T08:54:58.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:54:58 smithi136 ceph-mon[31791]: pgmap v297: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.4 MiB/s wr, 4.54k op/s 2024-04-21T08:54:58.889 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/devpts/built-in.a 2024-04-21T08:54:58.918 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sysfs/mount.o 2024-04-21T08:54:58.926 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/rmap.o 2024-04-21T08:54:58.938 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/netfs/buffered_read.o 2024-04-21T08:54:59.042 INFO:tasks.workunit.client.0.smithi057.stdout: CC security/device_cgroup.o 2024-04-21T08:54:59.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:54:58 smithi129 ceph-mon[31272]: pgmap v297: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.4 MiB/s wr, 4.54k op/s 2024-04-21T08:54:59.201 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/acpi/sleep.o 2024-04-21T08:54:59.307 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/vmalloc.o 2024-04-21T08:54:59.407 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/rcu/sync.o 2024-04-21T08:54:59.692 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/rcu/srcutree.o 2024-04-21T08:54:59.722 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sysfs/group.o 2024-04-21T08:54:59.763 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/netfs/io.o 2024-04-21T08:54:59.935 INFO:tasks.workunit.client.0.smithi057.stdout: AS arch/x86/kernel/acpi/wakeup_64.o 2024-04-21T08:54:59.987 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/acpi/cppc.o 2024-04-21T08:55:00.281 INFO:tasks.workunit.client.0.smithi057.stdout: AR security/built-in.a 2024-04-21T08:55:00.307 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/sysfs/built-in.a 2024-04-21T08:55:00.348 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/process_vm_access.o 2024-04-21T08:55:00.359 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/balloc.o 2024-04-21T08:55:00.518 INFO:tasks.workunit.client.0.smithi057.stdout: CC arch/x86/kernel/acpi/cstate.o 2024-04-21T08:55:00.610 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/jbd2/transaction.o 2024-04-21T08:55:00.659 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/netfs/iterator.o 2024-04-21T08:55:00.693 INFO:tasks.workunit.client.0.smithi057.stderr:./arch/x86/kernel/cpu/mkcapflags.sh: line 46: printf: write error: Resource temporarily unavailable 2024-04-21T08:55:00.697 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/rcu/tree.o 2024-04-21T08:55:00.699 INFO:tasks.workunit.client.0.smithi057.stderr:make[5]: *** [arch/x86/kernel/cpu/Makefile:64: arch/x86/kernel/cpu/capflags.c] Error 1 2024-04-21T08:55:00.699 INFO:tasks.workunit.client.0.smithi057.stderr:make[4]: *** [scripts/Makefile.build:480: arch/x86/kernel/cpu] Error 2 2024-04-21T08:55:00.699 INFO:tasks.workunit.client.0.smithi057.stderr:make[4]: *** Waiting for unfinished jobs.... 2024-04-21T08:55:00.706 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/page_alloc.o 2024-04-21T08:55:00.875 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:00 smithi129 ceph-mon[31272]: pgmap v298: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 928 KiB/s wr, 5.37k op/s 2024-04-21T08:55:00.875 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:00 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:00 smithi136 ceph-mon[31791]: pgmap v298: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 928 KiB/s wr, 5.37k op/s 2024-04-21T08:55:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:00 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:01.037 INFO:tasks.workunit.client.0.smithi057.stdout: AR arch/x86/kernel/acpi/built-in.a 2024-04-21T08:55:01.087 INFO:tasks.workunit.client.0.smithi057.stderr:make[3]: *** [scripts/Makefile.build:480: arch/x86/kernel] Error 2 2024-04-21T08:55:01.087 INFO:tasks.workunit.client.0.smithi057.stderr:make[2]: *** [scripts/Makefile.build:480: arch/x86] Error 2 2024-04-21T08:55:01.087 INFO:tasks.workunit.client.0.smithi057.stderr:make[2]: *** Waiting for unfinished jobs.... 2024-04-21T08:55:01.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:00 smithi057 ceph-mon[27968]: pgmap v298: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 928 KiB/s wr, 5.37k op/s 2024-04-21T08:55:01.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:00 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:01.096 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ramfs/inode.o 2024-04-21T08:55:01.114 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ramfs/file-mmu.o 2024-04-21T08:55:01.301 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/netfs/main.o 2024-04-21T08:55:01.673 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/init-mm.o 2024-04-21T08:55:01.797 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/ramfs/built-in.a 2024-04-21T08:55:01.884 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/bitmap.o 2024-04-21T08:55:01.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:01 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:55:01.958 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/hugetlbfs/inode.o 2024-04-21T08:55:01.992 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/jbd2/commit.o 2024-04-21T08:55:02.019 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/memblock.o 2024-04-21T08:55:02.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:01 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:55:02.097 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:01 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:55:02.423 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/netfs/objects.o 2024-04-21T08:55:02.793 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/cache.o 2024-04-21T08:55:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:02 smithi136 ceph-mon[31791]: pgmap v299: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 523 KiB/s wr, 5.34k op/s 2024-04-21T08:55:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:02 smithi136 ceph-mon[31791]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:55:03.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:02 smithi129 ceph-mon[31272]: pgmap v299: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 523 KiB/s wr, 5.34k op/s 2024-04-21T08:55:03.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:02 smithi129 ceph-mon[31272]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:55:03.093 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:02 smithi057 ceph-mon[27968]: pgmap v299: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 523 KiB/s wr, 5.34k op/s 2024-04-21T08:55:03.094 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:02 smithi057 ceph-mon[27968]: osdmap e103: 12 total, 12 up, 12 in 2024-04-21T08:55:03.150 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/block_validity.o 2024-04-21T08:55:03.635 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/netfs/built-in.a 2024-04-21T08:55:03.822 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/namei.o 2024-04-21T08:55:03.853 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/hugetlbfs/built-in.a 2024-04-21T08:55:03.984 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/dir.o 2024-04-21T08:55:03.996 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/madvise.o 2024-04-21T08:55:04.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:03 smithi129 ceph-mon[31272]: pgmap v301: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 432 KiB/s wr, 4.80k op/s 2024-04-21T08:55:04.081 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/client.o 2024-04-21T08:55:04.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:03 smithi057 ceph-mon[27968]: pgmap v301: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 432 KiB/s wr, 4.80k op/s 2024-04-21T08:55:04.123 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/jbd2/recovery.o 2024-04-21T08:55:04.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:03 smithi136 ceph-mon[31791]: pgmap v301: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 432 KiB/s wr, 4.80k op/s 2024-04-21T08:55:04.407 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/dir.o 2024-04-21T08:55:04.673 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/ext4_jbd2.o 2024-04-21T08:55:04.823 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/inode.o 2024-04-21T08:55:05.576 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/jbd2/checkpoint.o 2024-04-21T08:55:05.712 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/fatent.o 2024-04-21T08:55:06.082 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/page_io.o 2024-04-21T08:55:06.106 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/extents.o 2024-04-21T08:55:06.414 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/rcu/rcu_segcblist.o 2024-04-21T08:55:06.456 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/dir.o 2024-04-21T08:55:06.561 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/extents_status.o 2024-04-21T08:55:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:06 smithi129 ceph-mon[31272]: pgmap v302: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 557 KiB/s wr, 4.25k op/s 2024-04-21T08:55:06.580 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/dir.o 2024-04-21T08:55:06.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:06 smithi057 ceph-mon[27968]: pgmap v302: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 557 KiB/s wr, 4.25k op/s 2024-04-21T08:55:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:06 smithi136 ceph-mon[31791]: pgmap v302: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 557 KiB/s wr, 4.25k op/s 2024-04-21T08:55:06.997 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/jbd2/revoke.o 2024-04-21T08:55:07.381 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/file.o 2024-04-21T08:55:07.383 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/rcu/built-in.a 2024-04-21T08:55:07.567 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/util.o 2024-04-21T08:55:07.750 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/swap_state.o 2024-04-21T08:55:07.764 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/livepatch/built-in.a 2024-04-21T08:55:07.859 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma/mapping.o 2024-04-21T08:55:08.017 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/jbd2/journal.o 2024-04-21T08:55:08.521 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/rock.o 2024-04-21T08:55:08.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:08 smithi129 ceph-mon[31272]: pgmap v303: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 659 KiB/s wr, 3.37k op/s 2024-04-21T08:55:08.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:08 smithi057 ceph-mon[27968]: pgmap v303: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 659 KiB/s wr, 3.37k op/s 2024-04-21T08:55:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:08 smithi136 ceph-mon[31791]: pgmap v303: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 659 KiB/s wr, 3.37k op/s 2024-04-21T08:55:08.976 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/inode.o 2024-04-21T08:55:09.334 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/file.o 2024-04-21T08:55:09.343 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma/direct.o 2024-04-21T08:55:09.672 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/swapfile.o 2024-04-21T08:55:10.270 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/export.o 2024-04-21T08:55:10.565 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/misc.o 2024-04-21T08:55:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:10 smithi129 ceph-mon[31272]: pgmap v304: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 2.05k op/s 2024-04-21T08:55:10.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:10 smithi057 ceph-mon[27968]: pgmap v304: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 2.05k op/s 2024-04-21T08:55:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:10 smithi136 ceph-mon[31791]: pgmap v304: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 2.05k op/s 2024-04-21T08:55:10.721 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma/ops_helpers.o 2024-04-21T08:55:10.794 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/joliet.o 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 -- 172.21.15.57:0/2465727339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301810e330 msgr2=0x7f301810e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 --2- 172.21.15.57:0/2465727339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301810e330 0x7f301810e710 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f301000afd0 tx=0x7f301002cc50 comp rx=0 tx=0).stop 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 -- 172.21.15.57:0/2465727339 shutdown_connections 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 --2- 172.21.15.57:0/2465727339 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f301807a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 --2- 172.21.15.57:0/2465727339 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f301810ece0 0x7f301810d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 --2- 172.21.15.57:0/2465727339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301810e330 0x7f301810e710 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.895+0000 7f301ee81640 1 -- 172.21.15.57:0/2465727339 >> 172.21.15.57:0/2465727339 conn(0x7f301806cf00 msgr2=0x7f301806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.898+0000 7f301ee81640 1 -- 172.21.15.57:0/2465727339 shutdown_connections 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.898+0000 7f301ee81640 1 -- 172.21.15.57:0/2465727339 wait complete. 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.898+0000 7f301ee81640 1 Processor -- start 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.898+0000 7f301ee81640 1 -- start start 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301ee81640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f3018084b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301ee81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f301810e330 0x7f301807ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301ee81640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301807f190 0x7f301807f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301ee81640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f301807d550 con 0x7f301810e330 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301ee81640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f301807d3d0 con 0x7f301810d730 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301ee81640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f301807d6d0 con 0x7f301807f190 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f3018084b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f3018084b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44634/0 (socket says 172.21.15.57:44634) 2024-04-21T08:55:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 -- 172.21.15.57:0/326305706 learned_addr learned my addr 172.21.15.57:0/326305706 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:10.902 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301d41e640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301807f190 0x7f301807f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.902 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 -- 172.21.15.57:0/326305706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301807f190 msgr2=0x7f301807f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:10.902 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301807f190 0x7f301807f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.902 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 -- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f301810e330 msgr2=0x7f301807ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:10.902 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.899+0000 7f301cc1d640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f301810e330 0x7f301807ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.903 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.900+0000 7f301cc1d640 1 -- 172.21.15.57:0/326305706 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30181bc260 con 0x7f301810d730 2024-04-21T08:55:10.903 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.900+0000 7f301cc1d640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f3018084b90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f301000d090 tx=0x7f3010034900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.904+0000 7f3015ffb640 1 -- 172.21.15.57:0/326305706 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010087070 con 0x7f301810d730 2024-04-21T08:55:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.904+0000 7f3015ffb640 1 -- 172.21.15.57:0/326305706 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3010003e20 con 0x7f301810d730 2024-04-21T08:55:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.904+0000 7f3015ffb640 1 -- 172.21.15.57:0/326305706 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010004280 con 0x7f301810d730 2024-04-21T08:55:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.904+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301810f230 con 0x7f301810d730 2024-04-21T08:55:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.904+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3018113b10 con 0x7f301810d730 2024-04-21T08:55:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.914+0000 7f3021a76740 1 Processor -- start 2024-04-21T08:55:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.914+0000 7f3015ffb640 1 -- 172.21.15.57:0/326305706 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3010004420 con 0x7f301810d730 2024-04-21T08:55:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.914+0000 7f3021a76740 1 -- start start 2024-04-21T08:55:10.920 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.920+0000 7f3021a76740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 0x561f1e4ab810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.920+0000 7f3021a76740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.920+0000 7f301d41e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 0x561f1e4ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.920+0000 7f301d41e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 0x561f1e4ab810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44640/0 (socket says 172.21.15.57:44640) 2024-04-21T08:55:10.944 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.944+0000 7f3021a76740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4b52d0 0x561f1e4b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.944+0000 7f3021a76740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561f1e4108d0 con 0x561f1e4b52d0 2024-04-21T08:55:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.944+0000 7f3021a76740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561f1e3ade80 con 0x561f1e4acac0 2024-04-21T08:55:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.944+0000 7f3021a76740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561f1e40c890 con 0x561f1e4abd50 2024-04-21T08:55:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.944+0000 7f301d41e640 1 -- 172.21.15.57:0/2722101973 learned_addr learned my addr 172.21.15.57:0/2722101973 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.946+0000 7f3017fff640 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4b52d0 0x561f1e4b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.947 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.947+0000 7f301cc1d640 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.948+0000 7f301d41e640 1 -- 172.21.15.57:0/2722101973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 msgr2=0x561f1e4ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.948+0000 7f301d41e640 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.948+0000 7f301d41e640 1 -- 172.21.15.57:0/2722101973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4b52d0 msgr2=0x561f1e4b76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.948+0000 7f301d41e640 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4b52d0 0x561f1e4b76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.948+0000 7f301d41e640 1 -- 172.21.15.57:0/2722101973 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f1e3cf320 con 0x561f1e4acac0 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.948+0000 7f301d41e640 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 0x561f1e4ab810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f300800b630 tx=0x7f300800baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af672bcaff3e4aca server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.949+0000 7f2ffeffd640 1 -- 172.21.15.57:0/2722101973 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008003b20 con 0x561f1e4acac0 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.949+0000 7f2ffeffd640 1 -- 172.21.15.57:0/2722101973 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3008015070 con 0x561f1e4acac0 2024-04-21T08:55:10.949 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.949+0000 7f2ffeffd640 1 -- 172.21.15.57:0/2722101973 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30080114a0 con 0x561f1e4acac0 2024-04-21T08:55:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.950+0000 7f3021a76740 1 -- 172.21.15.57:0/2722101973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 msgr2=0x561f1e4ab810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:10.950 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.950+0000 7f3021a76740 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 0x561f1e4ab810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f300800b630 tx=0x7f300800baf0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.954+0000 7f3015ffb640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30000a3f20 0x7f30000a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.954+0000 7f301d41e640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30000a3f20 0x7f30000a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.954+0000 7f3015ffb640 1 -- 172.21.15.57:0/326305706 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f301007e030 con 0x7f301810d730 2024-04-21T08:55:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.955+0000 7f3021a76740 1 -- 172.21.15.57:0/2722101973 shutdown_connections 2024-04-21T08:55:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.955+0000 7f3021a76740 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4b52d0 0x561f1e4b76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.954 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.955+0000 7f3021a76740 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.955+0000 7f3021a76740 1 --2- 172.21.15.57:0/2722101973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4acac0 0x561f1e4ab810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.955+0000 7f3021a76740 1 -- 172.21.15.57:0/2722101973 >> 172.21.15.57:0/2722101973 conn(0x561f1e472c90 msgr2=0x561f1e4a6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:10.955 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.956+0000 7f3021a76740 1 -- 172.21.15.57:0/2722101973 shutdown_connections 2024-04-21T08:55:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.956+0000 7f3021a76740 1 -- 172.21.15.57:0/2722101973 wait complete. 2024-04-21T08:55:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 -- start start 2024-04-21T08:55:10.956 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 Processor -- start 2024-04-21T08:55:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 0x561f1e4b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.957 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 0x561f1e4b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561f1e4b8000 con 0x561f1e4abd50 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561f1e398eb0 con 0x561f1e4b52d0 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561f1e4b8180 con 0x561f1e4acac0 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.957+0000 7f3021a76740 10 client.0 _ll_register_callbacks cb 0x561f1e3c5b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 0x561f1e4b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 0x561f1e4b24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:44656/0 (socket says 172.21.15.57:44656) 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 -- 172.21.15.57:0/96082930 learned_addr learned my addr 172.21.15.57:0/96082930 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f3017fff640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 -- 172.21.15.57:0/96082930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 msgr2=0x561f1e4b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 -- 172.21.15.57:0/96082930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 msgr2=0x561f1e4b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 0x561f1e4b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f301d41e640 1 -- 172.21.15.57:0/96082930 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f1e123210 con 0x561f1e4b52d0 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.958+0000 7f3017fff640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.959+0000 7f301d41e640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 0x561f1e4b24a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3008005910 tx=0x7f300800b2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=752a75ee18a0a139 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.961+0000 7f301d41e640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30000a3f20 0x7f30000a63e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3008004420 tx=0x7f3008011680 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f2ff7fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f2ff7fff640 1 -- 172.21.15.57:0/96082930 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008035060 con 0x561f1e4b52d0 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3008015070 con 0x561f1e4b52d0 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f2ff7fff640 1 -- 172.21.15.57:0/96082930 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f3008010ba0 con 0x561f1e4b52d0 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f2ff7fff640 1 -- 172.21.15.57:0/96082930 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f300803c050 con 0x561f1e4b52d0 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f3021a76740 10 client.26594 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:10.967 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.967+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561f1e1ef620 con 0x561f1e4b52d0 2024-04-21T08:55:10.976 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f2ff7fff640 1 -- 172.21.15.57:0/96082930 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3008015070 con 0x561f1e4b52d0 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f2ff7fff640 1 -- 172.21.15.57:0/96082930 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f3008025560 con 0x561f1e4b52d0 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f3021a76740 10 client.26594 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f3021a76740 10 client.26594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f3021a76740 20 client.26594 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f3021a76740 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e4cde30 0x561f1e4d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f3021a76740 4 client.26594 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f1e424880 con 0x561f1e4cde30 2024-04-21T08:55:10.977 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.977+0000 7f301cc1d640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e4cde30 0x561f1e4d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.978+0000 7f2ff7fff640 10 client.26594 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.978+0000 7f301cc1d640 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e4cde30 0x561f1e4d02d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12e46ebddab391f4 server_cookie=a412c8890a0ec817 in_seq=0 out_seq=0 2024-04-21T08:55:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.978+0000 7f2ff7fff640 1 -- 172.21.15.57:0/96082930 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f1e424880 con 0x561f1e4cde30 2024-04-21T08:55:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.978+0000 7f2ff7fff640 10 client.26594 handle_command_reply: tid=0 2024-04-21T08:55:10.978 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.978+0000 7f3021a76740 1 client.26594 shutdown 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 msgr2=0x561f1e4b24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 0x561f1e4b24a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3008005910 tx=0x7f300800b2e0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 shutdown_connections 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e4cde30 0x561f1e4d02d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b52d0 0x561f1e4b24a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 --2- 172.21.15.57:0/96082930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 0x561f1e4b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 >> 172.21.15.57:0/96082930 conn(0x561f1e472c90 msgr2=0x561f1e4a8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:10.979 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.979+0000 7f2ff7fff640 0 client.26594 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.985+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 shutdown_connections 2024-04-21T08:55:10.984 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.985+0000 7f3021a76740 1 -- 172.21.15.57:0/96082930 wait complete. 2024-04-21T08:55:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:10.985+0000 7f3021a76740 20 client.26594 trim_cache size 0 max 16384 2024-04-21T08:55:11.039 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.039+0000 7f3021a76740 1 Processor -- start 2024-04-21T08:55:11.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.043+0000 7f3021a76740 1 -- start start 2024-04-21T08:55:11.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.043+0000 7f3021a76740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.043+0000 7f3021a76740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4acac0 0x561f1e4b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.043+0000 7f3021a76740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f3021a76740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f300400e150 con 0x561f1e4acac0 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f3021a76740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30040109b0 con 0x561f1e4b2d20 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f3021a76740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f300400e7a0 con 0x561f1e4abd50 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ad7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:60800/0 (socket says 172.21.15.57:60800) 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 -- 172.21.15.57:0/1506713856 learned_addr learned my addr 172.21.15.57:0/1506713856 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301d41e640 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 -- 172.21.15.57:0/1506713856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 msgr2=0x561f1e4b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:11.045 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 -- 172.21.15.57:0/1506713856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4acac0 msgr2=0x561f1e4b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4acac0 0x561f1e4b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.044+0000 7f301cc1d640 1 -- 172.21.15.57:0/1506713856 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f1e4ab020 con 0x561f1e4abd50 2024-04-21T08:55:11.046 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.045+0000 7f301cc1d640 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ad7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30100ff020 tx=0x7f3010084000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22c65c118e743887 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:11.060 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.061+0000 7f2ffe7fc640 1 -- 172.21.15.57:0/1506713856 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100078b0 con 0x561f1e4abd50 2024-04-21T08:55:11.061 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.061+0000 7f2ffe7fc640 1 -- 172.21.15.57:0/1506713856 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3010102d50 con 0x561f1e4abd50 2024-04-21T08:55:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.064+0000 7f2ffe7fc640 1 -- 172.21.15.57:0/1506713856 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010004710 con 0x561f1e4abd50 2024-04-21T08:55:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.065+0000 7f3021a76740 1 -- 172.21.15.57:0/1506713856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 msgr2=0x561f1e4ad7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.065+0000 7f3021a76740 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ad7c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30100ff020 tx=0x7f3010084000 comp rx=0 tx=0).stop 2024-04-21T08:55:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.065+0000 7f3021a76740 1 -- 172.21.15.57:0/1506713856 shutdown_connections 2024-04-21T08:55:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.065+0000 7f3021a76740 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.065+0000 7f3021a76740 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4acac0 0x561f1e4b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.065+0000 7f3021a76740 1 --2- 172.21.15.57:0/1506713856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4abd50 0x561f1e4ad7c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.066 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.066+0000 7f3021a76740 1 -- 172.21.15.57:0/1506713856 >> 172.21.15.57:0/1506713856 conn(0x561f1e472c90 msgr2=0x561f1e4ad0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:11.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.069+0000 7f3021a76740 1 -- 172.21.15.57:0/1506713856 shutdown_connections 2024-04-21T08:55:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.069+0000 7f3021a76740 1 -- 172.21.15.57:0/1506713856 wait complete. 2024-04-21T08:55:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.069+0000 7f3021a76740 1 -- start start 2024-04-21T08:55:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.069+0000 7f3021a76740 1 Processor -- start 2024-04-21T08:55:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 0x561f1e4c7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f300400ee70 con 0x561f1e4abd50 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30040102e0 con 0x561f1e4b2d20 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f300400f540 con 0x561f1e4acac0 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3021a76740 10 client.0 _ll_register_callbacks cb 0x7f300c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.070+0000 7f3017fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4c2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:60812/0 (socket says 172.21.15.57:60812) 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 -- 172.21.15.57:0/1215953503 learned_addr learned my addr 172.21.15.57:0/1215953503 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:11.071 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f301d41e640 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 -- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 msgr2=0x561f1e4c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 -- 172.21.15.57:0/1215953503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 msgr2=0x561f1e4c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 0x561f1e4c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 -- 172.21.15.57:0/1215953503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f1e535460 con 0x561f1e4acac0 2024-04-21T08:55:11.072 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.071+0000 7f3017fff640 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4c2ac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f301806dac0 tx=0x7f300c00bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffdb20ca97233443 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:11.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f2ff7fff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:55:11.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f2ff7fff640 1 -- 172.21.15.57:0/1215953503 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c011440 con 0x561f1e4acac0 2024-04-21T08:55:11.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3010102d50 con 0x561f1e4acac0 2024-04-21T08:55:11.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f2ff7fff640 1 -- 172.21.15.57:0/1215953503 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f300c0115e0 con 0x561f1e4acac0 2024-04-21T08:55:11.081 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f2ff7fff640 1 -- 172.21.15.57:0/1215953503 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f300c024050 con 0x561f1e4acac0 2024-04-21T08:55:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f3021a76740 10 client.26479 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.081+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561f1e5366a0 con 0x561f1e4acac0 2024-04-21T08:55:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f2ff7fff640 1 -- 172.21.15.57:0/1215953503 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3010102d50 con 0x561f1e4acac0 2024-04-21T08:55:11.082 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f2ff7fff640 1 -- 172.21.15.57:0/1215953503 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f300c01b920 con 0x561f1e4acac0 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f3021a76740 10 client.26479 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f3021a76740 10 client.26479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f3021a76740 20 client.26479 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f3021a76740 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e536e80 0x561f1e5392a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f3021a76740 4 client.26479 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f1e3b03d0 con 0x561f1e536e80 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.082+0000 7f301cc1d640 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e536e80 0x561f1e5392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.083+0000 7f2ff7fff640 10 client.26479 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:11.083 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:11.083+0000 7f301cc1d640 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e536e80 0x561f1e5392a0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b80a546c7c47ffb server_cookie=2b6f42587d1d26ef in_seq=0 out_seq=0 2024-04-21T08:55:11.093 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/file.o 2024-04-21T08:55:11.236 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/isofs/compress.o 2024-04-21T08:55:11.651 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma/dummy.o 2024-04-21T08:55:11.663 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/jbd2/built-in.a 2024-04-21T08:55:12.049 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/nfs.o 2024-04-21T08:55:12.087 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/fsmap.o 2024-04-21T08:55:12.168 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/fsync.o 2024-04-21T08:55:12.377 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/exportfs/expfs.o 2024-04-21T08:55:12.412 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/isofs/built-in.a 2024-04-21T08:55:12.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:12 smithi129 ceph-mon[31272]: pgmap v305: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 1.25k op/s 2024-04-21T08:55:12.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:12 smithi057 ceph-mon[27968]: pgmap v305: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 1.25k op/s 2024-04-21T08:55:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:12 smithi136 ceph-mon[31791]: pgmap v305: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 1.25k op/s 2024-04-21T08:55:12.651 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma/swiotlb.o 2024-04-21T08:55:12.686 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/swap_slots.o 2024-04-21T08:55:12.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:12.751 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:12.758 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/getroot.o 2024-04-21T08:55:13.124 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/namei_vfat.o 2024-04-21T08:55:13.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.376+0000 7f630e5cf640 1 -- 172.21.15.57:0/252043320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630810dfa0 msgr2=0x7f6308115b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.376+0000 7f630e5cf640 1 --2- 172.21.15.57:0/252043320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630810dfa0 0x7f6308115b70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6304009860 tx=0x7f630402bb90 comp rx=0 tx=0).stop 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 -- 172.21.15.57:0/252043320 shutdown_connections 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 --2- 172.21.15.57:0/252043320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f630810dfa0 0x7f6308115b70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 --2- 172.21.15.57:0/252043320 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f630810a400 0x7f630810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 --2- 172.21.15.57:0/252043320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 0x7f6308109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 -- 172.21.15.57:0/252043320 >> 172.21.15.57:0/252043320 conn(0x7f630806ced0 msgr2=0x7f630806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 -- 172.21.15.57:0/252043320 shutdown_connections 2024-04-21T08:55:13.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.381+0000 7f630e5cf640 1 -- 172.21.15.57:0/252043320 wait complete. 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 Processor -- start 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 -- start start 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 0x7f630819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f630810dfa0 0x7f630819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63081a0760 0x7f630819f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6308118700 con 0x7f630810dfa0 2024-04-21T08:55:13.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6308118580 con 0x7f63081a0760 2024-04-21T08:55:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.382+0000 7f630e5cf640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6308118880 con 0x7f6308109a50 2024-04-21T08:55:13.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630d5cd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 0x7f630819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630d5cd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 0x7f630819eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35722/0 (socket says 172.21.15.57:35722) 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630d5cd640 1 -- 172.21.15.57:0/938947178 learned_addr learned my addr 172.21.15.57:0/938947178 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630ddce640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63081a0760 0x7f630819f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630ddce640 1 -- 172.21.15.57:0/938947178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 msgr2=0x7f630819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630ddce640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 0x7f630819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630ddce640 1 -- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f630810dfa0 msgr2=0x7f630819f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630ddce640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f630810dfa0 0x7f630819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.383+0000 7f630ddce640 1 -- 172.21.15.57:0/938947178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63081b3ba0 con 0x7f63081a0760 2024-04-21T08:55:13.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.384+0000 7f630ddce640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63081a0760 0x7f630819f540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f630402c1a0 tx=0x7f6304004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:13.389 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.389+0000 7f62fe7fc640 1 -- 172.21.15.57:0/938947178 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304039040 con 0x7f63081a0760 2024-04-21T08:55:13.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.392+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63081b3e30 con 0x7f63081a0760 2024-04-21T08:55:13.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.392+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63081b42f0 con 0x7f63081a0760 2024-04-21T08:55:13.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.392+0000 7f62fe7fc640 1 -- 172.21.15.57:0/938947178 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6304034e50 con 0x7f63081a0760 2024-04-21T08:55:13.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.392+0000 7f62fe7fc640 1 -- 172.21.15.57:0/938947178 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304003e00 con 0x7f63081a0760 2024-04-21T08:55:13.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.399+0000 7f62fe7fc640 1 -- 172.21.15.57:0/938947178 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6304044020 con 0x7f63081a0760 2024-04-21T08:55:13.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.400+0000 7f6312425740 1 Processor -- start 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.400+0000 7f6312425740 1 -- start start 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f6312425740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ed0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f6312425740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed1020 0x563a65ed96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f6312425740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65edbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f6312425740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x563a65e351e0 con 0x563a65ed9c00 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f6312425740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563a65dd2390 con 0x563a65ed0700 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f6312425740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x563a65e311a0 con 0x563a65ed1020 2024-04-21T08:55:13.401 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f62fe7fc640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f62f40a3ea0 0x7f62f40a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ed0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ed0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55288/0 (socket says 172.21.15.57:55288) 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f630cdcc640 1 -- 172.21.15.57:0/3737063961 learned_addr learned my addr 172.21.15.57:0/3737063961 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65edbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.401+0000 7f62fe7fc640 1 -- 172.21.15.57:0/938947178 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f63040f1640 con 0x7f63081a0760 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.402+0000 7f630cdcc640 1 -- 172.21.15.57:0/3737063961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed1020 msgr2=0x563a65ed96c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.402+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed1020 0x563a65ed96c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.402+0000 7f630cdcc640 1 -- 172.21.15.57:0/3737063961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 msgr2=0x563a65edbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.402+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65edbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.402+0000 7f630cdcc640 1 -- 172.21.15.57:0/3737063961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a65c117d0 con 0x563a65ed0700 2024-04-21T08:55:13.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.402+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ed0ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6300005c40 tx=0x7f630000aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d703821ef29ce700 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:13.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.407+0000 7f630ddce640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f62f40a3ea0 0x7f62f40a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.428+0000 7f62f37fe640 1 -- 172.21.15.57:0/3737063961 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000c110 con 0x563a65ed0700 2024-04-21T08:55:13.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.428+0000 7f62f37fe640 1 -- 172.21.15.57:0/3737063961 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f630000c2b0 con 0x563a65ed0700 2024-04-21T08:55:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.429+0000 7f6312425740 1 -- 172.21.15.57:0/3737063961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 msgr2=0x563a65ed0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.429+0000 7f6312425740 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ed0ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6300005c40 tx=0x7f630000aaf0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.430+0000 7f6312425740 1 -- 172.21.15.57:0/3737063961 shutdown_connections 2024-04-21T08:55:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.430+0000 7f6312425740 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65edbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.430+0000 7f6312425740 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed1020 0x563a65ed96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.430+0000 7f6312425740 1 --2- 172.21.15.57:0/3737063961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ed0ae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.430+0000 7f6312425740 1 -- 172.21.15.57:0/3737063961 >> 172.21.15.57:0/3737063961 conn(0x563a65ecae30 msgr2=0x563a65ecb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.433+0000 7f630ddce640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f62f40a3ea0 0x7f62f40a6360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f63040c5a20 tx=0x7f63040348e0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:13.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.453+0000 7f6312425740 1 -- 172.21.15.57:0/3737063961 shutdown_connections 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.453+0000 7f6312425740 1 -- 172.21.15.57:0/3737063961 wait complete. 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.453+0000 7f6312425740 1 -- start start 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.453+0000 7f6312425740 1 Processor -- start 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ed6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed9c00 0x563a65ed75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x563a65eeda50 con 0x563a65ed1020 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563a65dee2f0 con 0x563a65ed9c00 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x563a65eedbd0 con 0x563a65ed0700 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f6312425740 10 client.0 _ll_register_callbacks cb 0x563a65dfb580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ed6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ed6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35734/0 (socket says 172.21.15.57:35734) 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f630cdcc640 1 -- 172.21.15.57:0/3471563982 learned_addr learned my addr 172.21.15.57:0/3471563982 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:13.455 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.454+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed9c00 0x563a65ed75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.455+0000 7f630cdcc640 1 -- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed9c00 msgr2=0x563a65ed75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.455+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed9c00 0x563a65ed75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.455+0000 7f630cdcc640 1 -- 172.21.15.57:0/3471563982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 msgr2=0x563a65ed7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.455+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.455+0000 7f630cdcc640 1 -- 172.21.15.57:0/3471563982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a65b473f0 con 0x563a65ed0700 2024-04-21T08:55:13.456 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.455+0000 7f630cdcc640 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ed6b40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f630000ad20 tx=0x7f630000c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d0a86e94814db70 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:13.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.460+0000 7f62f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:55:13.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.460+0000 7f62f0ff9640 1 -- 172.21.15.57:0/3471563982 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300025040 con 0x563a65ed0700 2024-04-21T08:55:13.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.460+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f630000c2b0 con 0x563a65ed0700 2024-04-21T08:55:13.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.460+0000 7f62f0ff9640 1 -- 172.21.15.57:0/3471563982 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f6300003b70 con 0x563a65ed0700 2024-04-21T08:55:13.460 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.460+0000 7f62f0ff9640 1 -- 172.21.15.57:0/3471563982 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6300032050 con 0x563a65ed0700 2024-04-21T08:55:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f6312425740 10 client.26485 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:13.468 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563a65b070f0 con 0x563a65ed0700 2024-04-21T08:55:13.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f62f0ff9640 1 -- 172.21.15.57:0/3471563982 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f630000c2b0 con 0x563a65ed0700 2024-04-21T08:55:13.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f62f0ff9640 1 -- 172.21.15.57:0/3471563982 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6300005810 con 0x563a65ed0700 2024-04-21T08:55:13.469 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f6312425740 10 client.26485 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f6312425740 10 client.26485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.469+0000 7f6312425740 20 client.26485 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.470+0000 7f6312425740 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65efb020 0x563a65efd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.470+0000 7f6312425740 4 client.26485 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.470+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a65e42c00 con 0x563a65efb020 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.470+0000 7f630d5cd640 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65efb020 0x563a65efd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.470 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.470+0000 7f62f0ff9640 10 client.26485 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:13.471 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.471+0000 7f630d5cd640 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65efb020 0x563a65efd480 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c39efe2ae829a5a server_cookie=f0afa55174624714 in_seq=0 out_seq=0 2024-04-21T08:55:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.567+0000 7f2ff7fff640 1 -- 172.21.15.57:0/1215953503 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f1e3b03d0 con 0x561f1e536e80 2024-04-21T08:55:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.567+0000 7f2ff7fff640 10 client.26479 handle_command_reply: tid=0 2024-04-21T08:55:13.567 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.567+0000 7f3021a76740 1 client.26479 shutdown 2024-04-21T08:55:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f62f0ff9640 1 -- 172.21.15.57:0/3471563982 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a65e42c00 con 0x563a65efb020 2024-04-21T08:55:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f62f0ff9640 10 client.26485 handle_command_reply: tid=0 2024-04-21T08:55:13.568 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 client.26485 shutdown 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 msgr2=0x561f1e4c2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4c2ac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f301806dac0 tx=0x7f300c00bf90 comp rx=0 tx=0).stop 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 shutdown_connections 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561f1e536e80 0x561f1e5392a0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561f1e4b2d20 0x561f1e4c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561f1e4acac0 0x561f1e4c2ac0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f2ff7fff640 0 client.26479 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 --2- 172.21.15.57:0/1215953503 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561f1e4abd50 0x561f1e4c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 >> 172.21.15.57:0/1215953503 conn(0x561f1e472c90 msgr2=0x561f1e4a5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 msgr2=0x563a65ed6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ed6b40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f630000ad20 tx=0x7f630000c460 comp rx=0 tx=0).stop 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 shutdown_connections 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65efb020 0x563a65efd480 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed9c00 0x563a65ed75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f62f0ff9640 0 client.26485 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 --2- 172.21.15.57:0/3471563982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ed6b40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 >> 172.21.15.57:0/3471563982 conn(0x563a65ecae30 msgr2=0x563a65eccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 shutdown_connections 2024-04-21T08:55:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.568+0000 7f3021a76740 1 -- 172.21.15.57:0/1215953503 wait complete. 2024-04-21T08:55:13.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.569+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 shutdown_connections 2024-04-21T08:55:13.571 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.569+0000 7f3021a76740 20 client.26479 trim_cache size 0 max 16384 2024-04-21T08:55:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.569+0000 7f6312425740 1 -- 172.21.15.57:0/3471563982 wait complete. 2024-04-21T08:55:13.572 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.569+0000 7f6312425740 20 client.26485 trim_cache size 0 max 16384 2024-04-21T08:55:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:55:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:55:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:55:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30000a3f20 msgr2=0x7f30000a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30000a3f20 0x7f30000a63e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3008004420 tx=0x7f3008011680 comp rx=0 tx=0).stop 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 msgr2=0x7f3018084b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f3018084b90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f301000d090 tx=0x7f3010034900 comp rx=0 tx=0).stop 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 shutdown_connections 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f30000a3f20 0x7f30000a63e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f301807f190 0x7f301807f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f301810e330 0x7f301807ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 --2- 172.21.15.57:0/326305706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f301810d730 0x7f3018084b90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.584 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 >> 172.21.15.57:0/326305706 conn(0x7f301806cf00 msgr2=0x7f301810b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 shutdown_connections 2024-04-21T08:55:13.585 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:13.584+0000 7f301ee81640 1 -- 172.21.15.57:0/326305706 wait complete. 2024-04-21T08:55:13.617 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/exportfs/built-in.a 2024-04-21T08:55:13.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.625+0000 7f6312425740 1 Processor -- start 2024-04-21T08:55:13.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 -- start start 2024-04-21T08:55:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ef69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed9c00 0x563a65f02d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec00e190 con 0x563a65ed1020 2024-04-21T08:55:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0109f0 con 0x563a65ed0700 2024-04-21T08:55:13.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f6312425740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec00e7e0 con 0x563a65ed9c00 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f630d5cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f630d5cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47296/0 (socket says 172.21.15.57:47296) 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ef69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f630d5cd640 1 -- 172.21.15.57:0/1991987925 learned_addr learned my addr 172.21.15.57:0/1991987925 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.628+0000 7f630cdcc640 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed9c00 0x563a65f02d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630d5cd640 1 -- 172.21.15.57:0/1991987925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed9c00 msgr2=0x563a65f02d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630d5cd640 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed9c00 0x563a65f02d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630d5cd640 1 -- 172.21.15.57:0/1991987925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 msgr2=0x563a65ef69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630d5cd640 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ef69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630d5cd640 1 -- 172.21.15.57:0/1991987925 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a65efdb00 con 0x563a65ed1020 2024-04-21T08:55:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630cdcc640 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ef69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:13.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.629+0000 7f630d5cd640 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed73e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f62f80099f0 tx=0x7f62f8015780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a57688f57fbdee server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:13.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.634+0000 7f62f2ffd640 1 -- 172.21.15.57:0/1991987925 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8015de0 con 0x563a65ed1020 2024-04-21T08:55:13.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.635+0000 7f62f2ffd640 1 -- 172.21.15.57:0/1991987925 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62f801d070 con 0x563a65ed1020 2024-04-21T08:55:13.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.636+0000 7f6312425740 1 -- 172.21.15.57:0/1991987925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 msgr2=0x563a65ed73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.636+0000 7f6312425740 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed73e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f62f80099f0 tx=0x7f62f8015780 comp rx=0 tx=0).stop 2024-04-21T08:55:13.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 -- 172.21.15.57:0/1991987925 shutdown_connections 2024-04-21T08:55:13.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed9c00 0x563a65f02d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed1020 0x563a65ed73e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 --2- 172.21.15.57:0/1991987925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed0700 0x563a65ef69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 -- 172.21.15.57:0/1991987925 >> 172.21.15.57:0/1991987925 conn(0x563a65ecae30 msgr2=0x563a65ecc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 -- 172.21.15.57:0/1991987925 shutdown_connections 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 -- 172.21.15.57:0/1991987925 wait complete. 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.645+0000 7f6312425740 1 -- start start 2024-04-21T08:55:13.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 Processor -- start 2024-04-21T08:55:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ef2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed1020 0x563a65ef3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65ed2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec00eeb0 con 0x563a65ed9c00 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec010320 con 0x563a65ed1020 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec00f580 con 0x563a65ed0700 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.646+0000 7f6312425740 10 client.0 _ll_register_callbacks cb 0x7f630000c5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ef2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630cdcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ef2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35764/0 (socket says 172.21.15.57:35764) 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630cdcc640 1 -- 172.21.15.57:0/2791226612 learned_addr learned my addr 172.21.15.57:0/2791226612 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630cdcc640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65ed2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed1020 0x563a65ef3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 -- 172.21.15.57:0/2791226612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 msgr2=0x563a65ef2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ef2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 -- 172.21.15.57:0/2791226612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 msgr2=0x563a65ed2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65ed2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 -- 172.21.15.57:0/2791226612 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a65ecac60 con 0x563a65ed1020 2024-04-21T08:55:13.648 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.647+0000 7f630d5cd640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed1020 0x563a65ef3440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x563a65ec6890 tx=0x7f62f8009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bb4894884707896 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:13.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.655+0000 7f62f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:13.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.655+0000 7f62f0ff9640 1 -- 172.21.15.57:0/2791226612 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8026020 con 0x563a65ed1020 2024-04-21T08:55:13.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.655+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62f801d070 con 0x563a65ed1020 2024-04-21T08:55:13.658 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.655+0000 7f62f0ff9640 1 -- 172.21.15.57:0/2791226612 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f62f8005970 con 0x563a65ed1020 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f62f0ff9640 1 -- 172.21.15.57:0/2791226612 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f62f8031050 con 0x563a65ed1020 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f6312425740 10 client.26618 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x563a65f60eb0 con 0x563a65ed1020 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f62f0ff9640 1 -- 172.21.15.57:0/2791226612 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62f801d070 con 0x563a65ed1020 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f62f0ff9640 1 -- 172.21.15.57:0/2791226612 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f62f8019160 con 0x563a65ed1020 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f6312425740 10 client.26618 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f6312425740 10 client.26618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.657+0000 7f6312425740 20 client.26618 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.658+0000 7f6312425740 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65f614d0 0x563a65f638f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.658+0000 7f6312425740 4 client.26618 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.658+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563a65e3ca70 con 0x563a65f614d0 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.658+0000 7f630cdcc640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65f614d0 0x563a65f638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.658+0000 7f62f0ff9640 10 client.26618 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.659+0000 7f630cdcc640 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65f614d0 0x563a65f638f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d88b411eda971e92 server_cookie=cf07b6dd4ad19961 in_seq=0 out_seq=0 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.659+0000 7f62f0ff9640 1 -- 172.21.15.57:0/2791226612 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563a65e3ca70 con 0x563a65f614d0 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.659+0000 7f62f0ff9640 10 client.26618 handle_command_reply: tid=0 2024-04-21T08:55:13.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.659+0000 7f6312425740 1 client.26618 shutdown 2024-04-21T08:55:13.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.673+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed1020 msgr2=0x563a65ef3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.673+0000 7f6312425740 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed1020 0x563a65ef3440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x563a65ec6890 tx=0x7f62f8009fc0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 shutdown_connections 2024-04-21T08:55:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f6312425740 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x563a65f614d0 0x563a65f638f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f6312425740 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x563a65ed9c00 0x563a65ed2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f6312425740 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563a65ed1020 0x563a65ef3440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f6312425740 1 --2- 172.21.15.57:0/2791226612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x563a65ed0700 0x563a65ef2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f62f0ff9640 0 client.26618 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:13.685 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.685+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 >> 172.21.15.57:0/2791226612 conn(0x563a65ecae30 msgr2=0x563a65ecc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.691+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 shutdown_connections 2024-04-21T08:55:13.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.691+0000 7f6312425740 1 -- 172.21.15.57:0/2791226612 wait complete. 2024-04-21T08:55:13.695 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.695+0000 7f6312425740 20 client.26618 trim_cache size 0 max 16384 2024-04-21T08:55:13.699 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:55:13.699 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:55:13.699 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:55:13.700 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:55:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.703+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f62f40a3ea0 msgr2=0x7f62f40a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.703+0000 7f630e5cf640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f62f40a3ea0 0x7f62f40a6360 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f63040c5a20 tx=0x7f63040348e0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.703+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63081a0760 msgr2=0x7f630819f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:13.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.703+0000 7f630e5cf640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63081a0760 0x7f630819f540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f630402c1a0 tx=0x7f6304004270 comp rx=0 tx=0).stop 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 shutdown_connections 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f62f40a3ea0 0x7f62f40a6360 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63081a0760 0x7f630819f540 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f630810dfa0 0x7f630819f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 --2- 172.21.15.57:0/938947178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6308109a50 0x7f630819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 >> 172.21.15.57:0/938947178 conn(0x7f630806ced0 msgr2=0x7f6308071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 shutdown_connections 2024-04-21T08:55:13.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:13.704+0000 7f630e5cf640 1 -- 172.21.15.57:0/938947178 wait complete. 2024-04-21T08:55:13.732 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:55:13.732 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:55:13.851 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/clntlock.o 2024-04-21T08:55:13.906 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/dmapool.o 2024-04-21T08:55:13.931 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/clntproc.o 2024-04-21T08:55:14.283 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/hash.o 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 -- 172.21.15.57:0/1145252422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f010eb30 msgr2=0x7ff0f010ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 --2- 172.21.15.57:0/1145252422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f010eb30 0x7ff0f010ef10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009900 tx=0x7ff0e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 -- 172.21.15.57:0/1145252422 shutdown_connections 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 --2- 172.21.15.57:0/1145252422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f010df00 0x7ff0f0115ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 --2- 172.21.15.57:0/1145252422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f010f4e0 0x7ff0f010d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 --2- 172.21.15.57:0/1145252422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f010eb30 0x7ff0f010ef10 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009900 tx=0x7ff0e002bc30 comp rx=0 tx=0).stop 2024-04-21T08:55:14.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.342+0000 7ff0f5012640 1 -- 172.21.15.57:0/1145252422 >> 172.21.15.57:0/1145252422 conn(0x7ff0f006ce60 msgr2=0x7ff0f006d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.352+0000 7ff0f5012640 1 -- 172.21.15.57:0/1145252422 shutdown_connections 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.352+0000 7ff0f5012640 1 -- 172.21.15.57:0/1145252422 wait complete. 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.352+0000 7ff0f5012640 1 Processor -- start 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.352+0000 7ff0f5012640 1 -- start start 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f5012640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f010df00 0x7ff0f0114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f5012640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f01153c0 0x7ff0f0115820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f5012640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 0x7ff0f011ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f5012640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f0117ff0 con 0x7ff0f01153c0 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f5012640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f0117e70 con 0x7ff0f011eac0 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f5012640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f0118170 con 0x7ff0f010df00 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f4811640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 0x7ff0f011ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f4811640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 0x7ff0f011ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55326/0 (socket says 172.21.15.57:55326) 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.353+0000 7ff0f4811640 1 -- 172.21.15.57:0/4264628363 learned_addr learned my addr 172.21.15.57:0/4264628363 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0effff640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f010df00 0x7ff0f0114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0f4811640 1 -- 172.21.15.57:0/4264628363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f010df00 msgr2=0x7ff0f0114e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0f4811640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f010df00 0x7ff0f0114e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0f4811640 1 -- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f01153c0 msgr2=0x7ff0f0115820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0f4811640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f01153c0 0x7ff0f0115820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:14.355 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0f4811640 1 -- 172.21.15.57:0/4264628363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f011f830 con 0x7ff0f011eac0 2024-04-21T08:55:14.356 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.354+0000 7ff0f4811640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 0x7ff0f011ef70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009fd0 tx=0x7ff0e400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.366 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/inode.o 2024-04-21T08:55:14.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.376+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4011830 con 0x7ff0f011eac0 2024-04-21T08:55:14.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.377+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0e4016070 con 0x7ff0f011eac0 2024-04-21T08:55:14.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.377+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4010e50 con 0x7ff0f011eac0 2024-04-21T08:55:14.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.377+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f0126f50 con 0x7ff0f011eac0 2024-04-21T08:55:14.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.377+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f0127470 con 0x7ff0f011eac0 2024-04-21T08:55:14.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.383+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff0e40119d0 con 0x7ff0f011eac0 2024-04-21T08:55:14.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.384+0000 7ff0ed7fa640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0d80a40d0 0x7ff0d80a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:14.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.384+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7ff0e40d01d0 con 0x7ff0f011eac0 2024-04-21T08:55:14.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.385+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f006e100 con 0x7ff0f011eac0 2024-04-21T08:55:14.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.391+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0e409c510 con 0x7ff0f011eac0 2024-04-21T08:55:14.414 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.409+0000 7ff0effff640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0d80a40d0 0x7ff0d80a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:14.467 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:14.467+0000 7ff0effff640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0d80a40d0 0x7ff0d80a6590 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff0e002c1c0 tx=0x7ff0e0005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:14.565 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/ialloc.o 2024-04-21T08:55:14.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:14 smithi129 ceph-mon[31272]: pgmap v306: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 1.04k op/s 2024-04-21T08:55:14.600 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:14 smithi057 ceph-mon[27968]: pgmap v306: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 1.04k op/s 2024-04-21T08:55:14.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:14 smithi136 ceph-mon[31791]: pgmap v306: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 1.04k op/s 2024-04-21T08:55:14.687 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fat/namei_msdos.o 2024-04-21T08:55:14.779 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma/remap.o 2024-04-21T08:55:15.140 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/hugetlb.o 2024-04-21T08:55:15.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.386+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0f011b4b0 con 0x7ff0f011eac0 2024-04-21T08:55:15.390 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.391+0000 7ff0ed7fa640 1 -- 172.21.15.57:0/4264628363 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7ff0f011b4b0 con 0x7ff0f011eac0 2024-04-21T08:55:15.393 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:55:15.393 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:55:15.394 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0d80a40d0 msgr2=0x7ff0d80a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0d80a40d0 0x7ff0d80a6590 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff0e002c1c0 tx=0x7ff0e0005fd0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 msgr2=0x7ff0f011ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 0x7ff0f011ef70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009fd0 tx=0x7ff0e400b8e0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 shutdown_connections 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff0d80a40d0 0x7ff0d80a6590 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0f011eac0 0x7ff0f011ef70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0f01153c0 0x7ff0f0115820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 --2- 172.21.15.57:0/4264628363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f010df00 0x7ff0f0114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:15.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.412+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 >> 172.21.15.57:0/4264628363 conn(0x7ff0f006ce60 msgr2=0x7ff0f0070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:15.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.422+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 shutdown_connections 2024-04-21T08:55:15.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:15.425+0000 7ff0f5012640 1 -- 172.21.15.57:0/4264628363 wait complete. 2024-04-21T08:55:15.454 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:55:15.644 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/indirect.o 2024-04-21T08:55:15.780 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/clntxdr.o 2024-04-21T08:55:15.928 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/dma/built-in.a 2024-04-21T08:55:16.053 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.053+0000 7f110c214640 1 -- 172.21.15.57:0/1675634482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 msgr2=0x7f110410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.054 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.053+0000 7f110c214640 1 --2- 172.21.15.57:0/1675634482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 0x7f110410ec50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f10f401a910 tx=0x7f10f403cc90 comp rx=0 tx=0).stop 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.059+0000 7f110c214640 1 -- 172.21.15.57:0/1675634482 shutdown_connections 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.059+0000 7f110c214640 1 --2- 172.21.15.57:0/1675634482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f110411b730 0x7f110411db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.059+0000 7f110c214640 1 --2- 172.21.15.57:0/1675634482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 0x7f110410ec50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.059+0000 7f110c214640 1 --2- 172.21.15.57:0/1675634482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11041107a0 0x7f110410e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.059+0000 7f110c214640 1 -- 172.21.15.57:0/1675634482 >> 172.21.15.57:0/1675634482 conn(0x7f110406d060 msgr2=0x7f110406d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.061+0000 7f110c214640 1 -- 172.21.15.57:0/1675634482 shutdown_connections 2024-04-21T08:55:16.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.061+0000 7f110c214640 1 -- 172.21.15.57:0/1675634482 wait complete. 2024-04-21T08:55:16.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.062+0000 7f110c214640 1 Processor -- start 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 -- start start 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 0x7f110411aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 0x7f1104116f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110411b730 0x7f1104117480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1104113860 con 0x7f110411b730 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f11041136e0 con 0x7f11041107a0 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f110c214640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f11041139e0 con 0x7f110410e7d0 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f11097af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 0x7f1104116f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f11097af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 0x7f1104116f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55336/0 (socket says 172.21.15.57:55336) 2024-04-21T08:55:16.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f11097af640 1 -- 172.21.15.57:0/2928051759 learned_addr learned my addr 172.21.15.57:0/2928051759 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:16.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.065+0000 7f1109fb0640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 0x7f110411aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f11097af640 1 -- 172.21.15.57:0/2928051759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 msgr2=0x7f110411aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f11097af640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 0x7f110411aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f11097af640 1 -- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110411b730 msgr2=0x7f1104117480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f11097af640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110411b730 0x7f1104117480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f11097af640 1 -- 172.21.15.57:0/2928051759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1104117bb0 con 0x7f11041107a0 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f11097af640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 0x7f1104116f40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f10f403d2a0 tx=0x7f10f40045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f10faffd640 1 -- 172.21.15.57:0/2928051759 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f403db50 con 0x7f11041107a0 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f10faffd640 1 -- 172.21.15.57:0/2928051759 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f4053070 con 0x7f11041107a0 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.066+0000 7f10faffd640 1 -- 172.21.15.57:0/2928051759 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f40039f0 con 0x7f11041107a0 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.069+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11041ab4c0 con 0x7f11041107a0 2024-04-21T08:55:16.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.069+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11041ab870 con 0x7f11041107a0 2024-04-21T08:55:16.073 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.073+0000 7f110ee09740 1 Processor -- start 2024-04-21T08:55:16.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.075+0000 7f10faffd640 1 -- 172.21.15.57:0/2928051759 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f10f4003b90 con 0x7f11041107a0 2024-04-21T08:55:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.075+0000 7f110ee09740 1 -- start start 2024-04-21T08:55:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f110ee09740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa724504d0 0x55fa724508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f110ee09740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 0x55fa72459430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f110ee09740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72459a00 0x55fa7245be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f110ee09740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fa723b51f0 con 0x55fa724504d0 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f110ee09740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fa723524b0 con 0x55fa72450e80 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f110ee09740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fa723b11b0 con 0x55fa72459a00 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f1109fb0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 0x55fa72459430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f1109fb0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 0x55fa72459430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55346/0 (socket says 172.21.15.57:55346) 2024-04-21T08:55:16.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f1109fb0640 1 -- 172.21.15.57:0/264046343 learned_addr learned my addr 172.21.15.57:0/264046343 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:16.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f11097af640 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72459a00 0x55fa7245be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.076+0000 7f10faffd640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10f00a3de0 0x7f10f00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.081 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f110a7b1640 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa724504d0 0x55fa724508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f1109fb0640 1 -- 172.21.15.57:0/264046343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72459a00 msgr2=0x55fa7245be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f1109fb0640 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72459a00 0x55fa7245be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f1109fb0640 1 -- 172.21.15.57:0/264046343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa724504d0 msgr2=0x55fa724508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f1109fb0640 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa724504d0 0x55fa724508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f1109fb0640 1 -- 172.21.15.57:0/264046343 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa723441b0 con 0x55fa72450e80 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f10faffd640 1 -- 172.21.15.57:0/2928051759 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f10f4104360 con 0x7f11041107a0 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.077+0000 7f1109fb0640 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 0x55fa72459430 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1100009a70 tx=0x7f1100009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f2707093a1de62 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.080+0000 7f110a7b1640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10f00a3de0 0x7f10f00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.082+0000 7f10effff640 1 -- 172.21.15.57:0/264046343 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100017070 con 0x55fa72450e80 2024-04-21T08:55:16.082 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.082+0000 7f10effff640 1 -- 172.21.15.57:0/264046343 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100013470 con 0x55fa72450e80 2024-04-21T08:55:16.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.083+0000 7f110ee09740 1 -- 172.21.15.57:0/264046343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 msgr2=0x55fa72459430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.083 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.083+0000 7f110ee09740 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 0x55fa72459430 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1100009a70 tx=0x7f1100009f30 comp rx=0 tx=0).stop 2024-04-21T08:55:16.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.084+0000 7f110ee09740 1 -- 172.21.15.57:0/264046343 shutdown_connections 2024-04-21T08:55:16.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.084+0000 7f110ee09740 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72459a00 0x55fa7245be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.084+0000 7f110ee09740 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72450e80 0x55fa72459430 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1100009a70 tx=0x7f1100009f30 comp rx=0 tx=0).stop 2024-04-21T08:55:16.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.084+0000 7f110ee09740 1 --2- 172.21.15.57:0/264046343 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa724504d0 0x55fa724508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.084+0000 7f110ee09740 1 -- 172.21.15.57:0/264046343 >> 172.21.15.57:0/264046343 conn(0x55fa7244aba0 msgr2=0x55fa7244af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:16.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.094+0000 7f110ee09740 1 -- 172.21.15.57:0/264046343 shutdown_connections 2024-04-21T08:55:16.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.094+0000 7f110ee09740 1 -- 172.21.15.57:0/264046343 wait complete. 2024-04-21T08:55:16.094 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.094+0000 7f110ee09740 1 -- start start 2024-04-21T08:55:16.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.095+0000 7f110ee09740 1 Processor -- start 2024-04-21T08:55:16.095 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.095+0000 7f110a7b1640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10f00a3de0 0x7f10f00a62a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00bce0 tx=0x7f10fc004af0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:16.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa72459350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72456640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72457030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55fa7245c9c0 con 0x55fa72459a00 2024-04-21T08:55:16.100 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fa7236e500 con 0x55fa72456bd0 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55fa7245cb40 con 0x55fa724504d0 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.100+0000 7f110ee09740 10 client.0 _ll_register_callbacks cb 0x55fa7233cb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f110a7b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72457030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa72459350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f110a7b1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72457030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55360/0 (socket says 172.21.15.57:55360) 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f110a7b1640 1 -- 172.21.15.57:0/1118585654 learned_addr learned my addr 172.21.15.57:0/1118585654 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:16.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 -- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 msgr2=0x55fa72457030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72457030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 -- 172.21.15.57:0/1118585654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 msgr2=0x55fa72456640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72456640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 -- 172.21.15.57:0/1118585654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa720c73f0 con 0x55fa724504d0 2024-04-21T08:55:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f110a7b1640 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72457030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:16.102 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.101+0000 7f1109fb0640 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa72459350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55fa7244bc10 tx=0x7f110000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=495a711afc06d4a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:16.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.104+0000 7f10ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:55:16.104 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.104+0000 7f10ed7fa640 1 -- 172.21.15.57:0/1118585654 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100017050 con 0x55fa724504d0 2024-04-21T08:55:16.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.104+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1100013470 con 0x55fa724504d0 2024-04-21T08:55:16.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.104+0000 7f10ed7fa640 1 -- 172.21.15.57:0/1118585654 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f1100012870 con 0x55fa724504d0 2024-04-21T08:55:16.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.111+0000 7f10ed7fa640 1 -- 172.21.15.57:0/1118585654 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f110002f050 con 0x55fa724504d0 2024-04-21T08:55:16.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.114+0000 7f110ee09740 10 client.26506 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:16.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.114+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fa721917d0 con 0x55fa724504d0 2024-04-21T08:55:16.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f10ed7fa640 1 -- 172.21.15.57:0/1118585654 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100013470 con 0x55fa724504d0 2024-04-21T08:55:16.120 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f10ed7fa640 1 -- 172.21.15.57:0/1118585654 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f110001c8c0 con 0x55fa724504d0 2024-04-21T08:55:16.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f110ee09740 10 client.26506 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:16.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f110ee09740 10 client.26506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:16.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f110ee09740 20 client.26506 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:16.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f110ee09740 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7246a370 0x55fa7246c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f110ee09740 4 client.26506 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:16.121 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.121+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa72354930 con 0x55fa7246a370 2024-04-21T08:55:16.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.124+0000 7f11097af640 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7246a370 0x55fa7246c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.138+0000 7f11097af640 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7246a370 0x55fa7246c800 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b1dd784bbe9c953 server_cookie=34c291c0d0d49e35 in_seq=0 out_seq=0 2024-04-21T08:55:16.138 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.138+0000 7f10ed7fa640 10 client.26506 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:16.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.142+0000 7f10ed7fa640 1 -- 172.21.15.57:0/1118585654 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fa72354930 con 0x55fa7246a370 2024-04-21T08:55:16.141 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.142+0000 7f10ed7fa640 10 client.26506 handle_command_reply: tid=0 2024-04-21T08:55:16.142 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.142+0000 7f110ee09740 1 client.26506 shutdown 2024-04-21T08:55:16.144 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 msgr2=0x55fa72459350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa72459350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55fa7244bc10 tx=0x7f110000ed70 comp rx=0 tx=0).stop 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 shutdown_connections 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa7246a370 0x55fa7246c800 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72457030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72456640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 --2- 172.21.15.57:0/1118585654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa72459350 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.145 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.145+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 >> 172.21.15.57:0/1118585654 conn(0x55fa7244aba0 msgr2=0x55fa7244c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:16.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.149+0000 7f10ed7fa640 0 client.26506 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:16.148 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.149+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 shutdown_connections 2024-04-21T08:55:16.156 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.157+0000 7f110ee09740 1 -- 172.21.15.57:0/1118585654 wait complete. 2024-04-21T08:55:16.157 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.157+0000 7f110ee09740 20 client.26506 trim_cache size 0 max 16384 2024-04-21T08:55:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.193+0000 7f110ee09740 1 Processor -- start 2024-04-21T08:55:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.194+0000 7f110ee09740 1 -- start start 2024-04-21T08:55:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.194+0000 7f110ee09740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa724565e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.194+0000 7f110ee09740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72465960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.194+0000 7f110ee09740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72458e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.195+0000 7f1109fb0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa724565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.195+0000 7f1109fb0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa724565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35830/0 (socket says 172.21.15.57:35830) 2024-04-21T08:55:16.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.195+0000 7f11097af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72465960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.195+0000 7f11097af640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72465960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55362/0 (socket says 172.21.15.57:55362) 2024-04-21T08:55:16.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.198+0000 7f110a7b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72458e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f110ee09740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10e800e190 con 0x55fa72459a00 2024-04-21T08:55:16.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f110a7b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72458e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47384/0 (socket says 172.21.15.57:47384) 2024-04-21T08:55:16.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f110a7b1640 1 -- 172.21.15.57:0/486611200 learned_addr learned my addr 172.21.15.57:0/486611200 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:16.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80109f0 con 0x55fa72456bd0 2024-04-21T08:55:16.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f10e800e7e0 con 0x55fa724504d0 2024-04-21T08:55:16.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f1109fb0640 1 -- 172.21.15.57:0/486611200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 msgr2=0x55fa72465960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f1109fb0640 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72465960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f1109fb0640 1 -- 172.21.15.57:0/486611200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 msgr2=0x55fa72458e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f1109fb0640 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72458e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f1109fb0640 1 -- 172.21.15.57:0/486611200 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa7244f7f0 con 0x55fa724504d0 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.199+0000 7f1109fb0640 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa724565e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f110001ccc0 tx=0x7f11000098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b02f1f410d8737 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.200+0000 7f10ef7fe640 1 -- 172.21.15.57:0/486611200 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100017040 con 0x55fa724504d0 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.200+0000 7f10ef7fe640 1 -- 172.21.15.57:0/486611200 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100013470 con 0x55fa724504d0 2024-04-21T08:55:16.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.200+0000 7f10ef7fe640 1 -- 172.21.15.57:0/486611200 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100026760 con 0x55fa724504d0 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 msgr2=0x55fa724565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa724565e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f110001ccc0 tx=0x7f11000098f0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 shutdown_connections 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa72458e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa72456bd0 0x55fa72465960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 --2- 172.21.15.57:0/486611200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa724504d0 0x55fa724565e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 >> 172.21.15.57:0/486611200 conn(0x55fa7244aba0 msgr2=0x55fa72455ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:16.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 shutdown_connections 2024-04-21T08:55:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.201+0000 7f110ee09740 1 -- 172.21.15.57:0/486611200 wait complete. 2024-04-21T08:55:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 -- start start 2024-04-21T08:55:16.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 Processor -- start 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa724504d0 0x55fa72462170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72456bd0 0x55fa7245ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa7245d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10e800eeb0 con 0x55fa72459a00 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10e8010320 con 0x55fa724504d0 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.202+0000 7f110ee09740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f10e800f580 con 0x55fa72456bd0 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.203+0000 7f110ee09740 10 client.0 _ll_register_callbacks cb 0x7f1100004640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.203+0000 7f11097af640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72456bd0 0x55fa7245ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.203+0000 7f11097af640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72456bd0 0x55fa7245ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35846/0 (socket says 172.21.15.57:35846) 2024-04-21T08:55:16.203 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.203+0000 7f11097af640 1 -- 172.21.15.57:0/3201996755 learned_addr learned my addr 172.21.15.57:0/3201996755 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.207+0000 7f110a7b1640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa7245d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa724504d0 0x55fa72462170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 -- 172.21.15.57:0/3201996755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72456bd0 msgr2=0x55fa7245ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72456bd0 0x55fa7245ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 -- 172.21.15.57:0/3201996755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 msgr2=0x55fa7245d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa7245d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 -- 172.21.15.57:0/3201996755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa7245dae0 con 0x55fa724504d0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.208+0000 7f1109fb0640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa724504d0 0x55fa72462170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11000094e0 tx=0x7f110001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed66f41c4eca904 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.209+0000 7f10ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.210+0000 7f10ed7fa640 1 -- 172.21.15.57:0/3201996755 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100017040 con 0x55fa724504d0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.210+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1100013470 con 0x55fa724504d0 2024-04-21T08:55:16.213 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.211+0000 7f10ed7fa640 1 -- 172.21.15.57:0/3201996755 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f1100005bc0 con 0x55fa724504d0 2024-04-21T08:55:16.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.211+0000 7f10ed7fa640 1 -- 172.21.15.57:0/3201996755 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1100042050 con 0x55fa724504d0 2024-04-21T08:55:16.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.214+0000 7f110ee09740 10 client.26639 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:16.214 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.214+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55fa724cf8c0 con 0x55fa724504d0 2024-04-21T08:55:16.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.216+0000 7f10ed7fa640 1 -- 172.21.15.57:0/3201996755 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100013470 con 0x55fa724504d0 2024-04-21T08:55:16.215 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/entry/common.o 2024-04-21T08:55:16.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f10ed7fa640 1 -- 172.21.15.57:0/3201996755 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f1100013830 con 0x55fa724504d0 2024-04-21T08:55:16.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f110ee09740 10 client.26639 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:16.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f110ee09740 10 client.26639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:16.222 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f110ee09740 20 client.26639 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:16.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f110ee09740 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa724cfe80 0x55fa724d2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:16.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f110ee09740 4 client.26639 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:55:16.223 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.221+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa723c2c10 con 0x55fa724cfe80 2024-04-21T08:55:16.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.232+0000 7f11097af640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa724cfe80 0x55fa724d2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:16.240 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.240+0000 7f10ed7fa640 10 client.26639 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:16.240 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.240+0000 7f11097af640 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa724cfe80 0x55fa724d2270 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d6361ceb9c8472b server_cookie=ab3d8e0d7f7eb53e in_seq=0 out_seq=0 2024-04-21T08:55:16.240 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.240+0000 7f10ed7fa640 1 -- 172.21.15.57:0/3201996755 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa723c2c10 con 0x55fa724cfe80 2024-04-21T08:55:16.240 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.240+0000 7f10ed7fa640 10 client.26639 handle_command_reply: tid=0 2024-04-21T08:55:16.240 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.240+0000 7f110ee09740 1 client.26639 shutdown 2024-04-21T08:55:16.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa724504d0 msgr2=0x55fa72462170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa724504d0 0x55fa72462170 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11000094e0 tx=0x7f110001c920 comp rx=0 tx=0).stop 2024-04-21T08:55:16.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 shutdown_connections 2024-04-21T08:55:16.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55fa724cfe80 0x55fa724d2270 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55fa72459a00 0x55fa7245d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55fa72456bd0 0x55fa7245ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 --2- 172.21.15.57:0/3201996755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fa724504d0 0x55fa72462170 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f10ed7fa640 0 client.26639 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.241+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 >> 172.21.15.57:0/3201996755 conn(0x55fa7244aba0 msgr2=0x55fa72449370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.242+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 shutdown_connections 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.242+0000 7f110ee09740 1 -- 172.21.15.57:0/3201996755 wait complete. 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.242+0000 7f110ee09740 20 client.26639 trim_cache size 0 max 16384 2024-04-21T08:55:16.242 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:55:16.261 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10f00a3de0 msgr2=0x7f10f00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10f00a3de0 0x7f10f00a62a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f10fc00bce0 tx=0x7f10fc004af0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 msgr2=0x7f1104116f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 0x7f1104116f40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f10f403d2a0 tx=0x7f10f40045a0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 shutdown_connections 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f10f00a3de0 0x7f10f00a62a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f110411b730 0x7f1104117480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11041107a0 0x7f1104116f40 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f10f403d2a0 tx=0x7f10f40045a0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 --2- 172.21.15.57:0/2928051759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f110410e7d0 0x7f110411aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:16.262 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.262+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 >> 172.21.15.57:0/2928051759 conn(0x7f110406d060 msgr2=0x7f110411c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.266+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 shutdown_connections 2024-04-21T08:55:16.265 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:16.266+0000 7f110c214640 1 -- 172.21.15.57:0/2928051759 wait complete. 2024-04-21T08:55:16.297 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/entry/syscall_user_dispatch.o 2024-04-21T08:55:16.321 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/fat/built-in.a 2024-04-21T08:55:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:16 smithi057 ceph-mon[27968]: pgmap v307: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.6 MiB/s wr, 465 op/s 2024-04-21T08:55:16.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:16 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4264628363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:16.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:16 smithi136 ceph-mon[31791]: pgmap v307: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.6 MiB/s wr, 465 op/s 2024-04-21T08:55:16.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:16 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4264628363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:16 smithi129 ceph-mon[31272]: pgmap v307: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.6 MiB/s wr, 465 op/s 2024-04-21T08:55:16.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:16 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4264628363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:55:16.829 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nls/nls_base.o 2024-04-21T08:55:17.083 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nls/nls_cp437.o 2024-04-21T08:55:17.294 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-21T08:55:17.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:55:17.295 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-21T08:55:17.541 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/host.o 2024-04-21T08:55:17.567 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/inline.o 2024-04-21T08:55:17.638 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nls/nls_ascii.o 2024-04-21T08:55:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.754+0000 7f4e8267e640 1 -- 172.21.15.57:0/2121810267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c10ecb0 msgr2=0x7f4e7c10bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.756 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.754+0000 7f4e8267e640 1 --2- 172.21.15.57:0/2121810267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c10ecb0 0x7f4e7c10bf80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009be0 tx=0x7f4e7402bfa0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.758 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.759+0000 7f4e8267e640 1 -- 172.21.15.57:0/2121810267 shutdown_connections 2024-04-21T08:55:17.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.759+0000 7f4e8267e640 1 --2- 172.21.15.57:0/2121810267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e7c10c690 0x7f4e7c07a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.759+0000 7f4e8267e640 1 --2- 172.21.15.57:0/2121810267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c10ecb0 0x7f4e7c10bf80 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009be0 tx=0x7f4e7402bfa0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.759+0000 7f4e8267e640 1 --2- 172.21.15.57:0/2121810267 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e7c10e390 0x7f4e7c10e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.759 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.759+0000 7f4e8267e640 1 -- 172.21.15.57:0/2121810267 >> 172.21.15.57:0/2121810267 conn(0x7f4e7c06ced0 msgr2=0x7f4e7c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:17.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.764+0000 7f4e8267e640 1 -- 172.21.15.57:0/2121810267 shutdown_connections 2024-04-21T08:55:17.766 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.766+0000 7f4e8267e640 1 -- 172.21.15.57:0/2121810267 wait complete. 2024-04-21T08:55:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.766+0000 7f4e8267e640 1 Processor -- start 2024-04-21T08:55:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 -- start start 2024-04-21T08:55:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e7c10c690 0x7f4e7c083b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e7c10e390 0x7f4e7c07dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 0x7f4e7c07e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.768 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c07c340 con 0x7f4e7c10e390 2024-04-21T08:55:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c07c1c0 con 0x7f4e7c07e150 2024-04-21T08:55:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e8267e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c07c4c0 con 0x7f4e7c10c690 2024-04-21T08:55:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e81e7d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 0x7f4e7c07e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e81e7d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 0x7f4e7c07e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55382/0 (socket says 172.21.15.57:55382) 2024-04-21T08:55:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1835647478 learned_addr learned my addr 172.21.15.57:0/1835647478 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:17.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.767+0000 7f4e80e7b640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e7c10e390 0x7f4e7c07dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.770 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.768+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1835647478 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e7c10c690 msgr2=0x7f4e7c083b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:55:17.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.768+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e7c10c690 0x7f4e7c083b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.768+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e7c10e390 msgr2=0x7f4e7c07dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.768+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e7c10e390 0x7f4e7c07dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.768+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1835647478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c1cd4f0 con 0x7f4e7c07e150 2024-04-21T08:55:17.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.772+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 0x7f4e7c07e600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800d750 tx=0x7f4e7800dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:17.777 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.778+0000 7f4e727fc640 1 -- 172.21.15.57:0/1835647478 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78017070 con 0x7f4e7c07e150 2024-04-21T08:55:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.778+0000 7f4e727fc640 1 -- 172.21.15.57:0/1835647478 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e780054d0 con 0x7f4e7c07e150 2024-04-21T08:55:17.778 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.778+0000 7f4e727fc640 1 -- 172.21.15.57:0/1835647478 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78004c30 con 0x7f4e7c07e150 2024-04-21T08:55:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.779+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7c1cd7e0 con 0x7f4e7c07e150 2024-04-21T08:55:17.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.779+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7c1cdcb0 con 0x7f4e7c07e150 2024-04-21T08:55:17.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 Processor -- start 2024-04-21T08:55:17.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 -- start start 2024-04-21T08:55:17.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 0x55ae71cc3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71ccc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71ccf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ae71c27f00 con 0x55ae71cccfb0 2024-04-21T08:55:17.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae71bc5590 con 0x55ae71cc3a80 2024-04-21T08:55:17.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.783+0000 7f4e865ab740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ae71c23ec0 con 0x55ae71cc4430 2024-04-21T08:55:17.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.785+0000 7f4e727fc640 1 -- 172.21.15.57:0/1835647478 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4e78010460 con 0x7f4e7c07e150 2024-04-21T08:55:17.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.785+0000 7f4e80e7b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 0x55ae71cc3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.785+0000 7f4e80e7b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 0x55ae71cc3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55384/0 (socket says 172.21.15.57:55384) 2024-04-21T08:55:17.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.785+0000 7f4e80e7b640 1 -- 172.21.15.57:0/3921011811 learned_addr learned my addr 172.21.15.57:0/3921011811 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:17.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e80e7b640 1 -- 172.21.15.57:0/3921011811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 msgr2=0x55ae71ccc9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:17.786 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e80e7b640 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71ccc9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e80e7b640 1 -- 172.21.15.57:0/3921011811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 msgr2=0x55ae71ccf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e80e7b640 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71ccf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e80e7b640 1 -- 172.21.15.57:0/3921011811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae71a027e0 con 0x55ae71cc3a80 2024-04-21T08:55:17.787 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e80e7b640 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 0x55ae71cc3e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4e74000f80 tx=0x7f4e740098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3862bde6d3f03b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:17.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.786+0000 7f4e677fe640 1 -- 172.21.15.57:0/3921011811 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7402da50 con 0x55ae71cc3a80 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.787+0000 7f4e677fe640 1 -- 172.21.15.57:0/3921011811 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e74003e20 con 0x55ae71cc3a80 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.787+0000 7f4e727fc640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4e680a3ed0 0x7f4e680a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.787+0000 7f4e727fc640 1 -- 172.21.15.57:0/1835647478 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f4e780d13f0 con 0x7f4e7c07e150 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.788+0000 7f4e865ab740 1 -- 172.21.15.57:0/3921011811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 msgr2=0x55ae71cc3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.788+0000 7f4e865ab740 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 0x55ae71cc3e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4e74000f80 tx=0x7f4e740098f0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 -- 172.21.15.57:0/3921011811 shutdown_connections 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71ccf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71ccc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 --2- 172.21.15.57:0/3921011811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc3a80 0x55ae71cc3e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 -- 172.21.15.57:0/3921011811 >> 172.21.15.57:0/3921011811 conn(0x55ae71cbe150 msgr2=0x55ae71cbe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4e680a3ed0 0x7f4e680a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 -- 172.21.15.57:0/3921011811 shutdown_connections 2024-04-21T08:55:17.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.789+0000 7f4e865ab740 1 -- 172.21.15.57:0/3921011811 wait complete. 2024-04-21T08:55:17.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.790+0000 7f4e865ab740 1 -- start start 2024-04-21T08:55:17.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.790+0000 7f4e865ab740 1 Processor -- start 2024-04-21T08:55:17.792 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/entry/built-in.a 2024-04-21T08:55:17.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.790+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cc9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.790+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71cca250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.790+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.790+0000 7f4e865ab740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ae71ccff10 con 0x55ae71cccfb0 2024-04-21T08:55:17.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.791+0000 7f4e865ab740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ae71be1580 con 0x55ae71cc4430 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.791+0000 7f4e865ab740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ae71cd0090 con 0x55ae71cca790 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.791+0000 7f4e81e7d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.791+0000 7f4e81e7d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35864/0 (socket says 172.21.15.57:35864) 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.791+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1104600669 learned_addr learned my addr 172.21.15.57:0/1104600669 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.791+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cc9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e865ab740 10 client.0 _ll_register_callbacks cb 0x55ae71be7650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 msgr2=0x55ae71cc9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cc9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1104600669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 msgr2=0x55ae71cca250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71cca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e81e7d640 1 -- 172.21.15.57:0/1104600669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae71a62f70 con 0x55ae71cca790 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd9460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e78004e40 tx=0x7f4e78017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7095fa2d4a615562 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:17.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.792+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4e680a3ed0 0x7f4e680a6390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55ae71cbf1c0 tx=0x7f4e6c005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:17.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.796+0000 7f4e64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:55:17.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.796+0000 7f4e64ff9640 1 -- 172.21.15.57:0/1104600669 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e78020040 con 0x55ae71cca790 2024-04-21T08:55:17.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.796+0000 7f4e64ff9640 1 -- 172.21.15.57:0/1104600669 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f4e780995f0 con 0x55ae71cca790 2024-04-21T08:55:17.796 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.797+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ae71c35920 con 0x55ae71cca790 2024-04-21T08:55:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.800+0000 7f4e64ff9640 1 -- 172.21.15.57:0/1104600669 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4e78009bd0 con 0x55ae71cca790 2024-04-21T08:55:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.802+0000 7f4e865ab740 10 client.26518 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:17.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.802+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55ae718fa170 con 0x55ae71cca790 2024-04-21T08:55:17.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.808+0000 7f4e64ff9640 1 -- 172.21.15.57:0/1104600669 <== mon.1 v2:172.21.15.136:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4e780b6200 con 0x55ae71cca790 2024-04-21T08:55:17.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.810+0000 7f4e865ab740 10 client.26518 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:17.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.810+0000 7f4e865ab740 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:17.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.810+0000 7f4e865ab740 20 client.26518 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:17.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.810+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71cdd720 0x55ae71cdfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.810+0000 7f4e865ab740 4 client.26518 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:17.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.810+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae71c2f790 con 0x55ae71cdd720 2024-04-21T08:55:17.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.813+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71cdd720 0x55ae71cdfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.814+0000 7f4e64ff9640 10 client.26518 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:17.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.814+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71cdd720 0x55ae71cdfb90 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aed874214e1e710 server_cookie=6287929edb392aa0 in_seq=0 out_seq=0 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.814+0000 7f4e64ff9640 1 -- 172.21.15.57:0/1104600669 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae71c2f790 con 0x55ae71cdd720 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.814+0000 7f4e64ff9640 10 client.26518 handle_command_reply: tid=0 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.814+0000 7f4e865ab740 1 client.26518 shutdown 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 msgr2=0x55ae71cd9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd9460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4e78004e40 tx=0x7f4e78017000 comp rx=0 tx=0).stop 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 shutdown_connections 2024-04-21T08:55:17.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71cdd720 0x55ae71cdfb90 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd9460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71cca250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e64ff9640 0 client.26518 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1104600669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cc9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 >> 172.21.15.57:0/1104600669 conn(0x55ae71cbe150 msgr2=0x55ae71cbe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 shutdown_connections 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 1 -- 172.21.15.57:0/1104600669 wait complete. 2024-04-21T08:55:17.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.815+0000 7f4e865ab740 20 client.26518 trim_cache size 0 max 16384 2024-04-21T08:55:17.847 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/inode.o 2024-04-21T08:55:17.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.876+0000 7f4e865ab740 1 Processor -- start 2024-04-21T08:55:17.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.881+0000 7f4e865ab740 1 -- start start 2024-04-21T08:55:17.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.881+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cbcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.881+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71ce54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cbcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cbcfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55412/0 (socket says 172.21.15.57:55412) 2024-04-21T08:55:17.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e80e7b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.882 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e80e7b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35868/0 (socket says 172.21.15.57:35868) 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 -- 172.21.15.57:0/1879596637 learned_addr learned my addr 172.21.15.57:0/1879596637 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6000dee0 con 0x55ae71cccfb0 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4e60010740 con 0x55ae71cc4430 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6000e530 con 0x55ae71cca790 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 -- 172.21.15.57:0/1879596637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 msgr2=0x55ae71cd8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 -- 172.21.15.57:0/1879596637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 msgr2=0x55ae71ce54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71ce54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.882+0000 7f4e8167c640 1 -- 172.21.15.57:0/1879596637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae71ce0210 con 0x55ae71cc4430 2024-04-21T08:55:17.883 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.883+0000 7f4e8167c640 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cbcfc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55ae71cc4830 tx=0x7f4e6c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=139a30e51197a6f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:17.885 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.886+0000 7f4e66ffd640 1 -- 172.21.15.57:0/1879596637 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c028070 con 0x55ae71cc4430 2024-04-21T08:55:17.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.886+0000 7f4e66ffd640 1 -- 172.21.15.57:0/1879596637 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e6c024c80 con 0x55ae71cc4430 2024-04-21T08:55:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 msgr2=0x55ae71cbcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cbcfc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x55ae71cc4830 tx=0x7f4e6c016000 comp rx=0 tx=0).stop 2024-04-21T08:55:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 shutdown_connections 2024-04-21T08:55:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cccfb0 0x55ae71ce54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cca790 0x55ae71cd8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 --2- 172.21.15.57:0/1879596637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cc4430 0x55ae71cbcfc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.887+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 >> 172.21.15.57:0/1879596637 conn(0x55ae71cbe150 msgr2=0x55ae71cbc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:17.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.894+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 shutdown_connections 2024-04-21T08:55:17.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.894+0000 7f4e865ab740 1 -- 172.21.15.57:0/1879596637 wait complete. 2024-04-21T08:55:17.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.894+0000 7f4e865ab740 1 -- start start 2024-04-21T08:55:17.894 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.894+0000 7f4e865ab740 1 Processor -- start 2024-04-21T08:55:17.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.895+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71cc6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.895+0000 7f4e865ab740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cca790 0x55ae71cc7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e8167c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71cc6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e8167c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71cc6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35874/0 (socket says 172.21.15.57:35874) 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e8167c640 1 -- 172.21.15.57:0/2641984592 learned_addr learned my addr 172.21.15.57:0/2641984592 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cccfb0 0x55ae71d42d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6000ec00 con 0x55ae71cca790 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cccfb0 0x55ae71d42d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4e60010070 con 0x55ae71cccfb0 2024-04-21T08:55:17.896 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.896+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6000f2d0 con 0x55ae71cc4430 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e865ab740 10 client.0 _ll_register_callbacks cb 0x7f4e7809cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e81e7d640 1 -- 172.21.15.57:0/2641984592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 msgr2=0x55ae71cc6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71cc6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e81e7d640 1 -- 172.21.15.57:0/2641984592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cca790 msgr2=0x55ae71cc7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cca790 0x55ae71cc7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e81e7d640 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae71d43250 con 0x55ae71cccfb0 2024-04-21T08:55:17.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e81e7d640 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cccfb0 0x55ae71d42d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4e7809b230 tx=0x7f4e78015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21ff2bb0e71e98b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.897+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e6c024c80 con 0x55ae71cccfb0 2024-04-21T08:55:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.898+0000 7f4e64ff9640 10 client.26657 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:17.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.898+0000 7f4e64ff9640 1 -- 172.21.15.57:0/2641984592 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e780aa020 con 0x55ae71cccfb0 2024-04-21T08:55:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.898+0000 7f4e64ff9640 1 -- 172.21.15.57:0/2641984592 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7f4e78005020 con 0x55ae71cccfb0 2024-04-21T08:55:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.899+0000 7f4e64ff9640 1 -- 172.21.15.57:0/2641984592 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f4e78020040 con 0x55ae71cccfb0 2024-04-21T08:55:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.899+0000 7f4e865ab740 10 client.26657 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:17.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.899+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({fsmap=21}) -- 0x55ae71d44280 con 0x55ae71cccfb0 2024-04-21T08:55:17.902 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e64ff9640 1 -- 172.21.15.57:0/2641984592 <== mon.2 v2:172.21.15.129:3300/0 4 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f4e7809c3d0 con 0x55ae71cccfb0 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e865ab740 10 client.26657 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e865ab740 10 client.26657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e865ab740 20 client.26657 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71d44700 0x55ae71d46af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e865ab740 4 client.26657 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f4e6c024c80 con 0x55ae71d44700 2024-04-21T08:55:17.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.903+0000 7f4e8167c640 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71d44700 0x55ae71d46af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:17.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.910+0000 7f4e64ff9640 10 client.26657 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:17.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.910+0000 7f4e8167c640 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71d44700 0x55ae71d46af0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21c569b8c82a9ec5 server_cookie=d6f9874c5460ae9d in_seq=0 out_seq=0 2024-04-21T08:55:17.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.914+0000 7f4e64ff9640 1 -- 172.21.15.57:0/2641984592 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f4e6c024c80 con 0x55ae71d44700 2024-04-21T08:55:17.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.914+0000 7f4e64ff9640 10 client.26657 handle_command_reply: tid=0 2024-04-21T08:55:17.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.915+0000 7f4e865ab740 1 client.26657 shutdown 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cccfb0 msgr2=0x55ae71d42d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cccfb0 0x55ae71d42d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4e7809b230 tx=0x7f4e78015000 comp rx=0 tx=0).stop 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 shutdown_connections 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ae71d44700 0x55ae71d46af0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ae71cccfb0 0x55ae71d42d10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ae71cca790 0x55ae71cc7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 --2- 172.21.15.57:0/2641984592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae71cc4430 0x55ae71cc6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e64ff9640 0 client.26657 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 >> 172.21.15.57:0/2641984592 conn(0x55ae71cbe150 msgr2=0x55ae71cbd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:17.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 shutdown_connections 2024-04-21T08:55:17.928 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.927+0000 7f4e865ab740 1 -- 172.21.15.57:0/2641984592 wait complete. 2024-04-21T08:55:17.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.939+0000 7f4e865ab740 20 client.26657 trim_cache size 0 max 16384 2024-04-21T08:55:17.941 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:55:17.941 INFO:teuthology.orchestra.run.smithi057.stdout: "return_code": 0, 2024-04-21T08:55:17.941 INFO:teuthology.orchestra.run.smithi057.stdout: "scrub_tag": "29ebec1e-383b-43cc-b1b5-8bf451215c20", 2024-04-21T08:55:17.941 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": "asynchronous" 2024-04-21T08:55:17.941 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:55:17.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.952+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4e680a3ed0 msgr2=0x7f4e680a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.952+0000 7f4e8267e640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4e680a3ed0 0x7f4e680a6390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x55ae71cbf1c0 tx=0x7f4e6c005f90 comp rx=0 tx=0).stop 2024-04-21T08:55:17.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.952+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 msgr2=0x7f4e7c07e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:17.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.952+0000 7f4e8267e640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 0x7f4e7c07e600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800d750 tx=0x7f4e7800dc10 comp rx=0 tx=0).stop 2024-04-21T08:55:17.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.958+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 shutdown_connections 2024-04-21T08:55:17.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.958+0000 7f4e8267e640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4e680a3ed0 0x7f4e680a6390 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.958+0000 7f4e8267e640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e7c07e150 0x7f4e7c07e600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.958+0000 7f4e8267e640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e7c10e390 0x7f4e7c07dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.958+0000 7f4e8267e640 1 --2- 172.21.15.57:0/1835647478 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e7c10c690 0x7f4e7c083b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:17.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.958+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 >> 172.21.15.57:0/1835647478 conn(0x7f4e7c06ced0 msgr2=0x7f4e7c10a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:17.959 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.959+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 shutdown_connections 2024-04-21T08:55:17.959 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:17.959+0000 7f4e8267e640 1 -- 172.21.15.57:0/1835647478 wait complete. 2024-04-21T08:55:17.986 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:17.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:18.000 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/unicode/built-in.a 2024-04-21T08:55:18.048 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/main.o 2024-04-21T08:55:18.088 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/strict_rwx.o 2024-04-21T08:55:18.232 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/super.o 2024-04-21T08:55:18.402 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nls/nls_iso8859-1.o 2024-04-21T08:55:18.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 -- 172.21.15.57:0/1043415850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410a400 msgr2=0x7ff2f410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 --2- 172.21.15.57:0/1043415850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410a400 0x7ff2f410d930 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009930 tx=0x7ff2e402bc60 comp rx=0 tx=0).stop 2024-04-21T08:55:18.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 -- 172.21.15.57:0/1043415850 shutdown_connections 2024-04-21T08:55:18.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 --2- 172.21.15.57:0/1043415850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f410e060 0x7ff2f4115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 --2- 172.21.15.57:0/1043415850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410a400 0x7ff2f410d930 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 --2- 172.21.15.57:0/1043415850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f4109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 -- 172.21.15.57:0/1043415850 >> 172.21.15.57:0/1043415850 conn(0x7ff2f406cf30 msgr2=0x7ff2f406d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:18.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 -- 172.21.15.57:0/1043415850 shutdown_connections 2024-04-21T08:55:18.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.490+0000 7ff2f9575640 1 -- 172.21.15.57:0/1043415850 wait complete. 2024-04-21T08:55:18.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.491+0000 7ff2f9575640 1 Processor -- start 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 -- start start 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f41a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410e060 0x7ff2f419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f419f240 0x7ff2f419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f4119830 con 0x7ff2f419f240 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41196b0 con 0x7ff2f410e060 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f9575640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f41199b0 con 0x7ff2f4109a50 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f41a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f41a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35884/0 (socket says 172.21.15.57:35884) 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.492+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/3794709140 learned_addr learned my addr 172.21.15.57:0/3794709140 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.493+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/3794709140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410e060 msgr2=0x7ff2f419ed00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.493+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410e060 0x7ff2f419ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.493+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f419f240 msgr2=0x7ff2f419f6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:55:18.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.498+0000 7ff2f37fe640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f419f240 0x7ff2f419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.498+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f419f240 0x7ff2f419f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.498+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/3794709140 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f41bc540 con 0x7ff2f4109a50 2024-04-21T08:55:18.499 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.498+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f41a6c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800b450 tx=0x7ff2e800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:18.505 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.506+0000 7ff2dbfff640 1 -- 172.21.15.57:0/3794709140 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8015020 con 0x7ff2f4109a50 2024-04-21T08:55:18.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.506+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f41bc7d0 con 0x7ff2f4109a50 2024-04-21T08:55:18.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.506+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f41bcd20 con 0x7ff2f4109a50 2024-04-21T08:55:18.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.506+0000 7ff2dbfff640 1 -- 172.21.15.57:0/3794709140 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e8011d40 con 0x7ff2f4109a50 2024-04-21T08:55:18.506 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.507+0000 7ff2dbfff640 1 -- 172.21.15.57:0/3794709140 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e800f340 con 0x7ff2f4109a50 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2dbfff640 1 -- 172.21.15.57:0/3794709140 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff2e8018020 con 0x7ff2f4109a50 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 Processor -- start 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 -- start start 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a279eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27a76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5634a27031e0 con 0x5634a279e700 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5634a26a0390 con 0x5634a279f020 2024-04-21T08:55:18.511 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.511+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5634a26ff1a0 con 0x5634a27a7c00 2024-04-21T08:55:18.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27a76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35888/0 (socket says 172.21.15.57:35888) 2024-04-21T08:55:18.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 -- 172.21.15.57:0/425619406 learned_addr learned my addr 172.21.15.57:0/425619406 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:18.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 -- 172.21.15.57:0/425619406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 msgr2=0x5634a27a76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27a76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 -- 172.21.15.57:0/425619406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 msgr2=0x5634a279eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.513 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.512+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a279eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.513+0000 7ff2dbfff640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff2dc0a3f20 0x7ff2dc0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.513+0000 7ff2dbfff640 1 -- 172.21.15.57:0/3794709140 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7ff2e80d15a0 con 0x7ff2f4109a50 2024-04-21T08:55:18.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.516+0000 7ff2f27fc640 1 -- 172.21.15.57:0/425619406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634a24df7d0 con 0x5634a27a7c00 2024-04-21T08:55:18.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.516+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a9ff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4005c20 tx=0x7ff2e4009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a04721c7badb8bc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:18.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.519+0000 7ff2d97fa640 1 -- 172.21.15.57:0/425619406 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4041020 con 0x5634a27a7c00 2024-04-21T08:55:18.518 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.519+0000 7ff2d97fa640 1 -- 172.21.15.57:0/425619406 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e402ccc0 con 0x5634a27a7c00 2024-04-21T08:55:18.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/425619406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 msgr2=0x5634a27a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a9ff0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4005c20 tx=0x7ff2e4009680 comp rx=0 tx=0).stop 2024-04-21T08:55:18.519 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/425619406 shutdown_connections 2024-04-21T08:55:18.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a9ff0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27a76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/425619406 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a279eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.520 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.520+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/425619406 >> 172.21.15.57:0/425619406 conn(0x5634a2798e30 msgr2=0x5634a2799210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:18.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.526+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/425619406 shutdown_connections 2024-04-21T08:55:18.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.526+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/425619406 wait complete. 2024-04-21T08:55:18.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.526+0000 7ff2fbdfd740 1 -- start start 2024-04-21T08:55:18.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.526+0000 7ff2fbdfd740 1 Processor -- start 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279f020 0x5634a27a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5634a27aa9e0 con 0x5634a279f020 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5634a26bc2f0 con 0x5634a279e700 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5634a27aab60 con 0x5634a27a7c00 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2fbdfd740 10 client.0 _ll_register_callbacks cb 0x5634a26c9580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55450/0 (socket says 172.21.15.57:55450) 2024-04-21T08:55:18.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/1990771065 learned_addr learned my addr 172.21.15.57:0/1990771065 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.527+0000 7ff2f37fe640 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.528+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/1990771065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 msgr2=0x5634a27a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.528+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.528+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/1990771065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279f020 msgr2=0x5634a27a50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.528+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279f020 0x5634a27a50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.528+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/1990771065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634a24153f0 con 0x5634a279e700 2024-04-21T08:55:18.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.546+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a4b60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009fd0 tx=0x7ff2e809cfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed117dc0fa6be219 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:18.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.546+0000 7ff2d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:18.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.549+0000 7ff2f37fe640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff2dc0a3f20 0x7ff2dc0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.555+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/1990771065 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e80d1370 con 0x5634a279e700 2024-04-21T08:55:18.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.555+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/1990771065 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7ff2e80a1150 con 0x5634a279e700 2024-04-21T08:55:18.555 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.555+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2e402ccc0 con 0x5634a279e700 2024-04-21T08:55:18.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.576+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/1990771065 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff2e809d8e0 con 0x5634a279e700 2024-04-21T08:55:18.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.578+0000 7ff2fbdfd740 10 client.26666 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:18.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.578+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5634a23d50f0 con 0x5634a279e700 2024-04-21T08:55:18.579 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.580+0000 7ff2f37fe640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff2dc0a3f20 0x7ff2dc0a63e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0048d0 tx=0x7ff2ec003a80 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:18.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/1990771065 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e402ccc0 con 0x5634a279e700 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/1990771065 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff2e809c630 con 0x5634a279e700 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2fbdfd740 10 client.26666 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2fbdfd740 10 client.26666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2fbdfd740 20 client.26666 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a27b7c20 0x5634a27ba060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2fbdfd740 4 client.26666 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634a2710c00 con 0x5634a27b7c20 2024-04-21T08:55:18.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.591+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a27b7c20 0x5634a27ba060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.593+0000 7ff2d6ff5640 10 client.26666 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:18.592 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.593+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a27b7c20 0x5634a27ba060 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=589436f7f2fa2d25 server_cookie=788af4d3f375052c in_seq=0 out_seq=0 2024-04-21T08:55:18.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.593+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/1990771065 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634a2710c00 con 0x5634a27b7c20 2024-04-21T08:55:18.593 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.594+0000 7ff2d6ff5640 10 client.26666 handle_command_reply: tid=0 2024-04-21T08:55:18.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.594+0000 7ff2fbdfd740 1 client.26666 shutdown 2024-04-21T08:55:18.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.594+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 msgr2=0x5634a27a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.594+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a4b60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009fd0 tx=0x7ff2e809cfb0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 shutdown_connections 2024-04-21T08:55:18.596 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a27b7c20 0x5634a27ba060 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a27a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279f020 0x5634a27a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1990771065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a4b60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2d6ff5640 0 client.26666 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:18.597 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.597+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 >> 172.21.15.57:0/1990771065 conn(0x5634a2798e30 msgr2=0x5634a279ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:18.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.606+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 shutdown_connections 2024-04-21T08:55:18.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.615+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1990771065 wait complete. 2024-04-21T08:55:18.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.622+0000 7ff2fbdfd740 20 client.26666 trim_cache size 0 max 16384 2024-04-21T08:55:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:18 smithi136 ceph-mon[31791]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-04-21T08:55:18.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.714+0000 7ff2fbdfd740 1 Processor -- start 2024-04-21T08:55:18.714 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.714+0000 7ff2fbdfd740 1 -- start start 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 0x5634a279aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a27a7c00 0x5634a27a5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e000dff0 con 0x5634a27a7c00 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e0010850 con 0x5634a279e700 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2f27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 0x5634a279aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.715 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2f27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 0x5634a279aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:35916/0 (socket says 172.21.15.57:35916) 2024-04-21T08:55:18.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2f27fc640 1 -- 172.21.15.57:0/1019332736 learned_addr learned my addr 172.21.15.57:0/1019332736 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:18.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.715+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1019332736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e000e640 con 0x5634a279f020 2024-04-21T08:55:18.716 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2f27fc640 1 -- 172.21.15.57:0/1019332736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 msgr2=0x5634a27a7290 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a7290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2f27fc640 1 -- 172.21.15.57:0/1019332736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a27a7c00 msgr2=0x5634a27a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a27a7c00 0x5634a27a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2f27fc640 1 -- 172.21.15.57:0/1019332736 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634a2798c60 con 0x5634a279f020 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 0x5634a279aaf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4006010 tx=0x7ff2e402bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fdd01af4dbfad91 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.716+0000 7ff2d8ff9640 1 -- 172.21.15.57:0/1019332736 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4041040 con 0x5634a279f020 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.717+0000 7ff2d8ff9640 1 -- 172.21.15.57:0/1019332736 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e404f070 con 0x5634a279f020 2024-04-21T08:55:18.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.717+0000 7ff2d8ff9640 1 -- 172.21.15.57:0/1019332736 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4004860 con 0x5634a279f020 2024-04-21T08:55:18.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.724+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1019332736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 msgr2=0x5634a279aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.723 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.724+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 0x5634a279aaf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4006010 tx=0x7ff2e402bc60 comp rx=0 tx=0).stop 2024-04-21T08:55:18.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.736+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1019332736 shutdown_connections 2024-04-21T08:55:18.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.736+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a27a7c00 0x5634a27a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.736+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a279f020 0x5634a279aaf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.736+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/1019332736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279e700 0x5634a27a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.736 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.736+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1019332736 >> 172.21.15.57:0/1019332736 conn(0x5634a2798e30 msgr2=0x5634a2797640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:18.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.742+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1019332736 shutdown_connections 2024-04-21T08:55:18.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.742+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/1019332736 wait complete. 2024-04-21T08:55:18.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.742+0000 7ff2fbdfd740 1 -- start start 2024-04-21T08:55:18.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.743+0000 7ff2fbdfd740 1 Processor -- start 2024-04-21T08:55:18.743 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.743+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a27abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.744+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27ac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.744+0000 7ff2fbdfd740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a281d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.744+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e000ed10 con 0x5634a279e700 2024-04-21T08:55:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.744+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e0010180 con 0x5634a279f020 2024-04-21T08:55:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.744+0000 7ff2fbdfd740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e000f3e0 con 0x5634a27a7c00 2024-04-21T08:55:18.744 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.744+0000 7ff2fbdfd740 10 client.0 _ll_register_callbacks cb 0x7ff2e8010f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:18.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.749+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a27abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.749+0000 7ff2f2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a27abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47510/0 (socket says 172.21.15.57:47510) 2024-04-21T08:55:18.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.749+0000 7ff2f2ffd640 1 -- 172.21.15.57:0/257671185 learned_addr learned my addr 172.21.15.57:0/257671185 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.750+0000 7ff2f37fe640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a281d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.750+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.751+0000 7ff2f27fc640 1 -- 172.21.15.57:0/257671185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 msgr2=0x5634a281d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.751+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a281d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.751+0000 7ff2f27fc640 1 -- 172.21.15.57:0/257671185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 msgr2=0x5634a27abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.751+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a27abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.751+0000 7ff2f27fc640 1 -- 172.21.15.57:0/257671185 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634a281d6e0 con 0x5634a279f020 2024-04-21T08:55:18.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.751+0000 7ff2f27fc640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27ac250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4040040 tx=0x7ff2e402ca40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2293b16c127c33fc server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.760+0000 7ff2d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.760+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/257671185 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4041040 con 0x5634a279f020 2024-04-21T08:55:18.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.760+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/257671185 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7ff2e4004240 con 0x5634a279f020 2024-04-21T08:55:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.764+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2e404f070 con 0x5634a279f020 2024-04-21T08:55:18.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.770+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/257671185 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7ff2e4004440 con 0x5634a279f020 2024-04-21T08:55:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2fbdfd740 10 client.26669 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5634a281e870 con 0x5634a279f020 2024-04-21T08:55:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/257671185 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2e4004d10 con 0x5634a279f020 2024-04-21T08:55:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/257671185 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7ff2e4003b40 con 0x5634a279f020 2024-04-21T08:55:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2fbdfd740 10 client.26669 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:18.771 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2fbdfd740 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2fbdfd740 20 client.26669 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.771+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a281eb40 0x5634a2820f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.772+0000 7ff2fbdfd740 4 client.26669 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.772+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7ff2e404f070 con 0x5634a281eb40 2024-04-21T08:55:18.772 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.772+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a281eb40 0x5634a2820f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.772+0000 7ff2d6ff5640 10 client.26669 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.772+0000 7ff2f2ffd640 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a281eb40 0x5634a2820f30 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=405bc8b992403abc server_cookie=5ab458e2819efc9c in_seq=0 out_seq=0 2024-04-21T08:55:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.773+0000 7ff2d6ff5640 1 -- 172.21.15.57:0/257671185 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x7ff2e404f070 con 0x5634a281eb40 2024-04-21T08:55:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.773+0000 7ff2d6ff5640 10 client.26669 handle_command_reply: tid=0 2024-04-21T08:55:18.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.773+0000 7ff2fbdfd740 1 client.26669 shutdown 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 msgr2=0x5634a27ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27ac250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4040040 tx=0x7ff2e402ca40 comp rx=0 tx=0).stop 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 shutdown_connections 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5634a281eb40 0x5634a2820f30 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5634a27a7c00 0x5634a281d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5634a279f020 0x5634a27ac250 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 --2- 172.21.15.57:0/257671185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5634a279e700 0x5634a27abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2d6ff5640 0 client.26669 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 >> 172.21.15.57:0/257671185 conn(0x5634a2798e30 msgr2=0x5634a279a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 shutdown_connections 2024-04-21T08:55:18.775 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 1 -- 172.21.15.57:0/257671185 wait complete. 2024-04-21T08:55:18.776 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.775+0000 7ff2fbdfd740 20 client.26669 trim_cache size 0 max 16384 2024-04-21T08:55:18.776 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (4019 inodes in the stack)", 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: "29ebec1e-383b-43cc-b1b5-8bf451215c20": { 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "29ebec1e-383b-43cc-b1b5-8bf451215c20", 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:55:18.777 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff2dc0a3f20 msgr2=0x7ff2dc0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff2dc0a3f20 0x7ff2dc0a63e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0048d0 tx=0x7ff2ec003a80 comp rx=0 tx=0).stop 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 msgr2=0x7ff2f41a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f41a6c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff2e800b450 tx=0x7ff2e800b910 comp rx=0 tx=0).stop 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 shutdown_connections 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7ff2dc0a3f20 0x7ff2dc0a63e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2f419f240 0x7ff2f419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2f410e060 0x7ff2f419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 --2- 172.21.15.57:0/3794709140 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f4109a50 0x7ff2f41a6c60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 >> 172.21.15.57:0/3794709140 conn(0x7ff2f406cf30 msgr2=0x7ff2f40729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:18.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 shutdown_connections 2024-04-21T08:55:18.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:18.794+0000 7ff2f9575640 1 -- 172.21.15.57:0/3794709140 wait complete. 2024-04-21T08:55:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:18 smithi129 ceph-mon[31272]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-04-21T08:55:18.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:18 smithi057 ceph-mon[27968]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-04-21T08:55:18.845 INFO:tasks.cephfs.filesystem:scrub status for tag:29ebec1e-383b-43cc-b1b5-8bf451215c20 - {'path': '/', 'tag': '29ebec1e-383b-43cc-b1b5-8bf451215c20', 'options': 'recursive,force'} 2024-04-21T08:55:18.876 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nls/nls_utf8.o 2024-04-21T08:55:19.291 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/time.o 2024-04-21T08:55:19.355 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/svc.o 2024-04-21T08:55:19.638 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/nls/built-in.a 2024-04-21T08:55:19.749 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/svclock.o 2024-04-21T08:55:19.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:19 smithi129 ceph-mon[31272]: scrub queued for path: / 2024-04-21T08:55:19.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:19 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:19.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:19 smithi129 ceph-mon[31272]: scrub summary: active paths [/] 2024-04-21T08:55:19.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:19 smithi057 ceph-mon[27968]: scrub queued for path: / 2024-04-21T08:55:19.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:19 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:19.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:19 smithi057 ceph-mon[27968]: scrub summary: active paths [/] 2024-04-21T08:55:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:19 smithi136 ceph-mon[31791]: scrub queued for path: / 2024-04-21T08:55:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:19 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:19.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:19 smithi136 ceph-mon[31791]: scrub summary: active paths [/] 2024-04-21T08:55:20.159 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/timer.o 2024-04-21T08:55:20.536 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/io.o 2024-04-21T08:55:20.545 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/hugetlb_vmemmap.o 2024-04-21T08:55:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:20 smithi129 ceph-mon[31272]: pgmap v309: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-21T08:55:20.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:20 smithi057 ceph-mon[27968]: pgmap v309: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-21T08:55:20.866 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/kmod.o 2024-04-21T08:55:20.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:20 smithi136 ceph-mon[31791]: pgmap v309: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 1.4 MiB/s wr, 193 op/s 2024-04-21T08:55:20.940 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/hrtimer.o 2024-04-21T08:55:21.147 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/timekeeping.o 2024-04-21T08:55:21.728 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/svcshare.o 2024-04-21T08:55:21.936 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mempolicy.o 2024-04-21T08:55:22.020 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/direct.o 2024-04-21T08:55:22.294 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/tree_lookup.o 2024-04-21T08:55:22.309 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/init.o 2024-04-21T08:55:22.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:22 smithi129 ceph-mon[31272]: pgmap v310: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 931 KiB/s wr, 171 op/s 2024-04-21T08:55:22.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:22 smithi057 ceph-mon[27968]: pgmap v310: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 931 KiB/s wr, 171 op/s 2024-04-21T08:55:22.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:22 smithi136 ceph-mon[31791]: pgmap v310: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 931 KiB/s wr, 171 op/s 2024-04-21T08:55:22.929 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/kallsyms.o 2024-04-21T08:55:23.006 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/ntp.o 2024-04-21T08:55:23.179 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/inode.o 2024-04-21T08:55:23.296 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/svcproc.o 2024-04-21T08:55:23.313 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/procfs.o 2024-04-21T08:55:23.429 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/ioctl.o 2024-04-21T08:55:24.012 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_super.o 2024-04-21T08:55:24.156 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/module/sysfs.o 2024-04-21T08:55:24.232 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/clocksource.o 2024-04-21T08:55:24.301 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/root.o 2024-04-21T08:55:24.577 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/pagelist.o 2024-04-21T08:55:24.740 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/sparse.o 2024-04-21T08:55:24.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:24 smithi129 ceph-mon[31272]: pgmap v311: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 463 KiB/s wr, 245 op/s 2024-04-21T08:55:24.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:24 smithi057 ceph-mon[27968]: pgmap v311: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 463 KiB/s wr, 245 op/s 2024-04-21T08:55:24.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:24 smithi136 ceph-mon[31791]: pgmap v311: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 463 KiB/s wr, 245 op/s 2024-04-21T08:55:24.959 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/svcsubs.o 2024-04-21T08:55:24.964 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_inode.o 2024-04-21T08:55:25.345 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/module/built-in.a 2024-04-21T08:55:25.363 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/jiffies.o 2024-04-21T08:55:25.419 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/symlink.o 2024-04-21T08:55:25.482 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/waitq.o 2024-04-21T08:55:25.625 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/mballoc.o 2024-04-21T08:55:25.960 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/sparse-vmemmap.o 2024-04-21T08:55:26.156 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/timer_list.o 2024-04-21T08:55:26.236 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/mmu_notifier.o 2024-04-21T08:55:26.373 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/mon.o 2024-04-21T08:55:26.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:26 smithi129 ceph-mon[31272]: pgmap v312: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 189 KiB/s wr, 238 op/s 2024-04-21T08:55:26.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:26 smithi057 ceph-mon[27968]: pgmap v312: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 189 KiB/s wr, 238 op/s 2024-04-21T08:55:26.841 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_inode_dotl.o 2024-04-21T08:55:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:26 smithi136 ceph-mon[31791]: pgmap v312: 129 pgs: 15 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 73 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 189 KiB/s wr, 238 op/s 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 -- 172.21.15.57:0/3972401637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810a8f0 msgr2=0x7fb77810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 --2- 172.21.15.57:0/3972401637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810a8f0 0x7fb77810acd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb774009900 tx=0x7fb77402bce0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 -- 172.21.15.57:0/3972401637 shutdown_connections 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 --2- 172.21.15.57:0/3972401637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb778113ea0 0x7fb778116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 --2- 172.21.15.57:0/3972401637 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb77810b210 0x7fb778113960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 --2- 172.21.15.57:0/3972401637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810a8f0 0x7fb77810acd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 -- 172.21.15.57:0/3972401637 >> 172.21.15.57:0/3972401637 conn(0x7fb77806d020 msgr2=0x7fb77806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 -- 172.21.15.57:0/3972401637 shutdown_connections 2024-04-21T08:55:27.042 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.041+0000 7fb780481640 1 -- 172.21.15.57:0/3972401637 wait complete. 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 Processor -- start 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 -- start start 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb77810a8f0 0x7fb7781a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 0x7fb7781a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb778113ea0 0x7fb7781a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb778119930 con 0x7fb77810a8f0 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7781197b0 con 0x7fb77810b210 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.042+0000 7fb780481640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb778119ab0 con 0x7fb778113ea0 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 0x7fb7781a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 0x7fb7781a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40574/0 (socket says 172.21.15.57:40574) 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 -- 172.21.15.57:0/1125764981 learned_addr learned my addr 172.21.15.57:0/1125764981 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb778113ea0 0x7fb7781a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 -- 172.21.15.57:0/1125764981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb778113ea0 msgr2=0x7fb7781a5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb778113ea0 0x7fb7781a5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 -- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb77810a8f0 msgr2=0x7fb7781a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb77810a8f0 0x7fb7781a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.043+0000 7fb77da1c640 1 -- 172.21.15.57:0/1125764981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7781a5eb0 con 0x7fb77810b210 2024-04-21T08:55:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.044+0000 7fb77da1c640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 0x7fb7781a1400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c890 tx=0x7fb76800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.051+0000 7fb76f7fe640 1 -- 172.21.15.57:0/1125764981 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768004220 con 0x7fb77810b210 2024-04-21T08:55:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.051+0000 7fb76f7fe640 1 -- 172.21.15.57:0/1125764981 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7680043c0 con 0x7fb77810b210 2024-04-21T08:55:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.051+0000 7fb76f7fe640 1 -- 172.21.15.57:0/1125764981 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768004a00 con 0x7fb77810b210 2024-04-21T08:55:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.051+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7781a61a0 con 0x7fb77810b210 2024-04-21T08:55:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.051+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb778114280 con 0x7fb77810b210 2024-04-21T08:55:27.056 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.056+0000 7fb76f7fe640 1 -- 172.21.15.57:0/1125764981 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb768004ba0 con 0x7fb77810b210 2024-04-21T08:55:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.057+0000 7fb76f7fe640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb7640a3ff0 0x7fb7640a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.057 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.057+0000 7fb76f7fe640 1 -- 172.21.15.57:0/1125764981 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7fb7680d18b0 con 0x7fb77810b210 2024-04-21T08:55:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.063+0000 7fb783076740 1 Processor -- start 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 -- start start 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15888fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a158944d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x559a157ed8d0 con 0x559a15888b60 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559a1578ae80 con 0x559a158898d0 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb783076740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x559a157e9890 con 0x559a158920e0 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15888fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15888fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34562/0 (socket says 172.21.15.57:34562) 2024-04-21T08:55:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb77da1c640 1 -- 172.21.15.57:0/2485301881 learned_addr learned my addr 172.21.15.57:0/2485301881 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.067+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a158944d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.068+0000 7fb77ea1e640 1 -- 172.21.15.57:0/2485301881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 msgr2=0x559a15888620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.068+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.068+0000 7fb77ea1e640 1 -- 172.21.15.57:0/2485301881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 msgr2=0x559a15888fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.068+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15888fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.068+0000 7fb77ea1e640 1 -- 172.21.15.57:0/2485301881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a157ac320 con 0x559a158920e0 2024-04-21T08:55:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.068+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a158944d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e50 tx=0x7fb7700107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7627965869628086 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.071+0000 7fb77e21d640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb7640a3ff0 0x7fb7640a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.098 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.099+0000 7fb77e21d640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb7640a3ff0 0x7fb7640a64b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb774036040 tx=0x7fb774000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.099+0000 7fb76cbfe640 1 -- 172.21.15.57:0/2485301881 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770003b70 con 0x559a158920e0 2024-04-21T08:55:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.099+0000 7fb76cbfe640 1 -- 172.21.15.57:0/2485301881 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb770015070 con 0x559a158920e0 2024-04-21T08:55:27.099 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.099+0000 7fb76cbfe640 1 -- 172.21.15.57:0/2485301881 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770011770 con 0x559a158920e0 2024-04-21T08:55:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.100+0000 7fb783076740 1 -- 172.21.15.57:0/2485301881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 msgr2=0x559a158944d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.100+0000 7fb783076740 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a158944d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e50 tx=0x7fb7700107a0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.101+0000 7fb783076740 1 -- 172.21.15.57:0/2485301881 shutdown_connections 2024-04-21T08:55:27.100 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.101+0000 7fb783076740 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a158944d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.101 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.101+0000 7fb783076740 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15888fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.101 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.101+0000 7fb783076740 1 --2- 172.21.15.57:0/2485301881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.101 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.101+0000 7fb783076740 1 -- 172.21.15.57:0/2485301881 >> 172.21.15.57:0/2485301881 conn(0x559a1584fc90 msgr2=0x559a15883850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.106+0000 7fb783076740 1 -- 172.21.15.57:0/2485301881 shutdown_connections 2024-04-21T08:55:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.109+0000 7fb783076740 1 -- 172.21.15.57:0/2485301881 wait complete. 2024-04-21T08:55:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.110+0000 7fb783076740 1 -- start start 2024-04-21T08:55:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.110+0000 7fb783076740 1 Processor -- start 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15891a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a1588ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a1588f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x559a15894e10 con 0x559a15888b60 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559a15775eb0 con 0x559a158898d0 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x559a15894f90 con 0x559a158920e0 2024-04-21T08:55:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb783076740 10 client.0 _ll_register_callbacks cb 0x559a157a2b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a1588ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a1588ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40584/0 (socket says 172.21.15.57:40584) 2024-04-21T08:55:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb77da1c640 1 -- 172.21.15.57:0/24256256 learned_addr learned my addr 172.21.15.57:0/24256256 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.111+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a1588f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.115+0000 7fb77e21d640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15891a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.117+0000 7fb77da1c640 1 -- 172.21.15.57:0/24256256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 msgr2=0x559a1588f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.117+0000 7fb77da1c640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a1588f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.117+0000 7fb77da1c640 1 -- 172.21.15.57:0/24256256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 msgr2=0x559a15891a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.117+0000 7fb77da1c640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15891a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.117+0000 7fb77da1c640 1 -- 172.21.15.57:0/24256256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a15500210 con 0x559a158898d0 2024-04-21T08:55:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.117+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a1588f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.121+0000 7fb77da1c640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a1588ee00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb76809d370 tx=0x7fb76809dba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9ef80042dc0cf40 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.121+0000 7fb761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.121+0000 7fb761ffb640 1 -- 172.21.15.57:0/24256256 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7680b5020 con 0x559a158898d0 2024-04-21T08:55:27.121 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.121+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb770015070 con 0x559a158898d0 2024-04-21T08:55:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.126+0000 7fb761ffb640 1 -- 172.21.15.57:0/24256256 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7fb7680cee10 con 0x559a158898d0 2024-04-21T08:55:27.125 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.126+0000 7fb761ffb640 1 -- 172.21.15.57:0/24256256 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb76801d040 con 0x559a158898d0 2024-04-21T08:55:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.126+0000 7fb783076740 10 client.26678 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.126+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559a155cc620 con 0x559a158898d0 2024-04-21T08:55:27.126 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.126+0000 7fb761ffb640 1 -- 172.21.15.57:0/24256256 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb770015070 con 0x559a158898d0 2024-04-21T08:55:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.127+0000 7fb761ffb640 1 -- 172.21.15.57:0/24256256 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb7680bc120 con 0x559a158898d0 2024-04-21T08:55:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb783076740 10 client.26678 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb783076740 10 client.26678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb783076740 20 client.26678 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb783076740 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a158a2690 0x559a158a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb783076740 4 client.26678 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a15801880 con 0x559a158a2690 2024-04-21T08:55:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb77e21d640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a158a2690 0x559a158a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.128+0000 7fb77e21d640 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a158a2690 0x559a158a4ad0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d47ed0b6517922b6 server_cookie=8e79caab01ec741e in_seq=0 out_seq=0 2024-04-21T08:55:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.129+0000 7fb761ffb640 10 client.26678 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.129+0000 7fb761ffb640 1 -- 172.21.15.57:0/24256256 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a15801880 con 0x559a158a2690 2024-04-21T08:55:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.129+0000 7fb761ffb640 10 client.26678 handle_command_reply: tid=0 2024-04-21T08:55:27.129 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.129+0000 7fb783076740 1 client.26678 shutdown 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.129+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 msgr2=0x559a1588ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.129+0000 7fb783076740 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a1588ee00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb76809d370 tx=0x7fb76809dba0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 shutdown_connections 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a158a2690 0x559a158a4ad0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a1588f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a1588ee00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb761ffb640 0 client.26678 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 --2- 172.21.15.57:0/24256256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a15891a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 >> 172.21.15.57:0/24256256 conn(0x559a1584fc90 msgr2=0x559a158855a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 shutdown_connections 2024-04-21T08:55:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 1 -- 172.21.15.57:0/24256256 wait complete. 2024-04-21T08:55:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.130+0000 7fb783076740 20 client.26678 trim_cache size 0 max 16384 2024-04-21T08:55:27.142 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/expire.o 2024-04-21T08:55:27.197 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.197+0000 7fb783076740 1 Processor -- start 2024-04-21T08:55:27.201 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.202+0000 7fb783076740 1 -- start start 2024-04-21T08:55:27.202 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.203+0000 7fb783076740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a1589dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.203+0000 7fb783076740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.203+0000 7fb77e21d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a1589dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.207+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.207+0000 7fb77da1c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:40592/0 (socket says 172.21.15.57:40592) 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.207+0000 7fb77da1c640 1 -- 172.21.15.57:0/2855460277 learned_addr learned my addr 172.21.15.57:0/2855460277 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.207+0000 7fb783076740 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a15891530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.207+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00df50 con 0x559a15888b60 2024-04-21T08:55:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.207+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a15891530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c010720 con 0x559a158898d0 2024-04-21T08:55:27.208 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00e510 con 0x559a158920e0 2024-04-21T08:55:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb77ea1e640 1 -- 172.21.15.57:0/2855460277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 msgr2=0x559a15888120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb77ea1e640 1 -- 172.21.15.57:0/2855460277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 msgr2=0x559a1589dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a1589dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.209+0000 7fb77ea1e640 1 -- 172.21.15.57:0/2855460277 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a158a5150 con 0x559a158920e0 2024-04-21T08:55:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.210+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a15891530 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb770005c10 tx=0x7fb770013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15765229403f542a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.216+0000 7fb763fff640 1 -- 172.21.15.57:0/2855460277 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770003b70 con 0x559a158920e0 2024-04-21T08:55:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.216+0000 7fb763fff640 1 -- 172.21.15.57:0/2855460277 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb770015050 con 0x559a158920e0 2024-04-21T08:55:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.216+0000 7fb763fff640 1 -- 172.21.15.57:0/2855460277 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77001c7a0 con 0x559a158920e0 2024-04-21T08:55:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.218+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 msgr2=0x559a15891530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.218+0000 7fb783076740 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a15891530 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb770005c10 tx=0x7fb770013040 comp rx=0 tx=0).stop 2024-04-21T08:55:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.220+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 shutdown_connections 2024-04-21T08:55:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.220+0000 7fb783076740 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a158920e0 0x559a15891530 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.220+0000 7fb783076740 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158898d0 0x559a15888120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.220+0000 7fb783076740 1 --2- 172.21.15.57:0/2855460277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a15888b60 0x559a1589dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.220+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 >> 172.21.15.57:0/2855460277 conn(0x559a1584fc90 msgr2=0x559a1588d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.233+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 shutdown_connections 2024-04-21T08:55:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.233+0000 7fb783076740 1 -- 172.21.15.57:0/2855460277 wait complete. 2024-04-21T08:55:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.233+0000 7fb783076740 1 -- start start 2024-04-21T08:55:27.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.233+0000 7fb783076740 1 Processor -- start 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a15888b60 0x559a1589a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a158898d0 0x559a1589a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158920e0 0x559a1588b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00ebe0 con 0x559a158898d0 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c010050 con 0x559a158920e0 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb75c00f2b0 con 0x559a15888b60 2024-04-21T08:55:27.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb783076740 10 client.0 _ll_register_callbacks cb 0x7fb768097f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb77e21d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a15888b60 0x559a1589a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb77e21d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a15888b60 0x559a1589a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:47322/0 (socket says 172.21.15.57:47322) 2024-04-21T08:55:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb77e21d640 1 -- 172.21.15.57:0/228692381 learned_addr learned my addr 172.21.15.57:0/228692381 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.234+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158920e0 0x559a1588b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.236+0000 7fb77da1c640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a158898d0 0x559a1589a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.239+0000 7fb77ea1e640 1 -- 172.21.15.57:0/228692381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a15888b60 msgr2=0x559a1589a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.239+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a15888b60 0x559a1589a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.239+0000 7fb77ea1e640 1 -- 172.21.15.57:0/228692381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a158898d0 msgr2=0x559a1589a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.239+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a158898d0 0x559a1589a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.239+0000 7fb77ea1e640 1 -- 172.21.15.57:0/228692381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a15887e30 con 0x559a158920e0 2024-04-21T08:55:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.240+0000 7fb77ea1e640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158920e0 0x559a1588b670 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb770000f80 tx=0x7fb770003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d016fcfd6a1e3fb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.262+0000 7fb761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.262+0000 7fb761ffb640 1 -- 172.21.15.57:0/228692381 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770030070 con 0x559a158920e0 2024-04-21T08:55:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.262+0000 7fb761ffb640 1 -- 172.21.15.57:0/228692381 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9563+0+0 (secure 0 0 0) 0x7fb770005310 con 0x559a158920e0 2024-04-21T08:55:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.264+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb770015050 con 0x559a158920e0 2024-04-21T08:55:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.265+0000 7fb761ffb640 1 -- 172.21.15.57:0/228692381 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb770005510 con 0x559a158920e0 2024-04-21T08:55:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.266+0000 7fb783076740 10 client.26684 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.266+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559a15908700 con 0x559a158920e0 2024-04-21T08:55:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb761ffb640 1 -- 172.21.15.57:0/228692381 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb770015050 con 0x559a158920e0 2024-04-21T08:55:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb761ffb640 1 -- 172.21.15.57:0/228692381 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb770033020 con 0x559a158920e0 2024-04-21T08:55:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb783076740 10 client.26684 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb783076740 10 client.26684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb783076740 20 client.26684 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb783076740 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a15908b20 0x559a1590af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb783076740 4 client.26684 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.270+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a1578d3d0 con 0x559a15908b20 2024-04-21T08:55:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.273+0000 7fb77e21d640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a15908b20 0x559a1590af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.276+0000 7fb761ffb640 10 client.26684 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:27.276+0000 7fb77e21d640 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a15908b20 0x559a1590af10 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15bedc4a73e624fe server_cookie=97cf566e7b2beeb4 in_seq=0 out_seq=0 2024-04-21T08:55:27.320 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/timeconv.o 2024-04-21T08:55:27.379 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/autofs/dev-ioctl.o 2024-04-21T08:55:27.382 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/read.o 2024-04-21T08:55:27.723 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/slub.o 2024-04-21T08:55:27.902 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/trace.o 2024-04-21T08:55:27.967 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/timecounter.o 2024-04-21T08:55:28.155 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/alarmtimer.o 2024-04-21T08:55:28.269 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_addr.o 2024-04-21T08:55:28.292 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/posix-timers.o 2024-04-21T08:55:28.404 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/autofs/built-in.a 2024-04-21T08:55:28.664 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/migrate.o 2024-04-21T08:55:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:28 smithi129 ceph-mon[31272]: pgmap v313: 129 pgs: 13 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 81 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 308 KiB/s wr, 239 op/s 2024-04-21T08:55:28.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:28 smithi057 ceph-mon[27968]: pgmap v313: 129 pgs: 13 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 81 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 308 KiB/s wr, 239 op/s 2024-04-21T08:55:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:28 smithi136 ceph-mon[31791]: pgmap v313: 129 pgs: 13 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 81 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 308 KiB/s wr, 239 op/s 2024-04-21T08:55:29.332 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_file.o 2024-04-21T08:55:29.495 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/xdr.o 2024-04-21T08:55:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.532+0000 7fb761ffb640 1 -- 172.21.15.57:0/228692381 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a1578d3d0 con 0x559a15908b20 2024-04-21T08:55:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.532+0000 7fb761ffb640 10 client.26684 handle_command_reply: tid=0 2024-04-21T08:55:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.532+0000 7fb783076740 1 client.26684 shutdown 2024-04-21T08:55:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.532+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158920e0 msgr2=0x559a1588b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:29.532 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.532+0000 7fb783076740 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158920e0 0x559a1588b670 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb770000f80 tx=0x7fb770003ba0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 shutdown_connections 2024-04-21T08:55:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb783076740 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559a15908b20 0x559a1590af10 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb783076740 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559a158920e0 0x559a1588b670 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb783076740 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559a158898d0 0x559a1589a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb783076740 1 --2- 172.21.15.57:0/228692381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a15888b60 0x559a1589a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 >> 172.21.15.57:0/228692381 conn(0x559a1584fc90 msgr2=0x559a15882040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.535+0000 7fb761ffb640 0 client.26684 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:29.538 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.539+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 shutdown_connections 2024-04-21T08:55:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.539+0000 7fb783076740 1 -- 172.21.15.57:0/228692381 wait complete. 2024-04-21T08:55:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.539+0000 7fb783076740 20 client.26684 trim_cache size 0 max 16384 2024-04-21T08:55:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:55:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:55:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:55:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:55:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb7640a3ff0 msgr2=0x7fb7640a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb7640a3ff0 0x7fb7640a64b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb774036040 tx=0x7fb774000bc0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 msgr2=0x7fb7781a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 0x7fb7781a1400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c890 tx=0x7fb76800cd50 comp rx=0 tx=0).stop 2024-04-21T08:55:29.553 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 shutdown_connections 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb7640a3ff0 0x7fb7640a64b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb778113ea0 0x7fb7781a5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb77810b210 0x7fb7781a1400 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 --2- 172.21.15.57:0/1125764981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb77810a8f0 0x7fb7781a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 >> 172.21.15.57:0/1125764981 conn(0x7fb77806d020 msgr2=0x7fb778071910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 shutdown_connections 2024-04-21T08:55:29.554 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:29.553+0000 7fb780481640 1 -- 172.21.15.57:0/1125764981 wait complete. 2024-04-21T08:55:29.720 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_dir.o 2024-04-21T08:55:29.907 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/symlink.o 2024-04-21T08:55:30.009 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/mmp.o 2024-04-21T08:55:30.191 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/posix-cpu-timers.o 2024-04-21T08:55:30.588 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/vfs_dentry.o 2024-04-21T08:55:30.590 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/posix-clock.o 2024-04-21T08:55:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:30 smithi129 ceph-mon[31272]: pgmap v314: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 529 KiB/s wr, 298 op/s 2024-04-21T08:55:30.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:30 smithi057 ceph-mon[27968]: pgmap v314: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 529 KiB/s wr, 298 op/s 2024-04-21T08:55:30.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:30 smithi136 ceph-mon[31791]: pgmap v314: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 529 KiB/s wr, 298 op/s 2024-04-21T08:55:31.050 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/clnt4xdr.o 2024-04-21T08:55:31.143 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/move_extent.o 2024-04-21T08:55:31.292 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/unlink.o 2024-04-21T08:55:31.534 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/v9fs.o 2024-04-21T08:55:31.573 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/namei.o 2024-04-21T08:55:31.590 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/itimer.o 2024-04-21T08:55:31.692 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/migrate.o 2024-04-21T08:55:31.712 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/clockevents.o 2024-04-21T08:55:32.222 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/tick-common.o 2024-04-21T08:55:32.539 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/memory-tiers.o 2024-04-21T08:55:32.618 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/xdr4.o 2024-04-21T08:55:32.673 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/page_counter.o 2024-04-21T08:55:32.737 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/fid.o 2024-04-21T08:55:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:32 smithi129 ceph-mon[31272]: pgmap v315: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 971 KiB/s wr, 369 op/s 2024-04-21T08:55:32.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:32 smithi057 ceph-mon[27968]: pgmap v315: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 971 KiB/s wr, 369 op/s 2024-04-21T08:55:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:32 smithi136 ceph-mon[31791]: pgmap v315: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 971 KiB/s wr, 369 op/s 2024-04-21T08:55:33.151 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/write.o 2024-04-21T08:55:33.277 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/9p/xattr.o 2024-04-21T08:55:33.440 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/tick-broadcast.o 2024-04-21T08:55:33.746 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/tick-broadcast-hrtimer.o 2024-04-21T08:55:33.837 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/tick-oneshot.o 2024-04-21T08:55:34.056 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/svc4proc.o 2024-04-21T08:55:34.288 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/tick-sched.o 2024-04-21T08:55:34.309 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/vsyscall.o 2024-04-21T08:55:34.316 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/hugetlb_cgroup.o 2024-04-21T08:55:34.318 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/9p/built-in.a 2024-04-21T08:55:34.423 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/page-io.o 2024-04-21T08:55:34.484 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/timekeeping_debug.o 2024-04-21T08:55:34.803 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/time/namespace.o 2024-04-21T08:55:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:34 smithi129 ceph-mon[31272]: pgmap v316: 129 pgs: 9 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 97 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1012 KiB/s wr, 526 op/s 2024-04-21T08:55:34.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:34 smithi057 ceph-mon[27968]: pgmap v316: 129 pgs: 9 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 97 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1012 KiB/s wr, 526 op/s 2024-04-21T08:55:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:34 smithi136 ceph-mon[31791]: pgmap v316: 129 pgs: 9 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 97 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1012 KiB/s wr, 526 op/s 2024-04-21T08:55:35.144 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/readpage.o 2024-04-21T08:55:35.298 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/resize.o 2024-04-21T08:55:35.458 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/super.o 2024-04-21T08:55:35.515 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/lockd/procfs.o 2024-04-21T08:55:35.625 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/early_ioremap.o 2024-04-21T08:55:35.711 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/time/built-in.a 2024-04-21T08:55:35.745 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/symlink.o 2024-04-21T08:55:35.848 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/futex/core.o 2024-04-21T08:55:36.095 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/namespace.o 2024-04-21T08:55:36.161 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/futex/syscalls.o 2024-04-21T08:55:36.370 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/lockd/built-in.a 2024-04-21T08:55:36.429 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/secretmem.o 2024-04-21T08:55:36.481 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/memfd.o 2024-04-21T08:55:36.575 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/sysfs.o 2024-04-21T08:55:36.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:36 smithi129 ceph-mon[31272]: pgmap v317: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 996 KiB/s wr, 695 op/s 2024-04-21T08:55:36.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:36 smithi057 ceph-mon[27968]: pgmap v317: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 996 KiB/s wr, 695 op/s 2024-04-21T08:55:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:36 smithi136 ceph-mon[31791]: pgmap v317: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 996 KiB/s wr, 695 op/s 2024-04-21T08:55:36.922 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/xattr.o 2024-04-21T08:55:37.121 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/futex/pi.o 2024-04-21T08:55:37.279 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/xattr_hurd.o 2024-04-21T08:55:37.280 INFO:tasks.workunit.client.0.smithi057.stdout: CC mm/ptdump.o 2024-04-21T08:55:37.282 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/mount_clnt.o 2024-04-21T08:55:37.538 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/xattr_trusted.o 2024-04-21T08:55:37.590 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/futex/requeue.o 2024-04-21T08:55:37.805 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/futex/waitwake.o 2024-04-21T08:55:37.871 INFO:tasks.workunit.client.0.smithi057.stdout: AR mm/built-in.a 2024-04-21T08:55:38.007 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/hostfs/built-in.a 2024-04-21T08:55:38.043 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/debugfs/inode.o 2024-04-21T08:55:38.046 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/xattr_user.o 2024-04-21T08:55:38.180 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/debugfs/file.o 2024-04-21T08:55:38.376 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/fast_commit.o 2024-04-21T08:55:38.398 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/futex/built-in.a 2024-04-21T08:55:38.410 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfstrace.o 2024-04-21T08:55:38.500 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/cgroup.o 2024-04-21T08:55:38.765 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_clock.o 2024-04-21T08:55:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:38 smithi057 ceph-mon[27968]: pgmap v318: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 998 KiB/s wr, 808 op/s 2024-04-21T08:55:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:38 smithi136 ceph-mon[31791]: pgmap v318: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 998 KiB/s wr, 808 op/s 2024-04-21T08:55:38.926 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/ring_buffer.o 2024-04-21T08:55:39.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:38 smithi129 ceph-mon[31272]: pgmap v318: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 998 KiB/s wr, 808 op/s 2024-04-21T08:55:39.118 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace.o 2024-04-21T08:55:39.216 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/export.o 2024-04-21T08:55:39.338 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/debugfs/built-in.a 2024-04-21T08:55:39.422 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/sysfs.o 2024-04-21T08:55:40.427 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/fs_context.o 2024-04-21T08:55:40.626 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_output.o 2024-04-21T08:55:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:40 smithi057 ceph-mon[27968]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 864 KiB/s wr, 1.23k op/s 2024-04-21T08:55:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:40 smithi057 ceph-mon[27968]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:55:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:40 smithi136 ceph-mon[31791]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 864 KiB/s wr, 1.23k op/s 2024-04-21T08:55:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:40 smithi136 ceph-mon[31791]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:55:40.901 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/bpf/core.o 2024-04-21T08:55:41.013 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfsroot.o 2024-04-21T08:55:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:40 smithi129 ceph-mon[31272]: pgmap v319: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 864 KiB/s wr, 1.23k op/s 2024-04-21T08:55:41.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:40 smithi129 ceph-mon[31272]: osdmap e104: 12 total, 12 up, 12 in 2024-04-21T08:55:41.911 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/sysctl.o 2024-04-21T08:55:42.069 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/rstat.o 2024-04-21T08:55:42.112 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/namespace.o 2024-04-21T08:55:42.142 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/cgroup-v1.o 2024-04-21T08:55:42.162 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs2super.o 2024-04-21T08:55:42.321 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_seq.o 2024-04-21T08:55:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1604850913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 msgr2=0x7fb8c41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1604850913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c41097f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc009ca0 tx=0x7fb8bc02c120 comp rx=0 tx=0).stop 2024-04-21T08:55:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1604850913 shutdown_connections 2024-04-21T08:55:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1604850913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8c410eb90 0x7fb8c407a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1604850913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8c410e1d0 0x7fb8c410e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1604850913 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c41097f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1604850913 >> 172.21.15.57:0/1604850913 conn(0x7fb8c406d020 msgr2=0x7fb8c406d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1604850913 shutdown_connections 2024-04-21T08:55:42.814 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.811+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1604850913 wait complete. 2024-04-21T08:55:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 Processor -- start 2024-04-21T08:55:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 -- start start 2024-04-21T08:55:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c4084a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8c410e1d0 0x7fb8c407eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8c410eb90 0x7fb8c407f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.816 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c407d5e0 con 0x7fb8c4109410 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c407d460 con 0x7fb8c410e1d0 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8cb57e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c407d760 con 0x7fb8c410eb90 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c4084a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c4084a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35108/0 (socket says 172.21.15.57:35108) 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8c931a640 1 -- 172.21.15.57:0/1987586904 learned_addr learned my addr 172.21.15.57:0/1987586904 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8c410e1d0 0x7fb8c407eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.813+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8c410eb90 0x7fb8c407f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.814+0000 7fb8c931a640 1 -- 172.21.15.57:0/1987586904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8c410eb90 msgr2=0x7fb8c407f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.814+0000 7fb8c931a640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8c410eb90 0x7fb8c407f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.814+0000 7fb8c931a640 1 -- 172.21.15.57:0/1987586904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8c410e1d0 msgr2=0x7fb8c407eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.814+0000 7fb8c931a640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8c410e1d0 0x7fb8c407eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.814+0000 7fb8c931a640 1 -- 172.21.15.57:0/1987586904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c407f8a0 con 0x7fb8c4109410 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.814+0000 7fb8c931a640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c4084a30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc02c6b0 tx=0x7fb8bc0347b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.817 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.815+0000 7fb8ba7fc640 1 -- 172.21.15.57:0/1987586904 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc041020 con 0x7fb8c4109410 2024-04-21T08:55:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.815+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c41373e0 con 0x7fb8c4109410 2024-04-21T08:55:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.815+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c41378a0 con 0x7fb8c4109410 2024-04-21T08:55:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.815+0000 7fb8ba7fc640 1 -- 172.21.15.57:0/1987586904 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8bc003e20 con 0x7fb8c4109410 2024-04-21T08:55:42.818 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.816+0000 7fb8ba7fc640 1 -- 172.21.15.57:0/1987586904 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc039040 con 0x7fb8c4109410 2024-04-21T08:55:42.819 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_stat.o 2024-04-21T08:55:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.821+0000 7fb8ba7fc640 1 -- 172.21.15.57:0/1987586904 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb8bc044020 con 0x7fb8c4109410 2024-04-21T08:55:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.823+0000 7fb8ba7fc640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8b00a3e40 0x7fb8b00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.823+0000 7fb8ba7fc640 1 -- 172.21.15.57:0/1987586904 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9487+0+0 (secure 0 0 0) 0x7fb8bc0f2a00 con 0x7fb8c4109410 2024-04-21T08:55:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.823+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8b00a3e40 0x7fb8b00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.823+0000 7fb8ce16f740 1 Processor -- start 2024-04-21T08:55:42.824 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.823+0000 7fb8ce16f740 1 -- start start 2024-04-21T08:55:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.824+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398ed7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.824+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8b00a3e40 0x7fb8b00a6300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4080260 tx=0x7fb8c0005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.824+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ed7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.824+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398ed7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.824+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398ed7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:52294/0 (socket says 172.21.15.57:52294) 2024-04-21T08:55:42.826 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.824+0000 7fb8c931a640 1 -- 172.21.15.57:0/1581202062 learned_addr learned my addr 172.21.15.57:0/1581202062 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.825+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ed7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.825+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.825+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.825+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562398e3c8d0 con 0x562398ee10e0 2024-04-21T08:55:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.825+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562398dd9e80 con 0x562398ed88d0 2024-04-21T08:55:42.827 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.825+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562398e38890 con 0x562398ed7b60 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.826+0000 7fb8c9b1b640 1 -- 172.21.15.57:0/1581202062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 msgr2=0x562398ed7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.826+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ed7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.826+0000 7fb8c9b1b640 1 -- 172.21.15.57:0/1581202062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 msgr2=0x562398ed7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.826+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398ed7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.826+0000 7fb8c9b1b640 1 -- 172.21.15.57:0/1581202062 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562398dfb320 con 0x562398ee10e0 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.827+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee34d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400ad00 tx=0x7fb8b400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5867fc04c45022 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.827+0000 7fb8af7fe640 1 -- 172.21.15.57:0/1581202062 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4005020 con 0x562398ee10e0 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.827+0000 7fb8af7fe640 1 -- 172.21.15.57:0/1581202062 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8b4005670 con 0x562398ee10e0 2024-04-21T08:55:42.828 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.827+0000 7fb8af7fe640 1 -- 172.21.15.57:0/1581202062 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40186a0 con 0x562398ee10e0 2024-04-21T08:55:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.828+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 msgr2=0x562398ee34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.828+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee34d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb8b400ad00 tx=0x7fb8b400f790 comp rx=0 tx=0).stop 2024-04-21T08:55:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 shutdown_connections 2024-04-21T08:55:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee34d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ed7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/1581202062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398ed7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 >> 172.21.15.57:0/1581202062 conn(0x562398e9ec90 msgr2=0x562398ed2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.830 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 shutdown_connections 2024-04-21T08:55:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.829+0000 7fb8ce16f740 1 -- 172.21.15.57:0/1581202062 wait complete. 2024-04-21T08:55:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.830+0000 7fb8ce16f740 1 -- start start 2024-04-21T08:55:42.831 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.830+0000 7fb8ce16f740 1 Processor -- start 2024-04-21T08:55:42.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.831+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ee0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.831+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398edde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.831+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ee0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.832 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.831+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ee0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:38682/0 (socket says 172.21.15.57:38682) 2024-04-21T08:55:42.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.831+0000 7fb8c931a640 1 -- 172.21.15.57:0/3928602894 learned_addr learned my addr 172.21.15.57:0/3928602894 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:42.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.832+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ede460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.832+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398edde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.833 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.832+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ede460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.832+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x562398ee3e10 con 0x562398ee10e0 2024-04-21T08:55:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.832+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562398dc4eb0 con 0x562398ed88d0 2024-04-21T08:55:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.832+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x562398ee3f90 con 0x562398ed7b60 2024-04-21T08:55:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.833+0000 7fb8ce16f740 10 client.0 _ll_register_callbacks cb 0x562398df1b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.833+0000 7fb8c8b19640 1 -- 172.21.15.57:0/3928602894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 msgr2=0x562398ee0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.834 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.833+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ee0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.833+0000 7fb8c8b19640 1 -- 172.21.15.57:0/3928602894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 msgr2=0x562398ede460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.833+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ede460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.833+0000 7fb8c8b19640 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562398b4f210 con 0x562398ed88d0 2024-04-21T08:55:42.835 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.835+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398edde50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40802f0 tx=0x7fb8c0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=910ab10ae5b22e80 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.846+0000 7fb8acff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:42.847 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.847+0000 7fb8acff9640 1 -- 172.21.15.57:0/3928602894 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c000eaf0 con 0x562398ed88d0 2024-04-21T08:55:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.847+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b4005670 con 0x562398ed88d0 2024-04-21T08:55:42.848 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.847+0000 7fb8acff9640 1 -- 172.21.15.57:0/3928602894 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9487+0+0 (secure 0 0 0) 0x7fb8c000ec90 con 0x562398ed88d0 2024-04-21T08:55:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.848+0000 7fb8acff9640 1 -- 172.21.15.57:0/3928602894 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb8c0028400 con 0x562398ed88d0 2024-04-21T08:55:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.848+0000 7fb8ce16f740 10 client.26693 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:42.849 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.848+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562398c1b620 con 0x562398ed88d0 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8acff9640 1 -- 172.21.15.57:0/3928602894 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8b4005670 con 0x562398ed88d0 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8acff9640 1 -- 172.21.15.57:0/3928602894 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb8c001c6c0 con 0x562398ed88d0 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8ce16f740 10 client.26693 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8ce16f740 10 client.26693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8ce16f740 20 client.26693 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398ef1440 0x562398ef38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8ce16f740 4 client.26693 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:42.855 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562398e50880 con 0x562398ef1440 2024-04-21T08:55:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.854+0000 7fb8c931a640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398ef1440 0x562398ef38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.855+0000 7fb8c931a640 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398ef1440 0x562398ef38c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=834e64a5122bd8ed server_cookie=bb0836e044778b6f in_seq=0 out_seq=0 2024-04-21T08:55:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.855+0000 7fb8acff9640 10 client.26693 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.859+0000 7fb8acff9640 1 -- 172.21.15.57:0/3928602894 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562398e50880 con 0x562398ef1440 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.859+0000 7fb8acff9640 10 client.26693 handle_command_reply: tid=0 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.859+0000 7fb8ce16f740 1 client.26693 shutdown 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 msgr2=0x562398edde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398edde50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40802f0 tx=0x7fb8c0004730 comp rx=0 tx=0).stop 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 shutdown_connections 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398ef1440 0x562398ef38c0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ede460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed88d0 0x562398edde50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8acff9640 0 client.26693 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3928602894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398ee0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 >> 172.21.15.57:0/3928602894 conn(0x562398e9ec90 msgr2=0x562398ed45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 shutdown_connections 2024-04-21T08:55:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3928602894 wait complete. 2024-04-21T08:55:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.860+0000 7fb8ce16f740 20 client.26693 trim_cache size 0 max 16384 2024-04-21T08:55:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:42 smithi136 ceph-mon[31791]: pgmap v321: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 238 KiB/s wr, 1.85k op/s 2024-04-21T08:55:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.895+0000 7fb8ce16f740 1 Processor -- start 2024-04-21T08:55:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.895+0000 7fb8ce16f740 1 -- start start 2024-04-21T08:55:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.895+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398eecda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ed88d0 0x562398edce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ee10e0 0x562398ee0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8ce16f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a800e080 con 0x562398ed88d0 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8ce16f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a8010900 con 0x562398ee10e0 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8ce16f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a800e6f0 con 0x562398ed7b60 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8c8b19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ed88d0 0x562398edce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8c931a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398eecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8c9b1b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ee10e0 0x562398ee0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8c9b1b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ee10e0 0x562398ee0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:52312/0 (socket says 172.21.15.57:52312) 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.896+0000 7fb8c9b1b640 1 -- 172.21.15.57:0/3118935224 learned_addr learned my addr 172.21.15.57:0/3118935224 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c931a640 1 -- 172.21.15.57:0/3118935224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ee10e0 msgr2=0x562398ee0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c931a640 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ee10e0 0x562398ee0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c931a640 1 -- 172.21.15.57:0/3118935224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ed88d0 msgr2=0x562398edce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c931a640 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ed88d0 0x562398edce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c931a640 1 -- 172.21.15.57:0/3118935224 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562398ed2a60 con 0x562398ed7b60 2024-04-21T08:55:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ed88d0 0x562398edce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8c931a640 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398eecda0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x562398ef4140 tx=0x7fb8bc0efde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da94ec2f63f4fea server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8aeffd640 1 -- 172.21.15.57:0/3118935224 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc004040 con 0x562398ed7b60 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.897+0000 7fb8aeffd640 1 -- 172.21.15.57:0/3118935224 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8bc0bed90 con 0x562398ed7b60 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3118935224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 msgr2=0x562398eecda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398eecda0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x562398ef4140 tx=0x7fb8bc0efde0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3118935224 shutdown_connections 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ee10e0 0x562398ee0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ed88d0 0x562398edce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/3118935224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed7b60 0x562398eecda0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.899+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3118935224 >> 172.21.15.57:0/3118935224 conn(0x562398e9ec90 msgr2=0x562398edc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:42.902 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.901+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3118935224 shutdown_connections 2024-04-21T08:55:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.901+0000 7fb8ce16f740 1 -- 172.21.15.57:0/3118935224 wait complete. 2024-04-21T08:55:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 -- start start 2024-04-21T08:55:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 Processor -- start 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed7b60 0x562398ee95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 0x562398ee4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a800edc0 con 0x562398ee10e0 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a8010230 con 0x562398ed7b60 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a800f490 con 0x562398ed88d0 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.902+0000 7fb8ce16f740 10 client.0 _ll_register_callbacks cb 0x7fb8c0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c8b19640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 0x562398ee4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c8b19640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 0x562398ee4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:38708/0 (socket says 172.21.15.57:38708) 2024-04-21T08:55:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c8b19640 1 -- 172.21.15.57:0/2844873573 learned_addr learned my addr 172.21.15.57:0/2844873573 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c931a640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed7b60 0x562398ee95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c931a640 1 -- 172.21.15.57:0/2844873573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 msgr2=0x562398ee4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c931a640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 0x562398ee4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.903+0000 7fb8c931a640 1 -- 172.21.15.57:0/2844873573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 msgr2=0x562398ee4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8c931a640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8c931a640 1 -- 172.21.15.57:0/2844873573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562398ede7d0 con 0x562398ed7b60 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 0x562398ee4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8c9b1b640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8c931a640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed7b60 0x562398ee95e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x562398ef4140 tx=0x7fb8bc0be710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af70c7a9ecc8764a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8acff9640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8acff9640 1 -- 172.21.15.57:0/2844873573 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc0098b0 con 0x562398ed7b60 2024-04-21T08:55:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8bc0bed90 con 0x562398ed7b60 2024-04-21T08:55:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.904+0000 7fb8acff9640 1 -- 172.21.15.57:0/2844873573 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9487+0+0 (secure 0 0 0) 0x7fb8bc0bc990 con 0x562398ed7b60 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8acff9640 1 -- 172.21.15.57:0/2844873573 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb8bc0c7300 con 0x562398ed7b60 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 10 client.26699 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x562398f577f0 con 0x562398ed7b60 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8acff9640 1 -- 172.21.15.57:0/2844873573 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8bc0bed90 con 0x562398ed7b60 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8acff9640 1 -- 172.21.15.57:0/2844873573 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fb8bc03c030 con 0x562398ed7b60 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 10 client.26699 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 10 client.26699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 20 client.26699 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398f57f40 0x562398f5a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 4 client.26699 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.905+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562398ddc3d0 con 0x562398f57f40 2024-04-21T08:55:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.906+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398f57f40 0x562398f5a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.906+0000 7fb8acff9640 10 client.26699 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:42.906+0000 7fb8c8b19640 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398f57f40 0x562398f5a360 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac446647cc1ee50f server_cookie=c9821925527c92e1 in_seq=0 out_seq=0 2024-04-21T08:55:42.910 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_printk.o 2024-04-21T08:55:42.988 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/pid_list.o 2024-04-21T08:55:43.068 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_sched_switch.o 2024-04-21T08:55:43.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:42 smithi129 ceph-mon[31272]: pgmap v321: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 238 KiB/s wr, 1.85k op/s 2024-04-21T08:55:43.100 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:42 smithi057 ceph-mon[27968]: pgmap v321: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 238 KiB/s wr, 1.85k op/s 2024-04-21T08:55:43.284 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/proc.o 2024-04-21T08:55:43.511 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/freezer.o 2024-04-21T08:55:43.603 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/orphan.o 2024-04-21T08:55:43.768 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/bpf/built-in.a 2024-04-21T08:55:43.887 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/acl.o 2024-04-21T08:55:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.930+0000 7fb8acff9640 1 -- 172.21.15.57:0/2844873573 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562398ddc3d0 con 0x562398f57f40 2024-04-21T08:55:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.930+0000 7fb8acff9640 10 client.26699 handle_command_reply: tid=0 2024-04-21T08:55:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 client.26699 shutdown 2024-04-21T08:55:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed7b60 msgr2=0x562398ee95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed7b60 0x562398ee95e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x562398ef4140 tx=0x7fb8bc0be710 comp rx=0 tx=0).stop 2024-04-21T08:55:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 shutdown_connections 2024-04-21T08:55:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x562398f57f40 0x562398f5a360 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x562398ee10e0 0x562398ee4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x562398ed88d0 0x562398ee4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 --2- 172.21.15.57:0/2844873573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562398ed7b60 0x562398ee95e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 >> 172.21.15.57:0/2844873573 conn(0x562398e9ec90 msgr2=0x562398ed1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.932+0000 7fb8acff9640 0 client.26699 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.933+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 shutdown_connections 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.933+0000 7fb8ce16f740 1 -- 172.21.15.57:0/2844873573 wait complete. 2024-04-21T08:55:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.933+0000 7fb8ce16f740 20 client.26699 trim_cache size 0 max 16384 2024-04-21T08:55:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:55:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:55:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:55:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:55:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.939+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8b00a3e40 msgr2=0x7fb8b00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.939+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8b00a3e40 0x7fb8b00a6300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4080260 tx=0x7fb8c0005f90 comp rx=0 tx=0).stop 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.939+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 msgr2=0x7fb8c4084a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.939+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c4084a30 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc02c6b0 tx=0x7fb8bc0347b0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.939+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 shutdown_connections 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.939+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8b00a3e40 0x7fb8b00a6300 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.940+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8c410eb90 0x7fb8c407f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.940+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8c410e1d0 0x7fb8c407eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.940+0000 7fb8cb57e640 1 --2- 172.21.15.57:0/1987586904 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8c4109410 0x7fb8c4084a30 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.940+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 >> 172.21.15.57:0/1987586904 conn(0x7fb8c406d020 msgr2=0x7fb8c4072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:43.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.945+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 shutdown_connections 2024-04-21T08:55:43.945 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:43.945+0000 7fb8cb57e640 1 -- 172.21.15.57:0/1987586904 wait complete. 2024-04-21T08:55:43.953 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ext4/xattr_security.o 2024-04-21T08:55:43.963 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_nop.o 2024-04-21T08:55:44.011 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/blktrace.o 2024-04-21T08:55:44.164 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs2xdr.o 2024-04-21T08:55:44.369 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/legacy_freezer.o 2024-04-21T08:55:44.560 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_events.o 2024-04-21T08:55:44.712 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_export.o 2024-04-21T08:55:44.715 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_event_perf.o 2024-04-21T08:55:44.728 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/pids.o 2024-04-21T08:55:44.730 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/ext4/built-in.a 2024-04-21T08:55:44.800 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_events_filter.o 2024-04-21T08:55:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:44 smithi136 ceph-mon[31791]: pgmap v322: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 191 KiB/s wr, 2.39k op/s 2024-04-21T08:55:45.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:44 smithi129 ceph-mon[31272]: pgmap v322: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 191 KiB/s wr, 2.39k op/s 2024-04-21T08:55:45.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:44 smithi057 ceph-mon[27968]: pgmap v322: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 191 KiB/s wr, 2.39k op/s 2024-04-21T08:55:45.249 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/rdma.o 2024-04-21T08:55:45.471 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/cpuset.o 2024-04-21T08:55:45.596 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_events_trigger.o 2024-04-21T08:55:45.609 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_eprobe.o 2024-04-21T08:55:45.611 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_kprobe.o 2024-04-21T08:55:45.715 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs3super.o 2024-04-21T08:55:46.032 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/error_report-traces.o 2024-04-21T08:55:46.068 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/misc.o 2024-04-21T08:55:46.520 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/power-traces.o 2024-04-21T08:55:46.757 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/rpm-traces.o 2024-04-21T08:55:46.767 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cgroup/debug.o 2024-04-21T08:55:46.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:46 smithi129 ceph-mon[31272]: pgmap v323: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 230 KiB/s wr, 3.53k op/s 2024-04-21T08:55:46.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:46 smithi129 ceph-mon[31272]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:55:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:46 smithi057 ceph-mon[27968]: pgmap v323: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 230 KiB/s wr, 3.53k op/s 2024-04-21T08:55:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:46 smithi057 ceph-mon[27968]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:55:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:46 smithi136 ceph-mon[31791]: pgmap v323: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 230 KiB/s wr, 3.53k op/s 2024-04-21T08:55:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:46 smithi136 ceph-mon[31791]: osdmap e105: 12 total, 12 up, 12 in 2024-04-21T08:55:46.916 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_dynevent.o 2024-04-21T08:55:46.977 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs3client.o 2024-04-21T08:55:47.177 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs3proc.o 2024-04-21T08:55:47.655 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs3xdr.o 2024-04-21T08:55:47.853 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/cgroup/built-in.a 2024-04-21T08:55:47.872 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_probe.o 2024-04-21T08:55:47.986 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/trace_uprobe.o 2024-04-21T08:55:48.046 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs3acl.o 2024-04-21T08:55:48.144 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/events/core.o 2024-04-21T08:55:48.319 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/events/ring_buffer.o 2024-04-21T08:55:48.526 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/trace/rethook.o 2024-04-21T08:55:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:48 smithi129 ceph-mon[31272]: pgmap v325: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 509 KiB/s wr, 4.03k op/s 2024-04-21T08:55:48.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:48 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:48 smithi057 ceph-mon[27968]: pgmap v325: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 509 KiB/s wr, 4.03k op/s 2024-04-21T08:55:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:48 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:48.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:55:48.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:55:48.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:48 smithi136 ceph-mon[31791]: pgmap v325: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 509 KiB/s wr, 4.03k op/s 2024-04-21T08:55:48.864 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:48 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:55:48.882 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4proc.o 2024-04-21T08:55:49.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.376+0000 7f98cacf1640 1 -- 172.21.15.57:0/1165325117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c4110550 msgr2=0x7f98c4110930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.376+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1165325117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c4110550 0x7f98c4110930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009ca0 tx=0x7f98bc02c1c0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.380+0000 7f98cacf1640 1 -- 172.21.15.57:0/1165325117 shutdown_connections 2024-04-21T08:55:49.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.380+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1165325117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98c410eb40 0x7f98c411d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.380+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1165325117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c410e0c0 0x7f98c410e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.380+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1165325117 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c4110550 0x7f98c4110930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.380+0000 7f98cacf1640 1 -- 172.21.15.57:0/1165325117 >> 172.21.15.57:0/1165325117 conn(0x7f98c406ce60 msgr2=0x7f98c406d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:49.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.382+0000 7f98cacf1640 1 -- 172.21.15.57:0/1165325117 shutdown_connections 2024-04-21T08:55:49.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.382+0000 7f98cacf1640 1 -- 172.21.15.57:0/1165325117 wait complete. 2024-04-21T08:55:49.383 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 Processor -- start 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 -- start start 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 0x7f98c41a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98c410eb40 0x7f98c41a7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c4110550 0x7f98c41abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4120e90 con 0x7f98c410eb40 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4120d10 con 0x7f98c4110550 2024-04-21T08:55:49.384 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.383+0000 7f98cacf1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4121010 con 0x7f98c410e0c0 2024-04-21T08:55:49.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 0x7f98c41a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 0x7f98c41a73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:38724/0 (socket says 172.21.15.57:38724) 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1444829500 learned_addr learned my addr 172.21.15.57:0/1444829500 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98c410eb40 0x7f98c41a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c928e640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c4110550 0x7f98c41abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1444829500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c4110550 msgr2=0x7f98c41abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c4110550 0x7f98c41abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98c410eb40 msgr2=0x7f98c41a7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98c410eb40 0x7f98c41a7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.384+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1444829500 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c41ac3c0 con 0x7f98c410e0c0 2024-04-21T08:55:49.386 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.385+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 0x7f98c41a73d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f98bc02c750 tx=0x7f98bc003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:49.411 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.411+0000 7f98c1ffb640 1 -- 172.21.15.57:0/1444829500 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc039040 con 0x7f98c410e0c0 2024-04-21T08:55:49.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.411+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c41ac650 con 0x7f98c410e0c0 2024-04-21T08:55:49.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.411+0000 7f98c1ffb640 1 -- 172.21.15.57:0/1444829500 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98bc044070 con 0x7f98c410e0c0 2024-04-21T08:55:49.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.411+0000 7f98c1ffb640 1 -- 172.21.15.57:0/1444829500 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0327a0 con 0x7f98c410e0c0 2024-04-21T08:55:49.412 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.411+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c41b3f40 con 0x7f98c410e0c0 2024-04-21T08:55:49.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.416+0000 7f98c1ffb640 1 -- 172.21.15.57:0/1444829500 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f98bc032a00 con 0x7f98c410e0c0 2024-04-21T08:55:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.417+0000 7f98cd9b6740 1 Processor -- start 2024-04-21T08:55:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.417+0000 7f98cd9b6740 1 -- start start 2024-04-21T08:55:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.417+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb8700 0x55c124cb8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.417+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cc16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.418 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.417+0000 7f98c928e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb8700 0x55c124cb8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cc1c00 0x55c124cc3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cc16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c1ffb640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f98ac0a3ea0 0x7f98ac0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c928e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb8700 0x55c124cb8ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:38728/0 (socket says 172.21.15.57:38728) 2024-04-21T08:55:49.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cc16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35164/0 (socket says 172.21.15.57:35164) 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55c124c1d1e0 con 0x55c124cb9020 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c124bba390 con 0x55c124cc1c00 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55c124c191a0 con 0x55c124cb8700 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c928e640 1 -- 172.21.15.57:0/2635650264 learned_addr learned my addr 172.21.15.57:0/2635650264 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c1ffb640 1 -- 172.21.15.57:0/1444829500 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9459+0+0 (secure 0 0 0) 0x7f98bc0f3110 con 0x7f98c410e0c0 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c3fff640 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cc1c00 0x55c124cc3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c928e640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f98ac0a3ea0 0x7f98ac0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 -- 172.21.15.57:0/2635650264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb8700 msgr2=0x55c124cb8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb8700 0x55c124cb8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 -- 172.21.15.57:0/2635650264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cc1c00 msgr2=0x55c124cc3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cc1c00 0x55c124cc3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.421 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.418+0000 7f98c8a8d640 1 -- 172.21.15.57:0/2635650264 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1249f97d0 con 0x55c124cb9020 2024-04-21T08:55:49.421 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.419+0000 7f98c3fff640 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cc1c00 0x55c124cc3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:49.421 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.419+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cc16c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0f05c0 tx=0x7f98bc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b3df00492077f server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:49.423 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.423+0000 7f98c928e640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f98ac0a3ea0 0x7f98ac0a6360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f98b4014950 tx=0x7f98b4004730 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.424+0000 7f98aaffd640 1 -- 172.21.15.57:0/2635650264 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0bf2a0 con 0x55c124cb9020 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.424+0000 7f98aaffd640 1 -- 172.21.15.57:0/2635650264 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98bc0333a0 con 0x55c124cb9020 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.424+0000 7f98aaffd640 1 -- 172.21.15.57:0/2635650264 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0bee30 con 0x55c124cb9020 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 -- 172.21.15.57:0/2635650264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 msgr2=0x55c124cc16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cc16c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0f05c0 tx=0x7f98bc0095c0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 -- 172.21.15.57:0/2635650264 shutdown_connections 2024-04-21T08:55:49.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cc1c00 0x55c124cc3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cc16c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/2635650264 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb8700 0x55c124cb8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.425+0000 7f98cd9b6740 1 -- 172.21.15.57:0/2635650264 >> 172.21.15.57:0/2635650264 conn(0x55c124cb2e30 msgr2=0x55c124cb3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:49.428 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4xdr.o 2024-04-21T08:55:49.429 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.429+0000 7f98cd9b6740 1 -- 172.21.15.57:0/2635650264 shutdown_connections 2024-04-21T08:55:49.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.429+0000 7f98cd9b6740 1 -- 172.21.15.57:0/2635650264 wait complete. 2024-04-21T08:55:49.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.429+0000 7f98cd9b6740 1 -- start start 2024-04-21T08:55:49.430 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.429+0000 7f98cd9b6740 1 Processor -- start 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124cbf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55c124cc47b0 con 0x55c124cc1c00 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c124bd62f0 con 0x55c124cb8700 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55c124cc4930 con 0x55c124cb9020 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.430+0000 7f98cd9b6740 10 client.0 _ll_register_callbacks cb 0x55c124be3580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.431+0000 7f98c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124cbf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.431+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.432+0000 7f98c928e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.432+0000 7f98c928e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:35178/0 (socket says 172.21.15.57:35178) 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.432+0000 7f98c928e640 1 -- 172.21.15.57:0/1409454556 learned_addr learned my addr 172.21.15.57:0/1409454556 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.433+0000 7f98c3fff640 1 -- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 msgr2=0x55c124cbec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.433+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.433+0000 7f98c3fff640 1 -- 172.21.15.57:0/1409454556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 msgr2=0x55c124cbf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.433+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.434 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.433+0000 7f98c3fff640 1 -- 172.21.15.57:0/1409454556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c12492f3f0 con 0x55c124cb9020 2024-04-21T08:55:49.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.433+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124cbf1d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98c4072f70 tx=0x7f98b80117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a584c700ca397e9d server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.435+0000 7f98a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:55:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.435+0000 7f98a3fff640 1 -- 172.21.15.57:0/1409454556 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8012650 con 0x55c124cb9020 2024-04-21T08:55:49.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.436+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98bc0333a0 con 0x55c124cb9020 2024-04-21T08:55:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.436+0000 7f98a3fff640 1 -- 172.21.15.57:0/1409454556 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9459+0+0 (secure 0 0 0) 0x7f98b80127f0 con 0x55c124cb9020 2024-04-21T08:55:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.436+0000 7f98a3fff640 1 -- 172.21.15.57:0/1409454556 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f98b80129f0 con 0x55c124cb9020 2024-04-21T08:55:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.436+0000 7f98cd9b6740 10 client.26587 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:49.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.436+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c1248ef0f0 con 0x55c124cb9020 2024-04-21T08:55:49.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98a3fff640 1 -- 172.21.15.57:0/1409454556 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98bc0333a0 con 0x55c124cb9020 2024-04-21T08:55:49.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98a3fff640 1 -- 172.21.15.57:0/1409454556 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f98b801a3d0 con 0x55c124cb9020 2024-04-21T08:55:49.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98cd9b6740 10 client.26587 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:49.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98cd9b6740 10 client.26587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:49.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98cd9b6740 20 client.26587 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:49.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124ce29f0 0x55c124ce4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98cd9b6740 4 client.26587 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:49.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.439+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c124c2ac00 con 0x55c124ce29f0 2024-04-21T08:55:49.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.440+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124ce29f0 0x55c124ce4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.441 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.441+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124ce29f0 0x55c124ce4e10 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcc0e7cc46647c03 server_cookie=cd78479071670485 in_seq=0 out_seq=0 2024-04-21T08:55:49.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.444+0000 7f98a3fff640 10 client.26587 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:49.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.444+0000 7f98a3fff640 1 -- 172.21.15.57:0/1409454556 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c124c2ac00 con 0x55c124ce29f0 2024-04-21T08:55:49.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.444+0000 7f98a3fff640 10 client.26587 handle_command_reply: tid=0 2024-04-21T08:55:49.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.444+0000 7f98cd9b6740 1 client.26587 shutdown 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.444+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 msgr2=0x55c124cbf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.444+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124cbf1d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f98c4072f70 tx=0x7f98b80117b0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 shutdown_connections 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124ce29f0 0x55c124ce4e10 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124cbf1d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1409454556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98a3fff640 0 client.26587 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 >> 172.21.15.57:0/1409454556 conn(0x55c124cb2e30 msgr2=0x55c124cb4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 shutdown_connections 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1409454556 wait complete. 2024-04-21T08:55:49.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.445+0000 7f98cd9b6740 20 client.26587 trim_cache size 0 max 16384 2024-04-21T08:55:49.514 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.513+0000 7f98cd9b6740 1 Processor -- start 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 -- start start 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124ccd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41a8720 con 0x55c124cc1c00 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41b61a0 con 0x55c124cb8700 2024-04-21T08:55:49.515 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.514+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41b4110 con 0x55c124cb9020 2024-04-21T08:55:49.516 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124ccd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124ccd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:38744/0 (socket says 172.21.15.57:38744) 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 -- 172.21.15.57:0/1275560102 learned_addr learned my addr 172.21.15.57:0/1275560102 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c928e640 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 -- 172.21.15.57:0/1275560102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 msgr2=0x55c124cbdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 -- 172.21.15.57:0/1275560102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 msgr2=0x55c124cbf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.515+0000 7f98c3fff640 1 -- 172.21.15.57:0/1275560102 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c124cb9460 con 0x55c124cb9020 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.516+0000 7f98c928e640 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:49.517 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.516+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124ccd5e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f98b801a7f0 tx=0x7f98b800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad8e3929f25fc35b server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:49.524 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.523+0000 7f98aa7fc640 1 -- 172.21.15.57:0/1275560102 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b802d070 con 0x55c124cb9020 2024-04-21T08:55:49.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.523+0000 7f98aa7fc640 1 -- 172.21.15.57:0/1275560102 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b801a3d0 con 0x55c124cb9020 2024-04-21T08:55:49.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.523+0000 7f98aa7fc640 1 -- 172.21.15.57:0/1275560102 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8012d90 con 0x55c124cb9020 2024-04-21T08:55:49.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.525+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1275560102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 msgr2=0x55c124ccd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.525 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.525+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124ccd5e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f98b801a7f0 tx=0x7f98b800b910 comp rx=0 tx=0).stop 2024-04-21T08:55:49.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.528+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1275560102 shutdown_connections 2024-04-21T08:55:49.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.528+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cc1c00 0x55c124cbf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.528+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cb9020 0x55c124ccd5e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.528+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1275560102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.528+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1275560102 >> 172.21.15.57:0/1275560102 conn(0x55c124cb2e30 msgr2=0x55c124cb49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:49.529 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.528+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1275560102 shutdown_connections 2024-04-21T08:55:49.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.529+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1275560102 wait complete. 2024-04-21T08:55:49.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.529+0000 7f98cd9b6740 1 -- start start 2024-04-21T08:55:49.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.529+0000 7f98cd9b6740 1 Processor -- start 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cbc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cc1c00 0x55c124d47f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41b4660 con 0x55c124cb9020 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41b5ad0 con 0x55c124cb8700 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f98c41b4d30 con 0x55c124cc1c00 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98cd9b6740 10 client.0 _ll_register_callbacks cb 0x7f98b800b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.531 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98c8a8d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:52384/0 (socket says 172.21.15.57:52384) 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1353323392 learned_addr learned my addr 172.21.15.57:0/1353323392 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cbc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.530+0000 7f98c928e640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cc1c00 0x55c124d47f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1353323392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cc1c00 msgr2=0x55c124d47f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cc1c00 0x55c124d47f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1353323392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 msgr2=0x55c124cbc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:49.532 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cbc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c8a8d640 1 -- 172.21.15.57:0/1353323392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c124d484b0 con 0x55c124cb8700 2024-04-21T08:55:49.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cbc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:49.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c928e640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cc1c00 0x55c124d47f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:55:49.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.531+0000 7f98c8a8d640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbc080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0f0670 tx=0x7f98bc004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9514093ed8884dac server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:49.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:49.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98a3fff640 1 -- 172.21.15.57:0/1353323392 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0bf010 con 0x55c124cb8700 2024-04-21T08:55:49.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98b801a3d0 con 0x55c124cb8700 2024-04-21T08:55:49.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98a3fff640 1 -- 172.21.15.57:0/1353323392 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9459+0+0 (secure 0 0 0) 0x7f98bc0f06a0 con 0x55c124cb8700 2024-04-21T08:55:49.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98a3fff640 1 -- 172.21.15.57:0/1353323392 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f98bc0bf2a0 con 0x55c124cb8700 2024-04-21T08:55:49.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98cd9b6740 10 client.26717 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:49.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.540+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55c124d49770 con 0x55c124cb8700 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98a3fff640 1 -- 172.21.15.57:0/1353323392 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b801a3d0 con 0x55c124cb8700 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98a3fff640 1 -- 172.21.15.57:0/1353323392 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f98bc03c030 con 0x55c124cb8700 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98cd9b6740 10 client.26717 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98cd9b6740 10 client.26717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98cd9b6740 20 client.26717 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124d49ef0 0x55c124d4c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:49.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98cd9b6740 4 client.26717 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:55:49.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c124c24a70 con 0x55c124d49ef0 2024-04-21T08:55:49.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.541+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124d49ef0 0x55c124d4c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:49.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.542+0000 7f98c3fff640 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124d49ef0 0x55c124d4c310 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eff168fde4a93d9 server_cookie=e9556d0a2981645e in_seq=0 out_seq=0 2024-04-21T08:55:49.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.542+0000 7f98a3fff640 10 client.26717 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:49.544 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.542+0000 7f98a3fff640 1 -- 172.21.15.57:0/1353323392 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x55c124c24a70 con 0x55c124d49ef0 2024-04-21T08:55:49.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.542+0000 7f98a3fff640 10 client.26717 handle_command_reply: tid=0 2024-04-21T08:55:49.545 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.542+0000 7f98cd9b6740 1 client.26717 shutdown 2024-04-21T08:55:49.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 msgr2=0x55c124cbc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbc080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0f0670 tx=0x7f98bc004740 comp rx=0 tx=0).stop 2024-04-21T08:55:49.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 shutdown_connections 2024-04-21T08:55:49.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55c124d49ef0 0x55c124d4c310 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55c124cc1c00 0x55c124d47f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55c124cb9020 0x55c124cbc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 --2- 172.21.15.57:0/1353323392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c124cb8700 0x55c124cbc080 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 >> 172.21.15.57:0/1353323392 conn(0x55c124cb2e30 msgr2=0x55c124cb4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:49.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98a3fff640 0 client.26717 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:49.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 shutdown_connections 2024-04-21T08:55:49.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.544+0000 7f98cd9b6740 1 -- 172.21.15.57:0/1353323392 wait complete. 2024-04-21T08:55:49.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.545+0000 7f98cd9b6740 20 client.26717 trim_cache size 0 max 16384 2024-04-21T08:55:49.549 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:55:49.549 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "scrub active (14206 inodes in the stack)", 2024-04-21T08:55:49.549 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": { 2024-04-21T08:55:49.549 INFO:teuthology.orchestra.run.smithi057.stdout: "29ebec1e-383b-43cc-b1b5-8bf451215c20": { 2024-04-21T08:55:49.550 INFO:teuthology.orchestra.run.smithi057.stdout: "path": "/", 2024-04-21T08:55:49.550 INFO:teuthology.orchestra.run.smithi057.stdout: "tag": "29ebec1e-383b-43cc-b1b5-8bf451215c20", 2024-04-21T08:55:49.550 INFO:teuthology.orchestra.run.smithi057.stdout: "options": "recursive,force" 2024-04-21T08:55:49.550 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:55:49.550 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:55:49.551 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.555+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f98ac0a3ea0 msgr2=0x7f98ac0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.555+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f98ac0a3ea0 0x7f98ac0a6360 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f98b4014950 tx=0x7f98b4004730 comp rx=0 tx=0).stop 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.555+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 msgr2=0x7f98c41a73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.555+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 0x7f98c41a73d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f98bc02c750 tx=0x7f98bc003ba0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 shutdown_connections 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f98ac0a3ea0 0x7f98ac0a6360 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98c4110550 0x7f98c41abc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98c410eb40 0x7f98c41a7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 --2- 172.21.15.57:0/1444829500 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98c410e0c0 0x7f98c41a73d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:49.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 >> 172.21.15.57:0/1444829500 conn(0x7f98c406ce60 msgr2=0x7f98c4071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:49.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 shutdown_connections 2024-04-21T08:55:49.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:55:49.556+0000 7f98cacf1640 1 -- 172.21.15.57:0/1444829500 wait complete. 2024-04-21T08:55:49.588 INFO:tasks.cephfs.filesystem:scrub status for tag:29ebec1e-383b-43cc-b1b5-8bf451215c20 - {'path': '/', 'tag': '29ebec1e-383b-43cc-b1b5-8bf451215c20', 'options': 'recursive,force'} 2024-04-21T08:55:49.601 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/tracefs/inode.o 2024-04-21T08:55:49.627 INFO:tasks.workunit.client.0.smithi057.stdout: CC [M] fs/efivarfs/inode.o 2024-04-21T08:55:49.698 INFO:tasks.workunit.client.0.smithi057.stdout: CC [M] fs/efivarfs/file.o 2024-04-21T08:55:49.975 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/trace/built-in.a 2024-04-21T08:55:50.053 INFO:tasks.workunit.client.0.smithi057.stdout: CC [M] fs/efivarfs/super.o 2024-04-21T08:55:50.077 INFO:tasks.workunit.client.0.smithi057.stdout: CC [M] fs/efivarfs/vars.o 2024-04-21T08:55:50.351 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/fork.o 2024-04-21T08:55:50.390 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4state.o 2024-04-21T08:55:50.399 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/tracefs/built-in.a 2024-04-21T08:55:50.485 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/open.o 2024-04-21T08:55:50.908 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/read_write.o 2024-04-21T08:55:50.965 INFO:tasks.workunit.client.0.smithi057.stdout: LD [M] fs/efivarfs/efivarfs.o 2024-04-21T08:55:51.025 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/file_table.o 2024-04-21T08:55:51.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: pgmap v326: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 4.01k op/s 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:50 smithi129 ceph-mon[31272]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T08:55:51.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: pgmap v326: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 4.01k op/s 2024-04-21T08:55:51.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:50 smithi057 ceph-mon[27968]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T08:55:51.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: pgmap v326: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.1 MiB/s wr, 4.01k op/s 2024-04-21T08:55:51.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:51.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:51.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:50 smithi136 ceph-mon[31791]: osdmap e106: 12 total, 12 up, 12 in 2024-04-21T08:55:52.035 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4renewd.o 2024-04-21T08:55:52.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:51 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-21T08:55:52.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:51 smithi129 ceph-mon[31272]: Unable to set osd_memory_target on smithi136 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-21T08:55:52.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:51 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:51 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:51 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:51 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:51 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-21T08:55:52.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:51 smithi057 ceph-mon[27968]: Unable to set osd_memory_target on smithi136 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-21T08:55:52.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:51 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:51 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:51 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:51 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:51 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-21T08:55:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:51 smithi136 ceph-mon[31791]: Unable to set osd_memory_target on smithi136 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-21T08:55:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:51 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:51 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:51 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:51 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:52.171 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4super.o 2024-04-21T08:55:52.534 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4file.o 2024-04-21T08:55:52.664 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/super.o 2024-04-21T08:55:52.693 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/exec_domain.o 2024-04-21T08:55:52.802 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/panic.o 2024-04-21T08:55:53.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:52 smithi129 ceph-mon[31272]: pgmap v328: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 4.33k op/s 2024-04-21T08:55:53.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:52 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-04-21T08:55:53.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:52 smithi129 ceph-mon[31272]: Unable to set osd_memory_target on smithi129 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-21T08:55:53.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:52 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:53.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:52 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:53.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:52 smithi057 ceph-mon[27968]: pgmap v328: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 4.33k op/s 2024-04-21T08:55:53.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:52 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-04-21T08:55:53.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:52 smithi057 ceph-mon[27968]: Unable to set osd_memory_target on smithi129 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-21T08:55:53.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:52 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:53.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:52 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:53.106 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/events/callchain.o 2024-04-21T08:55:53.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:52 smithi136 ceph-mon[31791]: pgmap v328: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.8 MiB/s wr, 4.33k op/s 2024-04-21T08:55:53.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:52 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi129 to 176.2M 2024-04-21T08:55:53.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:52 smithi136 ceph-mon[31791]: Unable to set osd_memory_target on smithi129 to 184783872: error parsing value: Value '184783872' is below minimum 939524096 2024-04-21T08:55:53.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:52 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:53.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:52 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:53.327 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/delegation.o 2024-04-21T08:55:53.331 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4idmap.o 2024-04-21T08:55:53.581 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/char_dev.o 2024-04-21T08:55:53.797 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/stat.o 2024-04-21T08:55:53.918 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/exec.o 2024-04-21T08:55:53.922 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/events/hw_breakpoint.o 2024-04-21T08:55:54.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:53 smithi129 ceph-mon[31272]: pgmap v329: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 3.34k op/s 2024-04-21T08:55:54.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:53 smithi057 ceph-mon[27968]: pgmap v329: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 3.34k op/s 2024-04-21T08:55:54.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:53 smithi136 ceph-mon[31791]: pgmap v329: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.7 MiB/s wr, 3.34k op/s 2024-04-21T08:55:54.164 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/events/uprobes.o 2024-04-21T08:55:54.307 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cpu.o 2024-04-21T08:55:54.539 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/exit.o 2024-04-21T08:55:54.797 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/softirq.o 2024-04-21T08:55:54.844 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/resource.o 2024-04-21T08:55:54.925 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sysctl.o 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: Adjusting osd_memory_target on smithi057 to 177.2M 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: Unable to set osd_memory_target on smithi057 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:55.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:54 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:55.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:55.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: Adjusting osd_memory_target on smithi057 to 177.2M 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: Unable to set osd_memory_target on smithi057 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:55.090 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:54 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:55.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: scrub summary: idle+waiting paths [/] 2024-04-21T08:55:55.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: Adjusting osd_memory_target on smithi057 to 177.2M 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: Unable to set osd_memory_target on smithi057 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:55:55.136 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:54 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:55:55.192 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/callback.o 2024-04-21T08:55:55.358 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/pipe.o 2024-04-21T08:55:55.553 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/events/built-in.a 2024-04-21T08:55:55.637 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/capability.o 2024-04-21T08:55:55.778 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/ptrace.o 2024-04-21T08:55:55.819 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/user.o 2024-04-21T08:55:55.956 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/signal.o 2024-04-21T08:55:55.996 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sys.o 2024-04-21T08:55:56.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:55 smithi129 ceph-mon[31272]: pgmap v330: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.5 MiB/s wr, 4.35k op/s 2024-04-21T08:55:56.083 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/callback_xdr.o 2024-04-21T08:55:56.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:55 smithi057 ceph-mon[27968]: pgmap v330: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.5 MiB/s wr, 4.35k op/s 2024-04-21T08:55:56.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:55 smithi136 ceph-mon[31791]: pgmap v330: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.5 MiB/s wr, 4.35k op/s 2024-04-21T08:55:56.186 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/umh.o 2024-04-21T08:55:56.339 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/workqueue.o 2024-04-21T08:55:56.369 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/pid.o 2024-04-21T08:55:56.457 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/task_work.o 2024-04-21T08:55:56.787 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/namei.o 2024-04-21T08:55:56.920 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/extable.o 2024-04-21T08:55:57.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:56 smithi129 ceph-mon[31272]: scrub summary: idle 2024-04-21T08:55:57.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:56 smithi057 ceph-mon[27968]: scrub summary: idle 2024-04-21T08:55:57.110 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/params.o 2024-04-21T08:55:57.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:56 smithi136 ceph-mon[31791]: scrub summary: idle 2024-04-21T08:55:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2442634930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f010b210 msgr2=0x7fc9f0113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2442634930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f010b210 0x7fc9f0113960 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009900 tx=0x7fc9e002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2442634930 shutdown_connections 2024-04-21T08:55:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2442634930 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9f0113ea0 0x7fc9f0116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2442634930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f010b210 0x7fc9f0113960 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2442634930 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2442634930 >> 172.21.15.57:0/2442634930 conn(0x7fc9f006d020 msgr2=0x7fc9f006d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2442634930 shutdown_connections 2024-04-21T08:55:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.215+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2442634930 wait complete. 2024-04-21T08:55:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.216+0000 7fc9f6e1e640 1 Processor -- start 2024-04-21T08:55:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.216+0000 7fc9f6e1e640 1 -- start start 2024-04-21T08:55:57.217 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f6e1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f6e1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f0113ea0 0x7fc9f010bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f6e1e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9f010c450 0x7fc9f01af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f6e1e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f0119930 con 0x7fc9f0113ea0 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f6e1e640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f01197b0 con 0x7fc9f010a8f0 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f6e1e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f0119ab0 con 0x7fc9f010c450 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f0113ea0 0x7fc9f010bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:36240/0 (socket says 172.21.15.57:36240) 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 -- 172.21.15.57:0/2572271544 learned_addr learned my addr 172.21.15.57:0/2572271544 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9f010c450 0x7fc9f01af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 -- 172.21.15.57:0/2572271544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9f010c450 msgr2=0x7fc9f01af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9f010c450 0x7fc9f01af310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 -- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f0113ea0 msgr2=0x7fc9f010bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f0113ea0 0x7fc9f010bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9f4bba640 1 -- 172.21.15.57:0/2572271544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f01afa10 con 0x7fc9f010a8f0 2024-04-21T08:55:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.217+0000 7fc9effff640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f0113ea0 0x7fc9f010bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.218+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010b9d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800ebc0 tx=0x7fc9e800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.234+0000 7fc9edffb640 1 -- 172.21.15.57:0/2572271544 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8010630 con 0x7fc9f010a8f0 2024-04-21T08:55:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.234+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f01afd00 con 0x7fc9f010a8f0 2024-04-21T08:55:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.234+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f01b01d0 con 0x7fc9f010a8f0 2024-04-21T08:55:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.234+0000 7fc9edffb640 1 -- 172.21.15.57:0/2572271544 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e8010c80 con 0x7fc9f010a8f0 2024-04-21T08:55:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.234+0000 7fc9edffb640 1 -- 172.21.15.57:0/2572271544 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8018b60 con 0x7fc9f010a8f0 2024-04-21T08:55:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.238+0000 7fc9edffb640 1 -- 172.21.15.57:0/2572271544 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc9e8018d00 con 0x7fc9f010a8f0 2024-04-21T08:55:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 Processor -- start 2024-04-21T08:55:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 -- start start 2024-04-21T08:55:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfbb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfbbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ef4df208d0 con 0x55ef4dfbc8d0 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ef4debde80 con 0x55ef4dfbbb60 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ef4df1c890 con 0x55ef4dfc50e0 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfbbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfbbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:36246/0 (socket says 172.21.15.57:36246) 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f4bba640 1 -- 172.21.15.57:0/3196232981 learned_addr learned my addr 172.21.15.57:0/3196232981 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9effff640 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.239+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfbb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f4bba640 1 -- 172.21.15.57:0/3196232981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 msgr2=0x55ef4dfc74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f4bba640 1 -- 172.21.15.57:0/3196232981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 msgr2=0x55ef4dfbb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfbb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9edffb640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc9d80a3fb0 0x7fc9d80a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f4bba640 1 -- 172.21.15.57:0/3196232981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4dedf320 con 0x55ef4dfbbb60 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfbb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9edffb640 1 -- 172.21.15.57:0/2572271544 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9431+0+0 (secure 0 0 0) 0x7fc9e80d1430 con 0x7fc9f010a8f0 2024-04-21T08:55:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.240+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfbbfe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80cedf0 tx=0x7fc9e80995a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f946a2e233c8cece server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.241+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc9d80a3fb0 0x7fc9d80a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.241+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc9d80a3fb0 0x7fc9d80a6470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009ed0 tx=0x7fc9e4005f90 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.257 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.257+0000 7fc9d6ffd640 1 -- 172.21.15.57:0/3196232981 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8026030 con 0x55ef4dfbbb60 2024-04-21T08:55:57.257 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.257+0000 7fc9d6ffd640 1 -- 172.21.15.57:0/3196232981 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e8010e30 con 0x55ef4dfbbb60 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3196232981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 msgr2=0x55ef4dfbbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfbbfe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80cedf0 tx=0x7fc9e80995a0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3196232981 shutdown_connections 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfbbfe0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3196232981 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfbb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.258 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.258+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3196232981 >> 172.21.15.57:0/3196232981 conn(0x55ef4df82c90 msgr2=0x55ef4dfb6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.260+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3196232981 shutdown_connections 2024-04-21T08:55:57.259 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.260+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3196232981 wait complete. 2024-04-21T08:55:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.260+0000 7fc9f9a05740 1 -- start start 2024-04-21T08:55:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.260+0000 7fc9f9a05740 1 Processor -- start 2024-04-21T08:55:57.260 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55ef4dfc7e00 con 0x55ef4dfc50e0 2024-04-21T08:55:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ef4dea8eb0 con 0x55ef4dfbc8d0 2024-04-21T08:55:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ef4dfc7f80 con 0x55ef4dfbbb60 2024-04-21T08:55:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f9a05740 10 client.0 _ll_register_callbacks cb 0x55ef4ded5b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9effff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:39832/0 (socket says 172.21.15.57:39832) 2024-04-21T08:55:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f4bba640 1 -- 172.21.15.57:0/467203726 learned_addr learned my addr 172.21.15.57:0/467203726 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9effff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:36250/0 (socket says 172.21.15.57:36250) 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.261+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f4bba640 1 -- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 msgr2=0x55ef4dfc1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f4bba640 1 -- 172.21.15.57:0/467203726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfc50e0 msgr2=0x55ef4dfc2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f4bba640 1 -- 172.21.15.57:0/467203726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4dc33210 con 0x55ef4dfbbb60 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9effff640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.262+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8010c50 tx=0x7fc9e8009200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2914b73579f62c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.270+0000 7fc9cffff640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:55:57.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.270+0000 7fc9cffff640 1 -- 172.21.15.57:0/467203726 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e809cba0 con 0x55ef4dfbbb60 2024-04-21T08:55:57.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.270+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9e8010e30 con 0x55ef4dfbbb60 2024-04-21T08:55:57.270 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.270+0000 7fc9cffff640 1 -- 172.21.15.57:0/467203726 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9431+0+0 (secure 0 0 0) 0x7fc9e80d4bf0 con 0x55ef4dfbbb60 2024-04-21T08:55:57.271 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.272+0000 7fc9cffff640 1 -- 172.21.15.57:0/467203726 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc9e8018420 con 0x55ef4dfbbb60 2024-04-21T08:55:57.272 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.272+0000 7fc9f9a05740 10 client.26602 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:57.272 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.272+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ef4dcff620 con 0x55ef4dfbbb60 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9cffff640 1 -- 172.21.15.57:0/467203726 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e8010e30 con 0x55ef4dfbbb60 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9cffff640 1 -- 172.21.15.57:0/467203726 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc9e80998b0 con 0x55ef4dfbbb60 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9f9a05740 10 client.26602 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9f9a05740 10 client.26602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9f9a05740 20 client.26602 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfdd6c0 0x55ef4dfdfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9f9a05740 4 client.26602 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef4df34880 con 0x55ef4dfdd6c0 2024-04-21T08:55:57.275 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.275+0000 7fc9effff640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfdd6c0 0x55ef4dfdfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.276+0000 7fc9cffff640 10 client.26602 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:57.276 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.276+0000 7fc9effff640 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfdd6c0 0x55ef4dfdfab0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14c54152f172b1f3 server_cookie=2513412791e8b53f in_seq=0 out_seq=0 2024-04-21T08:55:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9cffff640 1 -- 172.21.15.57:0/467203726 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ef4df34880 con 0x55ef4dfdd6c0 2024-04-21T08:55:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9cffff640 10 client.26602 handle_command_reply: tid=0 2024-04-21T08:55:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 client.26602 shutdown 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 msgr2=0x55ef4dfc4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8010c50 tx=0x7fc9e8009200 comp rx=0 tx=0).stop 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 shutdown_connections 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfdd6c0 0x55ef4dfdfab0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfc2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/467203726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4950 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 >> 172.21.15.57:0/467203726 conn(0x55ef4df82c90 msgr2=0x55ef4dfb85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9cffff640 0 client.26602 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 shutdown_connections 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 1 -- 172.21.15.57:0/467203726 wait complete. 2024-04-21T08:55:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.277+0000 7fc9f9a05740 20 client.26602 trim_cache size 0 max 16384 2024-04-21T08:55:57.310 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/callback_proc.o 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.312+0000 7fc9f9a05740 1 Processor -- start 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.312+0000 7fc9f9a05740 1 -- start start 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfe26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc00dfe0 con 0x55ef4dfbc8d0 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc010840 con 0x55ef4dfbbb60 2024-04-21T08:55:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc00e630 con 0x55ef4dfc50e0 2024-04-21T08:55:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f4bba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:36256/0 (socket says 172.21.15.57:36256) 2024-04-21T08:55:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f4bba640 1 -- 172.21.15.57:0/197742091 learned_addr learned my addr 172.21.15.57:0/197742091 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:38316/0 (socket says 172.21.15.57:38316) 2024-04-21T08:55:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.313+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfe26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9f4bba640 1 -- 172.21.15.57:0/197742091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 msgr2=0x55ef4dfe26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfe26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9f4bba640 1 -- 172.21.15.57:0/197742091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 msgr2=0x55ef4dfc4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9f4bba640 1 -- 172.21.15.57:0/197742091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4dfb1f30 con 0x55ef4dfbbb60 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9effff640 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4410 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8018710 tx=0x7fc9e800eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60367db4b2155e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.314+0000 7fc9d67fc640 1 -- 172.21.15.57:0/197742091 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8010e30 con 0x55ef4dfbbb60 2024-04-21T08:55:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.315+0000 7fc9d67fc640 1 -- 172.21.15.57:0/197742091 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e8010a90 con 0x55ef4dfbbb60 2024-04-21T08:55:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.315+0000 7fc9d67fc640 1 -- 172.21.15.57:0/197742091 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80998b0 con 0x55ef4dfbbb60 2024-04-21T08:55:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.316+0000 7fc9f9a05740 1 -- 172.21.15.57:0/197742091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 msgr2=0x55ef4dfc4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.316+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4410 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8018710 tx=0x7fc9e800eb10 comp rx=0 tx=0).stop 2024-04-21T08:55:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 -- 172.21.15.57:0/197742091 shutdown_connections 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfc50e0 0x55ef4dfe26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfc4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/197742091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfc4410 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 -- 172.21.15.57:0/197742091 >> 172.21.15.57:0/197742091 conn(0x55ef4df82c90 msgr2=0x55ef4dfc1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 -- 172.21.15.57:0/197742091 shutdown_connections 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 -- 172.21.15.57:0/197742091 wait complete. 2024-04-21T08:55:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 -- start start 2024-04-21T08:55:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.333+0000 7fc9f9a05740 1 Processor -- start 2024-04-21T08:55:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfdfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfdffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfc50e0 0x55ef4e03b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc00ed00 con 0x55ef4dfbbb60 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc010170 con 0x55ef4dfc50e0 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc00f3d0 con 0x55ef4dfbc8d0 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.334+0000 7fc9f9a05740 10 client.0 _ll_register_callbacks cb 0x7fc9e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfdffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfdffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:39850/0 (socket says 172.21.15.57:39850) 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9effff640 1 -- 172.21.15.57:0/3968426631 learned_addr learned my addr 172.21.15.57:0/3968426631 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfdfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfc50e0 0x55ef4e03b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f53bb640 1 -- 172.21.15.57:0/3968426631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbc8d0 msgr2=0x55ef4dfdffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfdffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f53bb640 1 -- 172.21.15.57:0/3968426631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbbb60 msgr2=0x55ef4dfdfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfdfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f53bb640 1 -- 172.21.15.57:0/3968426631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4e03be90 con 0x55ef4dfc50e0 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.335+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfdfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:55:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.336+0000 7fc9f53bb640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfc50e0 0x55ef4e03b8d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4012cb0 tx=0x7fc9e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f0380e021f73865 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:55:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.344+0000 7fc9cffff640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:55:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.344+0000 7fc9cffff640 1 -- 172.21.15.57:0/3968426631 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e402f060 con 0x55ef4dfc50e0 2024-04-21T08:55:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.344+0000 7fc9cffff640 1 -- 172.21.15.57:0/3968426631 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9431+0+0 (secure 0 0 0) 0x7fc9e4003d00 con 0x55ef4dfc50e0 2024-04-21T08:55:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.344+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9e8010a90 con 0x55ef4dfc50e0 2024-04-21T08:55:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.356+0000 7fc9cffff640 1 -- 172.21.15.57:0/3968426631 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc9e4036050 con 0x55ef4dfc50e0 2024-04-21T08:55:57.355 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.356+0000 7fc9f9a05740 10 client.26735 fetch_fsmap learned FSMap version 21 2024-04-21T08:55:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.356+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55ef4dfc8bf0 con 0x55ef4dfc50e0 2024-04-21T08:55:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.356+0000 7fc9cffff640 1 -- 172.21.15.57:0/3968426631 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9e8010a90 con 0x55ef4dfc50e0 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9cffff640 1 -- 172.21.15.57:0/3968426631 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7fc9e401d410 con 0x55ef4dfc50e0 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f9a05740 10 client.26735 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f9a05740 10 client.26735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f9a05740 20 client.26735 populate_metadata read hostname 'smithi057' 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfc9320 0x55ef4dfcb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f9a05740 4 client.26735 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ef4dec03d0 con 0x55ef4dfc9320 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.357+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfc9320 0x55ef4dfcb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:55:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.358+0000 7fc9cffff640 10 client.26735 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:57.358+0000 7fc9f4bba640 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfc9320 0x55ef4dfcb740 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d07ff002bcd00e94 server_cookie=3d1095c923722aaf in_seq=0 out_seq=0 2024-04-21T08:55:57.797 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/kthread.o 2024-04-21T08:55:57.830 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/sys_ni.o 2024-04-21T08:55:58.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:55:57 smithi129 ceph-mon[31272]: pgmap v331: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.3 MiB/s wr, 4.04k op/s 2024-04-21T08:55:58.092 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:55:57 smithi057 ceph-mon[27968]: pgmap v331: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.3 MiB/s wr, 4.04k op/s 2024-04-21T08:55:58.111 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/nsproxy.o 2024-04-21T08:55:58.135 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:55:57 smithi136 ceph-mon[31791]: pgmap v331: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 1.3 MiB/s wr, 4.04k op/s 2024-04-21T08:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9cffff640 1 -- 172.21.15.57:0/3968426631 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ef4dec03d0 con 0x55ef4dfc9320 2024-04-21T08:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9cffff640 10 client.26735 handle_command_reply: tid=0 2024-04-21T08:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 client.26735 shutdown 2024-04-21T08:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfc50e0 msgr2=0x55ef4e03b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfc50e0 0x55ef4e03b8d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4012cb0 tx=0x7fc9e4004730 comp rx=0 tx=0).stop 2024-04-21T08:55:58.233 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 shutdown_connections 2024-04-21T08:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55ef4dfc9320 0x55ef4dfcb740 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ef4dfc50e0 0x55ef4e03b8d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9cffff640 0 client.26735 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ef4dfbc8d0 0x55ef4dfdffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 --2- 172.21.15.57:0/3968426631 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55ef4dfbbb60 0x55ef4dfdfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.233+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 >> 172.21.15.57:0/3968426631 conn(0x55ef4df82c90 msgr2=0x55ef4dfbcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.234+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 shutdown_connections 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.234+0000 7fc9f9a05740 1 -- 172.21.15.57:0/3968426631 wait complete. 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.234+0000 7fc9f9a05740 20 client.26735 trim_cache size 0 max 16384 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:55:58.235 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/notifier.o 2024-04-21T08:55:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc9d80a3fb0 msgr2=0x7fc9d80a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:58.247 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc9d80a3fb0 0x7fc9d80a6470 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009ed0 tx=0x7fc9e4005f90 comp rx=0 tx=0).stop 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 msgr2=0x7fc9f010b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010b9d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800ebc0 tx=0x7fc9e800c6a0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 shutdown_connections 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc9d80a3fb0 0x7fc9d80a6470 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9f010c450 0x7fc9f01af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc9f0113ea0 0x7fc9f010bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 --2- 172.21.15.57:0/2572271544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9f010a8f0 0x7fc9f010b9d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 >> 172.21.15.57:0/2572271544 conn(0x7fc9f006d020 msgr2=0x7fc9f0109980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 shutdown_connections 2024-04-21T08:55:58.248 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:55:58.247+0000 7fc9f6e1e640 1 -- 172.21.15.57:0/2572271544 wait complete. 2024-04-21T08:55:58.561 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/ksysfs.o 2024-04-21T08:55:58.570 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/cred.o 2024-04-21T08:55:58.827 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4namespace.o 2024-04-21T08:55:58.975 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fcntl.o 2024-04-21T08:55:58.984 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/ioctl.o 2024-04-21T08:55:58.999 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/reboot.o 2024-04-21T08:55:59.073 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/async.o 2024-04-21T08:55:59.117 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/range.o 2024-04-21T08:55:59.217 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/smpboot.o 2024-04-21T08:55:59.347 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/ucount.o 2024-04-21T08:55:59.515 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/regset.o 2024-04-21T08:55:59.637 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/readdir.o 2024-04-21T08:55:59.738 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/select.o 2024-04-21T08:55:59.797 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/dcache.o 2024-04-21T08:55:59.804 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/inode.o 2024-04-21T08:55:59.813 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4getroot.o 2024-04-21T08:55:59.838 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/attr.o 2024-04-21T08:55:59.854 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/ksyms_common.o 2024-04-21T08:55:59.903 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/groups.o 2024-04-21T08:56:00.342 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/kcmp.o 2024-04-21T08:56:00.484 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/bad_inode.o 2024-04-21T08:56:00.502 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/file.o 2024-04-21T08:56:00.560 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/filesystems.o 2024-04-21T08:56:00.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:00 smithi129 ceph-mon[31272]: pgmap v332: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 717 KiB/s wr, 3.53k op/s 2024-04-21T08:56:00.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:00 smithi057 ceph-mon[27968]: pgmap v332: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 717 KiB/s wr, 3.53k op/s 2024-04-21T08:56:00.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:00 smithi136 ceph-mon[31791]: pgmap v332: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 717 KiB/s wr, 3.53k op/s 2024-04-21T08:56:00.736 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4client.o 2024-04-21T08:56:01.001 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/namespace.o 2024-04-21T08:56:01.049 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/freezer.o 2024-04-21T08:56:01.122 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/seq_file.o 2024-04-21T08:56:01.239 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/xattr.o 2024-04-21T08:56:01.258 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/libfs.o 2024-04-21T08:56:01.261 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fs-writeback.o 2024-04-21T08:56:01.707 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/pnode.o 2024-04-21T08:56:01.716 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/profile.o 2024-04-21T08:56:01.849 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4session.o 2024-04-21T08:56:01.960 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/splice.o 2024-04-21T08:56:02.273 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sync.o 2024-04-21T08:56:02.287 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/utimes.o 2024-04-21T08:56:02.315 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/d_path.o 2024-04-21T08:56:02.429 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/stacktrace.o 2024-04-21T08:56:02.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:02 smithi129 ceph-mon[31272]: pgmap v333: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 213 KiB/s wr, 2.66k op/s 2024-04-21T08:56:02.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:02 smithi057 ceph-mon[27968]: pgmap v333: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 213 KiB/s wr, 2.66k op/s 2024-04-21T08:56:02.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:02 smithi136 ceph-mon[31791]: pgmap v333: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 213 KiB/s wr, 2.66k op/s 2024-04-21T08:56:02.834 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/dns_resolve.o 2024-04-21T08:56:02.996 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/dma.o 2024-04-21T08:56:03.017 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/smp.o 2024-04-21T08:56:03.025 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/uid16.o 2024-04-21T08:56:03.028 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/stack.o 2024-04-21T08:56:03.053 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fs_struct.o 2024-04-21T08:56:03.152 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/statfs.o 2024-04-21T08:56:03.246 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fs_pin.o 2024-04-21T08:56:03.460 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/kallsyms.o 2024-04-21T08:56:03.463 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nsfs.o 2024-04-21T08:56:03.546 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fs_types.o 2024-04-21T08:56:03.655 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4trace.o 2024-04-21T08:56:03.730 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fs_context.o 2024-04-21T08:56:03.798 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/acct.o 2024-04-21T08:56:03.883 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fs_parser.o 2024-04-21T08:56:03.976 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/crash_core.o 2024-04-21T08:56:03.988 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fsopen.o 2024-04-21T08:56:04.033 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/init.o 2024-04-21T08:56:04.445 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/kexec_core.o 2024-04-21T08:56:04.461 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/kernel_read_file.o 2024-04-21T08:56:04.507 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/mnt_idmapping.o 2024-04-21T08:56:04.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:04 smithi129 ceph-mon[31272]: pgmap v334: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 290 KiB/s wr, 2.40k op/s 2024-04-21T08:56:04.574 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/kexec.o 2024-04-21T08:56:04.578 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/compat.o 2024-04-21T08:56:04.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:04 smithi057 ceph-mon[27968]: pgmap v334: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 290 KiB/s wr, 2.40k op/s 2024-04-21T08:56:04.611 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/remap_range.o 2024-04-21T08:56:04.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:04 smithi136 ceph-mon[31791]: pgmap v334: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 290 KiB/s wr, 2.40k op/s 2024-04-21T08:56:04.680 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/buffer.o 2024-04-21T08:56:04.977 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/mpage.o 2024-04-21T08:56:04.994 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/proc_namespace.o 2024-04-21T08:56:05.248 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/utsname.o 2024-04-21T08:56:05.288 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/pid_namespace.o 2024-04-21T08:56:05.413 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/direct-io.o 2024-04-21T08:56:05.438 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/stop_machine.o 2024-04-21T08:56:05.592 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/eventpoll.o 2024-04-21T08:56:05.661 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/audit.o 2024-04-21T08:56:05.899 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/anon_inodes.o 2024-04-21T08:56:05.996 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/auditfilter.o 2024-04-21T08:56:06.104 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/nfs/nfs4sysctl.o 2024-04-21T08:56:06.153 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/auditsc.o 2024-04-21T08:56:06.345 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/signalfd.o 2024-04-21T08:56:06.436 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/timerfd.o 2024-04-21T08:56:06.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:06 smithi129 ceph-mon[31272]: pgmap v335: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 293 KiB/s wr, 1.87k op/s 2024-04-21T08:56:06.591 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:06 smithi057 ceph-mon[27968]: pgmap v335: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 293 KiB/s wr, 1.87k op/s 2024-04-21T08:56:06.605 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/eventfd.o 2024-04-21T08:56:06.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:06 smithi136 ceph-mon[31791]: pgmap v335: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 293 KiB/s wr, 1.87k op/s 2024-04-21T08:56:06.969 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/nfs/built-in.a 2024-04-21T08:56:07.013 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/aio.o 2024-04-21T08:56:07.020 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/locks.o 2024-04-21T08:56:07.097 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/binfmt_misc.o 2024-04-21T08:56:07.502 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/audit_watch.o 2024-04-21T08:56:07.608 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/binfmt_script.o 2024-04-21T08:56:07.623 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/binfmt_elf.o 2024-04-21T08:56:07.947 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/audit_fsnotify.o 2024-04-21T08:56:08.349 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/compat_binfmt_elf.o 2024-04-21T08:56:08.505 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/mbcache.o 2024-04-21T08:56:08.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:08 smithi129 ceph-mon[31272]: pgmap v336: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 245 KiB/s wr, 578 op/s 2024-04-21T08:56:08.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:08 smithi129 ceph-mon[31272]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T08:56:08.599 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:08 smithi057 ceph-mon[27968]: pgmap v336: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 245 KiB/s wr, 578 op/s 2024-04-21T08:56:08.600 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:08 smithi057 ceph-mon[27968]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T08:56:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:08 smithi136 ceph-mon[31791]: pgmap v336: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 245 KiB/s wr, 578 op/s 2024-04-21T08:56:08.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:08 smithi136 ceph-mon[31791]: osdmap e107: 12 total, 12 up, 12 in 2024-04-21T08:56:08.841 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/posix_acl.o 2024-04-21T08:56:08.853 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/audit_tree.o 2024-04-21T08:56:08.939 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/kprobes.o 2024-04-21T08:56:09.285 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/coredump.o 2024-04-21T08:56:09.442 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/drop_caches.o 2024-04-21T08:56:09.847 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/sysctls.o 2024-04-21T08:56:09.855 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/seccomp.o 2024-04-21T08:56:10.132 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/relay.o 2024-04-21T08:56:10.141 INFO:tasks.workunit.client.0.smithi057.stdout: CC fs/fhandle.o 2024-04-21T08:56:10.268 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/utsname_sysctl.o 2024-04-21T08:56:10.442 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/delayacct.o 2024-04-21T08:56:10.488 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/taskstats.o 2024-04-21T08:56:10.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:10 smithi129 ceph-mon[31272]: pgmap v338: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 796 KiB/s wr, 107 op/s 2024-04-21T08:56:10.593 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:10 smithi057 ceph-mon[27968]: pgmap v338: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 796 KiB/s wr, 107 op/s 2024-04-21T08:56:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:10 smithi136 ceph-mon[31791]: pgmap v338: 129 pgs: 8 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 98 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 796 KiB/s wr, 107 op/s 2024-04-21T08:56:10.645 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/tsacct.o 2024-04-21T08:56:10.996 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/tracepoint.o 2024-04-21T08:56:11.296 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/irq_work.o 2024-04-21T08:56:11.362 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/static_call.o 2024-04-21T08:56:11.419 INFO:tasks.workunit.client.0.smithi057.stdout: AR fs/built-in.a 2024-04-21T08:56:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.650+0000 7f6620fe1640 1 -- 172.21.15.57:0/2107038965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 msgr2=0x7f661c113960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.650+0000 7f6620fe1640 1 --2- 172.21.15.57:0/2107038965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c113960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6610009900 tx=0x7f661002bcd0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.655+0000 7f6620fe1640 1 -- 172.21.15.57:0/2107038965 shutdown_connections 2024-04-21T08:56:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.655+0000 7f6620fe1640 1 --2- 172.21.15.57:0/2107038965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f661c113ea0 0x7f661c116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.655+0000 7f6620fe1640 1 --2- 172.21.15.57:0/2107038965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c113960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.655+0000 7f6620fe1640 1 --2- 172.21.15.57:0/2107038965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f661c10a8f0 0x7f661c10acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.655+0000 7f6620fe1640 1 -- 172.21.15.57:0/2107038965 >> 172.21.15.57:0/2107038965 conn(0x7f661c06ce90 msgr2=0x7f661c06d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.662+0000 7f6620fe1640 1 -- 172.21.15.57:0/2107038965 shutdown_connections 2024-04-21T08:56:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.663+0000 7f6620fe1640 1 -- 172.21.15.57:0/2107038965 wait complete. 2024-04-21T08:56:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.664+0000 7f6620fe1640 1 Processor -- start 2024-04-21T08:56:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.664+0000 7f6620fe1640 1 -- start start 2024-04-21T08:56:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f6620fe1640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f661c10a8f0 0x7f661c1a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f6620fe1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c1a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f6620fe1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f661c113ea0 0x7f661c19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f6620fe1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f661c118590 con 0x7f661c113ea0 2024-04-21T08:56:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f6620fe1640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f661c118410 con 0x7f661c10a8f0 2024-04-21T08:56:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f6620fe1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f661c118710 con 0x7f661c10b210 2024-04-21T08:56:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c1a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c1a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55912/0 (socket says 172.21.15.57:55912) 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 -- 172.21.15.57:0/3205511073 learned_addr learned my addr 172.21.15.57:0/3205511073 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661b7fe640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f661c10a8f0 0x7f661c1a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 -- 172.21.15.57:0/3205511073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f661c10a8f0 msgr2=0x7f661c1a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f661c10a8f0 0x7f661c1a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 -- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f661c113ea0 msgr2=0x7f661c19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f661c113ea0 0x7f661c19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.665+0000 7f661affd640 1 -- 172.21.15.57:0/3205511073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c19f010 con 0x7f661c10b210 2024-04-21T08:56:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.666+0000 7f661affd640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c1a4c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6610009230 tx=0x7f661002d890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.681+0000 7f6618ff9640 1 -- 172.21.15.57:0/3205511073 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661002dd50 con 0x7f661c10b210 2024-04-21T08:56:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.681+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661c19f2a0 con 0x7f661c10b210 2024-04-21T08:56:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.681+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661c1ab5a0 con 0x7f661c10b210 2024-04-21T08:56:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.681+0000 7f6618ff9640 1 -- 172.21.15.57:0/3205511073 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f661003eb00 con 0x7f661c10b210 2024-04-21T08:56:11.682 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.682+0000 7f6618ff9640 1 -- 172.21.15.57:0/3205511073 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610003c10 con 0x7f661c10b210 2024-04-21T08:56:11.684 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.684+0000 7f6624e38740 1 Processor -- start 2024-04-21T08:56:11.686 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.686+0000 7f6618ff9640 1 -- 172.21.15.57:0/3205511073 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f661003e650 con 0x7f661c10b210 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.686+0000 7f6624e38740 1 -- start start 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.687+0000 7f6618ff9640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66040a3ea0 0x7f66040a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.687+0000 7f6618ff9640 1 -- 172.21.15.57:0/3205511073 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f66100f6670 con 0x7f661c10b210 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f6624e38740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff77ac0 0x561a5ff76810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f6624e38740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f6624e38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff826c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f6624e38740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561a5fedb8d0 con 0x561a5ff77ac0 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f6624e38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561a5fe78e80 con 0x561a5ff802d0 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f6624e38740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561a5fed7890 con 0x561a5ff76d50 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55914/0 (socket says 172.21.15.57:55914) 2024-04-21T08:56:11.688 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.688+0000 7f661affd640 1 -- 172.21.15.57:0/3918449230 learned_addr learned my addr 172.21.15.57:0/3918449230 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.689+0000 7f661bfff640 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff826c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.689 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661bfff640 1 -- 172.21.15.57:0/3918449230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 msgr2=0x561a5ff771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661bfff640 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff771d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661bfff640 1 -- 172.21.15.57:0/3918449230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff77ac0 msgr2=0x561a5ff76810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661bfff640 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff77ac0 0x561a5ff76810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661bfff640 1 -- 172.21.15.57:0/3918449230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5fe9a320 con 0x561a5ff802d0 2024-04-21T08:56:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661affd640 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:56:11.690 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.690+0000 7f661bfff640 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff826c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f660c005fd0 tx=0x7f660c00bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bdd5fef96164c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.692+0000 7f661bfff640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66040a3ea0 0x7f66040a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.696+0000 7f6601ffb640 1 -- 172.21.15.57:0/3918449230 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c01b070 con 0x561a5ff802d0 2024-04-21T08:56:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.696+0000 7f6601ffb640 1 -- 172.21.15.57:0/3918449230 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f660c014350 con 0x561a5ff802d0 2024-04-21T08:56:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.697+0000 7f6624e38740 1 -- 172.21.15.57:0/3918449230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 msgr2=0x561a5ff826c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.697+0000 7f6624e38740 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff826c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f660c005fd0 tx=0x7f660c00bef0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f6624e38740 1 -- 172.21.15.57:0/3918449230 shutdown_connections 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f6624e38740 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff826c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f6624e38740 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f6624e38740 1 --2- 172.21.15.57:0/3918449230 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff77ac0 0x561a5ff76810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f6624e38740 1 -- 172.21.15.57:0/3918449230 >> 172.21.15.57:0/3918449230 conn(0x561a5ff3dc90 msgr2=0x561a5ff71a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f6624e38740 1 -- 172.21.15.57:0/3918449230 shutdown_connections 2024-04-21T08:56:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.700+0000 7f661bfff640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66040a3ea0 0x7f66040a6360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x561a5ff72bd0 tx=0x7f660c00bef0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.701+0000 7f6624e38740 1 -- 172.21.15.57:0/3918449230 wait complete. 2024-04-21T08:56:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.701+0000 7f6624e38740 1 -- start start 2024-04-21T08:56:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.701+0000 7f6624e38740 1 Processor -- start 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 0x561a5ff7faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff7ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x561a5ff83050 con 0x561a5ff802d0 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x561a5fe63eb0 con 0x561a5ff76d50 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x561a5ff831d0 con 0x561a5ff77ac0 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f6624e38740 10 client.0 _ll_register_callbacks cb 0x561a5fe90b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f661b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 0x561a5ff7faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f661b7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 0x561a5ff7faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55172/0 (socket says 172.21.15.57:55172) 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f661b7fe640 1 -- 172.21.15.57:0/1920252577 learned_addr learned my addr 172.21.15.57:0/1920252577 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.702+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff7ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.703+0000 7f661b7fe640 1 -- 172.21.15.57:0/1920252577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 msgr2=0x561a5ff7ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.703+0000 7f661b7fe640 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff7ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.703+0000 7f661b7fe640 1 -- 172.21.15.57:0/1920252577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 msgr2=0x561a5ff7d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.703+0000 7f661b7fe640 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.703+0000 7f661b7fe640 1 -- 172.21.15.57:0/1920252577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5fbee210 con 0x561a5ff76d50 2024-04-21T08:56:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.703+0000 7f661b7fe640 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 0x561a5ff7faf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f661400e9f0 tx=0x7f661400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63ca3b6954dd95f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.707+0000 7f65faffd640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:56:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.708+0000 7f65faffd640 1 -- 172.21.15.57:0/1920252577 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614004d20 con 0x561a5ff76d50 2024-04-21T08:56:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.708+0000 7f65faffd640 1 -- 172.21.15.57:0/1920252577 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f661400cac0 con 0x561a5ff76d50 2024-04-21T08:56:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.710+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f660c014350 con 0x561a5ff76d50 2024-04-21T08:56:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.713+0000 7f65faffd640 1 -- 172.21.15.57:0/1920252577 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f661400ccc0 con 0x561a5ff76d50 2024-04-21T08:56:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 10 client.26747 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561a5fcba620 con 0x561a5ff76d50 2024-04-21T08:56:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f65faffd640 1 -- 172.21.15.57:0/1920252577 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f660c014350 con 0x561a5ff76d50 2024-04-21T08:56:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f65faffd640 1 -- 172.21.15.57:0/1920252577 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f6614024440 con 0x561a5ff76d50 2024-04-21T08:56:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 10 client.26747 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:11.719 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 10 client.26747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 20 client.26747 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5ff90330 0x561a5ff927c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 4 client.26747 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.719+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a5feef880 con 0x561a5ff90330 2024-04-21T08:56:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.720+0000 7f661affd640 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5ff90330 0x561a5ff927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.721+0000 7f661affd640 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5ff90330 0x561a5ff927c0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae184b40c18a81e9 server_cookie=99989725fb768c14 in_seq=0 out_seq=0 2024-04-21T08:56:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.724+0000 7f65faffd640 10 client.26747 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.724+0000 7f65faffd640 1 -- 172.21.15.57:0/1920252577 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a5feef880 con 0x561a5ff90330 2024-04-21T08:56:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.724+0000 7f65faffd640 10 client.26747 handle_command_reply: tid=0 2024-04-21T08:56:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.724+0000 7f6624e38740 1 client.26747 shutdown 2024-04-21T08:56:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.724+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 msgr2=0x561a5ff7faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 0x561a5ff7faf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f661400e9f0 tx=0x7f661400eeb0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 shutdown_connections 2024-04-21T08:56:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5ff90330 0x561a5ff927c0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff7ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 --2- 172.21.15.57:0/1920252577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff76d50 0x561a5ff7faf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f65faffd640 0 client.26747 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 >> 172.21.15.57:0/1920252577 conn(0x561a5ff3dc90 msgr2=0x561a5ff73790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.725+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 shutdown_connections 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.726+0000 7f6624e38740 1 -- 172.21.15.57:0/1920252577 wait complete. 2024-04-21T08:56:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.726+0000 7f6624e38740 20 client.26747 trim_cache size 0 max 16384 2024-04-21T08:56:11.780 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.780+0000 7f6624e38740 1 Processor -- start 2024-04-21T08:56:11.781 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.782+0000 7f6624e38740 1 -- start start 2024-04-21T08:56:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.782+0000 7f6624e38740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff8c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.782 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.782+0000 7f6624e38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff77ac0 0x561a5ff76270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f6624e38740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f661affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff77ac0 0x561a5ff76270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f661affd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff77ac0 0x561a5ff76270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:55188/0 (socket says 172.21.15.57:55188) 2024-04-21T08:56:11.783 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f661affd640 1 -- 172.21.15.57:0/680878304 learned_addr learned my addr 172.21.15.57:0/680878304 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f660800e0f0 con 0x561a5ff802d0 2024-04-21T08:56:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6608010950 con 0x561a5ff77ac0 2024-04-21T08:56:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.783+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f660800e740 con 0x561a5ff76d50 2024-04-21T08:56:11.784 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.784+0000 7f661b7fe640 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff8c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.786 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.787+0000 7f661bfff640 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f661b7fe640 1 -- 172.21.15.57:0/680878304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff77ac0 msgr2=0x561a5ff76270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f661b7fe640 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff77ac0 0x561a5ff76270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f661b7fe640 1 -- 172.21.15.57:0/680878304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 msgr2=0x561a5ff7be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f661b7fe640 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f661b7fe640 1 -- 172.21.15.57:0/680878304 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a5ff71f80 con 0x561a5ff76d50 2024-04-21T08:56:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f661b7fe640 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff8c000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6614024a50 tx=0x7f661400c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66a5abaaf2bdd1f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.788+0000 7f66017fa640 1 -- 172.21.15.57:0/680878304 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614029070 con 0x561a5ff76d50 2024-04-21T08:56:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.789+0000 7f66017fa640 1 -- 172.21.15.57:0/680878304 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6614003e20 con 0x561a5ff76d50 2024-04-21T08:56:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.789+0000 7f66017fa640 1 -- 172.21.15.57:0/680878304 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614024660 con 0x561a5ff76d50 2024-04-21T08:56:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.790+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 msgr2=0x561a5ff8c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.790+0000 7f6624e38740 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff8c000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6614024a50 tx=0x7f661400c750 comp rx=0 tx=0).stop 2024-04-21T08:56:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.792+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 shutdown_connections 2024-04-21T08:56:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.792+0000 7f6624e38740 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff802d0 0x561a5ff7be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.792+0000 7f6624e38740 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff77ac0 0x561a5ff76270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.792+0000 7f6624e38740 1 --2- 172.21.15.57:0/680878304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff76d50 0x561a5ff8c000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.792 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.792+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 >> 172.21.15.57:0/680878304 conn(0x561a5ff3dc90 msgr2=0x561a5ff7b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.794+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 shutdown_connections 2024-04-21T08:56:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.794+0000 7f6624e38740 1 -- 172.21.15.57:0/680878304 wait complete. 2024-04-21T08:56:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.794+0000 7f6624e38740 1 -- start start 2024-04-21T08:56:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.795+0000 7f6624e38740 1 Processor -- start 2024-04-21T08:56:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.795+0000 7f6624e38740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff76d50 0x561a5ff88950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.796+0000 7f6624e38740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff83cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.796+0000 7f6624e38740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff84220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.796+0000 7f6624e38740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f660800ee10 con 0x561a5ff76d50 2024-04-21T08:56:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.796+0000 7f6624e38740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6608010280 con 0x561a5ff802d0 2024-04-21T08:56:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.796+0000 7f6624e38740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f660800f4e0 con 0x561a5ff77ac0 2024-04-21T08:56:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.796+0000 7f6624e38740 10 client.0 _ll_register_callbacks cb 0x7f66140075a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.797+0000 7f661bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff84220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.797+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff83cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.797+0000 7f661affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff83cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55958/0 (socket says 172.21.15.57:55958) 2024-04-21T08:56:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.797+0000 7f661affd640 1 -- 172.21.15.57:0/2057008528 learned_addr learned my addr 172.21.15.57:0/2057008528 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.798+0000 7f661affd640 1 -- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 msgr2=0x561a5ff84220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.798+0000 7f661affd640 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff84220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.798+0000 7f661affd640 1 -- 172.21.15.57:0/2057008528 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff76d50 msgr2=0x561a5ff88950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.798+0000 7f661affd640 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff76d50 0x561a5ff88950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.798+0000 7f661affd640 1 -- 172.21.15.57:0/2057008528 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a5ff84a50 con 0x561a5ff77ac0 2024-04-21T08:56:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.798+0000 7f661affd640 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff83cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f661002c820 tx=0x7f6610040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74278c835057a7e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:11.799 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/static_call_inline.o 2024-04-21T08:56:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.800+0000 7f65faffd640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:56:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.800+0000 7f65faffd640 1 -- 172.21.15.57:0/2057008528 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66100df020 con 0x561a5ff77ac0 2024-04-21T08:56:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.800+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6614003e20 con 0x561a5ff77ac0 2024-04-21T08:56:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.800+0000 7f65faffd640 1 -- 172.21.15.57:0/2057008528 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f66100c03b0 con 0x561a5ff77ac0 2024-04-21T08:56:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.810+0000 7f65faffd640 1 -- 172.21.15.57:0/2057008528 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f66100e12a0 con 0x561a5ff77ac0 2024-04-21T08:56:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.810+0000 7f6624e38740 10 client.26626 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.810+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x561a5fff6a90 con 0x561a5ff77ac0 2024-04-21T08:56:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.810+0000 7f65faffd640 1 -- 172.21.15.57:0/2057008528 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6614003e20 con 0x561a5ff77ac0 2024-04-21T08:56:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.810+0000 7f65faffd640 1 -- 172.21.15.57:0/2057008528 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f66100d1ce0 con 0x561a5ff77ac0 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f6624e38740 10 client.26626 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f6624e38740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f6624e38740 20 client.26626 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f6624e38740 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5fff7140 0x561a5fff9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f6624e38740 4 client.26626 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561a5fe7b3d0 con 0x561a5fff7140 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.812+0000 7f661b7fe640 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5fff7140 0x561a5fff9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:11.812 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.813+0000 7f661b7fe640 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5fff7140 0x561a5fff9580 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=685d2449547f8b65 server_cookie=da51a678e3cd7a68 in_seq=0 out_seq=0 2024-04-21T08:56:11.813 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:11.813+0000 7f65faffd640 10 client.26626 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:11.887 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/crash_dump.o 2024-04-21T08:56:11.930 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/jump_label.o 2024-04-21T08:56:12.109 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/context_tracking.o 2024-04-21T08:56:12.130 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/iomem.o 2024-04-21T08:56:12.266 INFO:tasks.workunit.client.0.smithi057.stdout: CC kernel/rseq.o 2024-04-21T08:56:12.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:12 smithi129 ceph-mon[31272]: pgmap v339: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 1.3 MiB/s wr, 102 op/s 2024-04-21T08:56:12.592 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:12 smithi057 ceph-mon[27968]: pgmap v339: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 1.3 MiB/s wr, 102 op/s 2024-04-21T08:56:12.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:12 smithi136 ceph-mon[31791]: pgmap v339: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 1.3 MiB/s wr, 102 op/s 2024-04-21T08:56:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.498+0000 7f65faffd640 1 -- 172.21.15.57:0/2057008528 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561a5fe7b3d0 con 0x561a5fff7140 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.498+0000 7f65faffd640 10 client.26626 handle_command_reply: tid=0 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.498+0000 7f6624e38740 1 client.26626 shutdown 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.498+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 msgr2=0x561a5ff83cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.498+0000 7f6624e38740 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff83cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f661002c820 tx=0x7f6610040040 comp rx=0 tx=0).stop 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 shutdown_connections 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x561a5fff7140 0x561a5fff9580 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x561a5ff802d0 0x561a5ff84220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x561a5ff77ac0 0x561a5ff83cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 --2- 172.21.15.57:0/2057008528 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x561a5ff76d50 0x561a5ff88950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 >> 172.21.15.57:0/2057008528 conn(0x561a5ff3dc90 msgr2=0x561a5ff7bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f65faffd640 0 client.26626 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 shutdown_connections 2024-04-21T08:56:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 1 -- 172.21.15.57:0/2057008528 wait complete. 2024-04-21T08:56:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.499+0000 7f6624e38740 20 client.26626 trim_cache size 0 max 16384 2024-04-21T08:56:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:{ 2024-04-21T08:56:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "message": "", 2024-04-21T08:56:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout: "return_code": 0 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stdout:} 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.500+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66040a3ea0 msgr2=0x7f66040a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.500+0000 7f6620fe1640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66040a3ea0 0x7f66040a6360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x561a5ff72bd0 tx=0x7f660c00bef0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.500+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 msgr2=0x7f661c1a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.500+0000 7f6620fe1640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c1a4c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6610009230 tx=0x7f661002d890 comp rx=0 tx=0).stop 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.501+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 shutdown_connections 2024-04-21T08:56:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.501+0000 7f6620fe1640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f66040a3ea0 0x7f66040a6360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.501+0000 7f6620fe1640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f661c113ea0 0x7f661c19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.501+0000 7f6620fe1640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f661c10b210 0x7f661c1a4c20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.501+0000 7f6620fe1640 1 --2- 172.21.15.57:0/3205511073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f661c10a8f0 0x7f661c1a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.501+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 >> 172.21.15.57:0/3205511073 conn(0x7f661c06ce90 msgr2=0x7f661c070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.502+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 shutdown_connections 2024-04-21T08:56:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi057.front.sepia.ceph.com.smithi057.stderr:2024-04-21T08:56:13.502+0000 7f6620fe1640 1 -- 172.21.15.57:0/3205511073 wait complete. 2024-04-21T08:56:13.634 INFO:tasks.workunit.client.0.smithi057.stdout: AR kernel/built-in.a 2024-04-21T08:56:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-21T08:56:13.800 INFO:tasks.workunit.client.0.smithi057.stderr:make[1]: *** [/home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/Makefile:2034: .] Error 2 2024-04-21T08:56:13.800 INFO:tasks.workunit.client.0.smithi057.stderr:make: *** [Makefile:234: __sub-make] Error 2 2024-04-21T08:56:13.801 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.0... 2024-04-21T08:56:13.801 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-21T08:56:14.335 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi057 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-21T08:56:14.578 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d7a0e8de39354bb580519907c654895b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi057 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-21T08:56:14.580 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-21T08:56:14.590 INFO:tasks.quiescer:joining Quiescers 2024-04-21T08:56:14.591 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:56:14.596 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:14 smithi057 ceph-mon[27968]: pgmap v340: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2024-04-21T08:56:14.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:14 smithi136 ceph-mon[31791]: pgmap v340: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2024-04-21T08:56:14.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.795+0000 7f1073132640 1 -- 172.21.15.57:0/4072718104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 msgr2=0x7f106c109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:14.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.795+0000 7f1073132640 1 --2- 172.21.15.57:0/4072718104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c109490 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1058009900 tx=0x7f105802bc60 comp rx=0 tx=0).stop 2024-04-21T08:56:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.804+0000 7f1073132640 1 -- 172.21.15.57:0/4072718104 shutdown_connections 2024-04-21T08:56:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.804+0000 7f1073132640 1 --2- 172.21.15.57:0/4072718104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f106c109b70 0x7f106c11d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.804+0000 7f1073132640 1 --2- 172.21.15.57:0/4072718104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c109490 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.804+0000 7f1073132640 1 --2- 172.21.15.57:0/4072718104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c1106c0 0x7f106c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:14.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.804+0000 7f1073132640 1 -- 172.21.15.57:0/4072718104 >> 172.21.15.57:0/4072718104 conn(0x7f106c06e410 msgr2=0x7f106c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:14.810 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.810+0000 7f1073132640 1 -- 172.21.15.57:0/4072718104 shutdown_connections 2024-04-21T08:56:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.813+0000 7f1073132640 1 -- 172.21.15.57:0/4072718104 wait complete. 2024-04-21T08:56:14.813 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.813+0000 7f1073132640 1 Processor -- start 2024-04-21T08:56:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.815+0000 7f1073132640 1 -- start start 2024-04-21T08:56:14.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.816+0000 7f1073132640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c1106c0 0x7f106c114bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.816+0000 7f1073132640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c115110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.816+0000 7f1073132640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f106c119500 0x7f106c1199b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.816+0000 7f1073132640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f106c1113e0 con 0x7f106c110b30 2024-04-21T08:56:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.816+0000 7f1073132640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f106c111260 con 0x7f106c1106c0 2024-04-21T08:56:14.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.816+0000 7f1073132640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f106c111560 con 0x7f106c119500 2024-04-21T08:56:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.818+0000 7f106bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c115110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.818+0000 7f106bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c115110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52284/0 (socket says 172.21.15.57:52284) 2024-04-21T08:56:14.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.818+0000 7f106bfff640 1 -- 172.21.15.57:0/199242864 learned_addr learned my addr 172.21.15.57:0/199242864 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:14.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.819+0000 7f1070ece640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c1106c0 0x7f106c114bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:14.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.819+0000 7f106bfff640 1 -- 172.21.15.57:0/199242864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f106c119500 msgr2=0x7f106c1199b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:56:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.819+0000 7f106bfff640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f106c119500 0x7f106c1199b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.819+0000 7f106bfff640 1 -- 172.21.15.57:0/199242864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c1106c0 msgr2=0x7f106c114bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.819+0000 7f106bfff640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c1106c0 0x7f106c114bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.819+0000 7f106bfff640 1 -- 172.21.15.57:0/199242864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c119ff0 con 0x7f106c110b30 2024-04-21T08:56:14.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.820+0000 7f106bfff640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c115110 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f105802c270 tx=0x7f10580045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:14.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:14 smithi129 ceph-mon[31272]: pgmap v340: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2024-04-21T08:56:14.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.821+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058042020 con 0x7f106c110b30 2024-04-21T08:56:14.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.821+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10580035f0 con 0x7f106c110b30 2024-04-21T08:56:14.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.821+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105803e7e0 con 0x7f106c110b30 2024-04-21T08:56:14.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.825+0000 7f1073132640 1 -- 172.21.15.57:0/199242864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c11a280 con 0x7f106c110b30 2024-04-21T08:56:14.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.825+0000 7f1073132640 1 -- 172.21.15.57:0/199242864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c1bcb30 con 0x7f106c110b30 2024-04-21T08:56:14.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.829+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104c005290 con 0x7f106c110b30 2024-04-21T08:56:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.833+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1058045020 con 0x7f106c110b30 2024-04-21T08:56:14.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.834+0000 7f1069ffb640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f105c0a3f30 0x7f105c0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.835+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f10580f24e0 con 0x7f106c110b30 2024-04-21T08:56:14.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.836+0000 7f1070ece640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f105c0a3f30 0x7f105c0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.845+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f105804c060 con 0x7f106c110b30 2024-04-21T08:56:14.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:14.845+0000 7f1070ece640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f105c0a3f30 0x7f105c0a63f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f10600058c0 tx=0x7f106000a4c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.294+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f104c005850 con 0x7f106c110b30 2024-04-21T08:56:15.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.295+0000 7f1069ffb640 1 -- 172.21.15.57:0/199242864 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f10580be820 con 0x7f106c110b30 2024-04-21T08:56:15.296 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:56:15.296 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:56:15.296 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:56:15.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f105c0a3f30 msgr2=0x7f105c0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f105c0a3f30 0x7f105c0a63f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f10600058c0 tx=0x7f106000a4c0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 msgr2=0x7f106c115110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.299 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c115110 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f105802c270 tx=0x7f10580045c0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 shutdown_connections 2024-04-21T08:56:15.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f105c0a3f30 0x7f105c0a63f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f106c119500 0x7f106c1199b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f106c110b30 0x7f106c115110 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 --2- 172.21.15.57:0/199242864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c1106c0 0x7f106c114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.300 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.299+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 >> 172.21.15.57:0/199242864 conn(0x7f106c06e410 msgr2=0x7f106c11ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.300+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 shutdown_connections 2024-04-21T08:56:15.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.300+0000 7f10577fe640 1 -- 172.21.15.57:0/199242864 wait complete. 2024-04-21T08:56:15.320 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24503 quiesce db --cancel --all' 2024-04-21T08:56:15.321 DEBUG:teuthology.orchestra.run.smithi057:> timeout 45 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.24503 quiesce db --cancel --all 2024-04-21T08:56:15.522 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.522+0000 7f20f5ea0640 1 -- 172.21.15.57:0/628064416 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f0104970 msgr2=0x7f20f0104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.523 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.522+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/628064416 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f0104970 0x7f20f0104e00 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009cd0 tx=0x7f20e002c120 comp rx=0 tx=0).stop 2024-04-21T08:56:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.527+0000 7f20f5ea0640 1 -- 172.21.15.57:0/628064416 shutdown_connections 2024-04-21T08:56:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.527+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/628064416 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f0104970 0x7f20f0104e00 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.527+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/628064416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20f006a8a0 0x7f20f006ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.527 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.527+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/628064416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20f0106eb0 0x7f20f006a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.528 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.527+0000 7f20f5ea0640 1 -- 172.21.15.57:0/628064416 >> 172.21.15.57:0/628064416 conn(0x7f20f0077ef0 msgr2=0x7f20f0078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.530 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.531+0000 7f20f5ea0640 1 -- 172.21.15.57:0/628064416 shutdown_connections 2024-04-21T08:56:15.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.532+0000 7f20f5ea0640 1 -- 172.21.15.57:0/628064416 wait complete. 2024-04-21T08:56:15.533 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.533+0000 7f20f5ea0640 1 Processor -- start 2024-04-21T08:56:15.536 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.537+0000 7f20f5ea0640 1 -- start start 2024-04-21T08:56:15.537 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.537+0000 7f20f5ea0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 0x7f20f01103b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20f5ea0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20f0104970 0x7f20f0109480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20ef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 0x7f20f01103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20ef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 0x7f20f01103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52302/0 (socket says 172.21.15.57:52302) 2024-04-21T08:56:15.538 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20ef7fe640 1 -- 172.21.15.57:0/4162358458 learned_addr learned my addr 172.21.15.57:0/4162358458 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:15.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20f0106eb0 0x7f20f01099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0118880 con 0x7f20f006a8a0 2024-04-21T08:56:15.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0118700 con 0x7f20f0104970 2024-04-21T08:56:15.539 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.538+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0118a00 con 0x7f20f0106eb0 2024-04-21T08:56:15.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.541+0000 7f20eeffd640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20f0104970 0x7f20f0109480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.541 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.541+0000 7f20ef7fe640 1 -- 172.21.15.57:0/4162358458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20f0106eb0 msgr2=0x7f20f01099c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:56:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.541+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20f0106eb0 0x7f20f01099c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.541+0000 7f20ef7fe640 1 -- 172.21.15.57:0/4162358458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20f0104970 msgr2=0x7f20f0109480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.541+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20f0104970 0x7f20f0109480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.541+0000 7f20ef7fe640 1 -- 172.21.15.57:0/4162358458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f010a0f0 con 0x7f20f006a8a0 2024-04-21T08:56:15.542 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.543+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 0x7f20f01103b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f20e800b450 tx=0x7f20e800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.543 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.543+0000 7f20ecff9640 1 -- 172.21.15.57:0/4162358458 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8015020 con 0x7f20f006a8a0 2024-04-21T08:56:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.546+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f010ded0 con 0x7f20f006a8a0 2024-04-21T08:56:15.547 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.546+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f010e470 con 0x7f20f006a8a0 2024-04-21T08:56:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.548+0000 7f20ecff9640 1 -- 172.21.15.57:0/4162358458 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e8011d40 con 0x7f20f006a8a0 2024-04-21T08:56:15.548 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.548+0000 7f20ecff9640 1 -- 172.21.15.57:0/4162358458 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e800f340 con 0x7f20f006a8a0 2024-04-21T08:56:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.553+0000 7f20ecff9640 1 -- 172.21.15.57:0/4162358458 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f20e8018020 con 0x7f20f006a8a0 2024-04-21T08:56:15.552 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.553+0000 7f20f8b68740 1 Processor -- start 2024-04-21T08:56:15.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.554+0000 7f20ecff9640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f20dc0a3e20 0x7f20dc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.554 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.554+0000 7f20ecff9640 1 -- 172.21.15.57:0/4162358458 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f20e80d0540 con 0x7f20f006a8a0 2024-04-21T08:56:15.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 -- start start 2024-04-21T08:56:15.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b6361d10 0x5598b63620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.557 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b63626c0 0x5598b636ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b636d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5598b62c7000 con 0x5598b63626c0 2024-04-21T08:56:15.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598b6263b20 con 0x5598b6361d10 2024-04-21T08:56:15.558 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.557+0000 7f20f8b68740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5598b62c2fc0 con 0x5598b636b240 2024-04-21T08:56:15.560 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.561+0000 7f20effff640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f20dc0a3e20 0x7f20dc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.562+0000 7f20effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b636d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.562+0000 7f20effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b636d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43440/0 (socket says 172.21.15.57:43440) 2024-04-21T08:56:15.562 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.562+0000 7f20effff640 1 -- 172.21.15.57:0/3245344203 learned_addr learned my addr 172.21.15.57:0/3245344203 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:15.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.563+0000 7f20effff640 1 -- 172.21.15.57:0/3245344203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b6361d10 msgr2=0x5598b63620f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:15.563 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.563+0000 7f20effff640 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b6361d10 0x5598b63620f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.563+0000 7f20effff640 1 -- 172.21.15.57:0/3245344203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b63626c0 msgr2=0x5598b636ac70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:15.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.563+0000 7f20effff640 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b63626c0 0x5598b636ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.563+0000 7f20effff640 1 -- 172.21.15.57:0/3245344203 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598b627dea0 con 0x5598b636b240 2024-04-21T08:56:15.564 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.564+0000 7f20effff640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f20dc0a3e20 0x7f20dc0a62e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009bf0 tx=0x7f20e0032da0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.571 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.572+0000 7f20effff640 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b636d690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009cd0 tx=0x7f20e0044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff753eb857006301 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.576+0000 7f20e5ffb640 1 -- 172.21.15.57:0/3245344203 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0039040 con 0x5598b636b240 2024-04-21T08:56:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.576+0000 7f20e5ffb640 1 -- 172.21.15.57:0/3245344203 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e00044f0 con 0x5598b636b240 2024-04-21T08:56:15.576 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.577+0000 7f20e5ffb640 1 -- 172.21.15.57:0/3245344203 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e00348c0 con 0x5598b636b240 2024-04-21T08:56:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.577+0000 7f20f8b68740 1 -- 172.21.15.57:0/3245344203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 msgr2=0x5598b636d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.577 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.577+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b636d690 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009cd0 tx=0x7f20e0044000 comp rx=0 tx=0).stop 2024-04-21T08:56:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.578+0000 7f20f8b68740 1 -- 172.21.15.57:0/3245344203 shutdown_connections 2024-04-21T08:56:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.578+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b636d690 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.578+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b63626c0 0x5598b636ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.578+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3245344203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b6361d10 0x5598b63620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.578 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.578+0000 7f20f8b68740 1 -- 172.21.15.57:0/3245344203 >> 172.21.15.57:0/3245344203 conn(0x5598b635c3e0 msgr2=0x5598b635c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.581 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.581+0000 7f20f8b68740 1 -- 172.21.15.57:0/3245344203 shutdown_connections 2024-04-21T08:56:15.583 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.583+0000 7f20f8b68740 1 -- 172.21.15.57:0/3245344203 wait complete. 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.583+0000 7f20f8b68740 1 -- start start 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.583+0000 7f20f8b68740 1 Processor -- start 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b63680e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6368620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b6368b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5598b637f1b0 con 0x5598b636b240 2024-04-21T08:56:15.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598b6261510 con 0x5598b63626c0 2024-04-21T08:56:15.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5598b637f330 con 0x5598b6361d10 2024-04-21T08:56:15.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.584+0000 7f20f8b68740 10 client.0 _ll_register_callbacks cb 0x5598b6260f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.585+0000 7f20effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b6368b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.585+0000 7f20effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b6368b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52312/0 (socket says 172.21.15.57:52312) 2024-04-21T08:56:15.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.585+0000 7f20effff640 1 -- 172.21.15.57:0/3977915096 learned_addr learned my addr 172.21.15.57:0/3977915096 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:15.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.585+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b63680e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.587 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.585+0000 7f20eeffd640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6368620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/199242864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:15.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.590+0000 7f20effff640 1 -- 172.21.15.57:0/3977915096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 msgr2=0x5598b63680e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.590 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.590+0000 7f20effff640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b63680e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.590+0000 7f20effff640 1 -- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 msgr2=0x5598b6368620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.590+0000 7f20effff640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6368620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.591 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.591+0000 7f20effff640 1 -- 172.21.15.57:0/3977915096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598b60a5a20 con 0x5598b636b240 2024-04-21T08:56:15.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.592+0000 7f20eeffd640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6368620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:15.594 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.592+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b63680e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:56:15.598 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.599+0000 7f20effff640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b6368b60 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009ca0 tx=0x7f20e0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236a244a8d4144a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.599 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.600+0000 7f20ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:56:15.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.607+0000 7f20ceffd640 1 -- 172.21.15.57:0/3977915096 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e005a020 con 0x5598b636b240 2024-04-21T08:56:15.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.607+0000 7f20ceffd640 1 -- 172.21.15.57:0/3977915096 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f20e00046a0 con 0x5598b636b240 2024-04-21T08:56:15.610 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.610+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20e00044f0 con 0x5598b636b240 2024-04-21T08:56:15.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.615+0000 7f20ceffd640 1 -- 172.21.15.57:0/3977915096 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f20e0039070 con 0x5598b636b240 2024-04-21T08:56:15.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.616+0000 7f20f8b68740 10 client.17244 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:15.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.616+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5598b61061b0 con 0x5598b636b240 2024-04-21T08:56:15.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.623+0000 7f20ceffd640 1 -- 172.21.15.57:0/3977915096 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e00044f0 con 0x5598b636b240 2024-04-21T08:56:15.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.623+0000 7f20ceffd640 1 -- 172.21.15.57:0/3977915096 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f20e0067020 con 0x5598b636b240 2024-04-21T08:56:15.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.624+0000 7f20f8b68740 10 client.17244 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:15.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.624+0000 7f20f8b68740 10 client.17244 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-21T08:56:15.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.624+0000 7f20f8b68740 20 client.17244 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:15.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.624+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b638c560 0x5598b638e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.624+0000 7f20f8b68740 4 client.17244 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:15.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.624+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598b62d4a20 con 0x5598b638c560 2024-04-21T08:56:15.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.625+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b638c560 0x5598b638e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.634+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b638c560 0x5598b638e950 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4836066004e2486b server_cookie=d41b4d32c868d3cd in_seq=0 out_seq=0 2024-04-21T08:56:15.639 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/199242864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:15.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.635+0000 7f20ceffd640 10 client.17244 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:15.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.637+0000 7f20ceffd640 1 -- 172.21.15.57:0/3977915096 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598b62d4a20 con 0x5598b638c560 2024-04-21T08:56:15.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.637+0000 7f20ceffd640 10 client.17244 handle_command_reply: tid=0 2024-04-21T08:56:15.640 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.638+0000 7f20f8b68740 1 client.17244 shutdown 2024-04-21T08:56:15.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.640+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 msgr2=0x5598b6368b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.640+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b6368b60 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009ca0 tx=0x7f20e0032f40 comp rx=0 tx=0).stop 2024-04-21T08:56:15.642 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 shutdown_connections 2024-04-21T08:56:15.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b638c560 0x5598b638e950 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b6368b60 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6368620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20f8b68740 1 --2- 172.21.15.57:0/3977915096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b63680e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 >> 172.21.15.57:0/3977915096 conn(0x5598b635c3e0 msgr2=0x5598b635b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.643 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.641+0000 7f20ceffd640 0 client.17244 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:15.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.644+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 shutdown_connections 2024-04-21T08:56:15.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.647+0000 7f20f8b68740 1 -- 172.21.15.57:0/3977915096 wait complete. 2024-04-21T08:56:15.650 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.647+0000 7f20f8b68740 20 client.17244 trim_cache size 0 max 16384 2024-04-21T08:56:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.682+0000 7f20f8b68740 1 Processor -- start 2024-04-21T08:56:15.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.683+0000 7f20f8b68740 1 -- start start 2024-04-21T08:56:15.683 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.683+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b6368980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.683+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6387d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.683+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b63653a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.684+0000 7f20f8b68740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f20d000e150 con 0x5598b636b240 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.684+0000 7f20f8b68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20d00109b0 con 0x5598b63626c0 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.684+0000 7f20f8b68740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f20d000e7a0 con 0x5598b6361d10 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.684+0000 7f20effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b63653a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.684+0000 7f20effff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b63653a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52328/0 (socket says 172.21.15.57:52328) 2024-04-21T08:56:15.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.684+0000 7f20effff640 1 -- 172.21.15.57:0/1096828490 learned_addr learned my addr 172.21.15.57:0/1096828490 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.685+0000 7f20effff640 1 -- 172.21.15.57:0/1096828490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 msgr2=0x5598b6368980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.685+0000 7f20effff640 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b6368980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.687 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.685+0000 7f20effff640 1 -- 172.21.15.57:0/1096828490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 msgr2=0x5598b6387d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.685+0000 7f20effff640 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6387d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.685+0000 7f20effff640 1 -- 172.21.15.57:0/1096828490 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598b5f99310 con 0x5598b636b240 2024-04-21T08:56:15.688 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.686+0000 7f20effff640 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b63653a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f20e004d430 tx=0x7f20e0032e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c976279cb26fe7e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.689+0000 7f20e57fa640 1 -- 172.21.15.57:0/1096828490 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e005a070 con 0x5598b636b240 2024-04-21T08:56:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.689+0000 7f20e57fa640 1 -- 172.21.15.57:0/1096828490 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e002c810 con 0x5598b636b240 2024-04-21T08:56:15.689 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.689+0000 7f20e57fa640 1 -- 172.21.15.57:0/1096828490 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e004dbf0 con 0x5598b636b240 2024-04-21T08:56:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.691+0000 7f20f8b68740 1 -- 172.21.15.57:0/1096828490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 msgr2=0x5598b63653a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.691+0000 7f20f8b68740 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b63653a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f20e004d430 tx=0x7f20e0032e10 comp rx=0 tx=0).stop 2024-04-21T08:56:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.694+0000 7f20f8b68740 1 -- 172.21.15.57:0/1096828490 shutdown_connections 2024-04-21T08:56:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.694+0000 7f20f8b68740 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b636b240 0x5598b63653a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.694+0000 7f20f8b68740 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b6387d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.694+0000 7f20f8b68740 1 --2- 172.21.15.57:0/1096828490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b6361d10 0x5598b6368980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.694 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.694+0000 7f20f8b68740 1 -- 172.21.15.57:0/1096828490 >> 172.21.15.57:0/1096828490 conn(0x5598b635c3e0 msgr2=0x5598b6387610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.699 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.699+0000 7f20f8b68740 1 -- 172.21.15.57:0/1096828490 shutdown_connections 2024-04-21T08:56:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.699+0000 7f20f8b68740 1 -- 172.21.15.57:0/1096828490 wait complete. 2024-04-21T08:56:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.700+0000 7f20f8b68740 1 -- start start 2024-04-21T08:56:15.700 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.700+0000 7f20f8b68740 1 Processor -- start 2024-04-21T08:56:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b6361d10 0x5598b638fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b638ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b63f1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f20d000ee70 con 0x5598b6361d10 2024-04-21T08:56:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20d00102e0 con 0x5598b63626c0 2024-04-21T08:56:15.701 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f20d000f540 con 0x5598b636b240 2024-04-21T08:56:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.701+0000 7f20f8b68740 10 client.0 _ll_register_callbacks cb 0x7f20e0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:15.702 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.702+0000 7f20effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b63f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.702+0000 7f20eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b638ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.702+0000 7f20effff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b63f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43458/0 (socket says 172.21.15.57:43458) 2024-04-21T08:56:15.703 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.702+0000 7f20effff640 1 -- 172.21.15.57:0/2245215313 learned_addr learned my addr 172.21.15.57:0/2245215313 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:15.704 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.705+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b6361d10 0x5598b638fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.706 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.706+0000 7f20effff640 1 -- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 msgr2=0x5598b638ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.706+0000 7f20effff640 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b638ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.706+0000 7f20effff640 1 -- 172.21.15.57:0/2245215313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b6361d10 msgr2=0x5598b638fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.706+0000 7f20effff640 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b6361d10 0x5598b638fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.707 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.706+0000 7f20effff640 1 -- 172.21.15.57:0/2245215313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598b63f1c80 con 0x5598b636b240 2024-04-21T08:56:15.712 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.712+0000 7f20effff640 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b63f1740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f20e0034ba0 tx=0x7f20e0034730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5f41fb324cd1e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:15.713 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.713+0000 7f20ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:56:15.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.717+0000 7f20ceffd640 1 -- 172.21.15.57:0/2245215313 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e005a040 con 0x5598b636b240 2024-04-21T08:56:15.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.717+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20e002c810 con 0x5598b636b240 2024-04-21T08:56:15.718 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.717+0000 7f20ceffd640 1 -- 172.21.15.57:0/2245215313 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f20e004d970 con 0x5598b636b240 2024-04-21T08:56:15.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.724+0000 7f20ceffd640 1 -- 172.21.15.57:0/2245215313 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f20e004db70 con 0x5598b636b240 2024-04-21T08:56:15.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.725+0000 7f20f8b68740 10 client.26632 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:15.728 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.725+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5598b63f2f00 con 0x5598b636b240 2024-04-21T08:56:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20ceffd640 1 -- 172.21.15.57:0/2245215313 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e002c810 con 0x5598b636b240 2024-04-21T08:56:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20ceffd640 1 -- 172.21.15.57:0/2245215313 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f20e0067030 con 0x5598b636b240 2024-04-21T08:56:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20f8b68740 10 client.26632 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20f8b68740 10 client.26632 resolve_mds: validated gid 24503 aka daemon mds.b 2024-04-21T08:56:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20f8b68740 20 client.26632 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:15.732 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20f8b68740 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b63f3290 0x5598b63f5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:15.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20f8b68740 4 client.26632 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-21T08:56:15.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.732+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x5598b62ce890 con 0x5598b63f3290 2024-04-21T08:56:15.733 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.733+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b63f3290 0x5598b63f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:15.741 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.742+0000 7f20ef7fe640 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b63f3290 0x5598b63f5680 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a469082f458f206b server_cookie=3652ebd6a2a7aa86 in_seq=0 out_seq=0 2024-04-21T08:56:15.742 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.742+0000 7f20ceffd640 10 client.26632 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:15.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.743+0000 7f20ceffd640 1 -- 172.21.15.57:0/2245215313 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+79 (crc 0 0 0) 0x5598b62ce890 con 0x5598b63f3290 2024-04-21T08:56:15.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.743+0000 7f20ceffd640 10 client.26632 handle_command_reply: tid=0 2024-04-21T08:56:15.745 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.743+0000 7f20f8b68740 1 client.26632 shutdown 2024-04-21T08:56:15.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.750+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 msgr2=0x5598b63f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.750+0000 7f20f8b68740 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b63f1740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f20e0034ba0 tx=0x7f20e0034730 comp rx=0 tx=0).stop 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 shutdown_connections 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20f8b68740 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x5598b63f3290 0x5598b63f5680 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20f8b68740 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5598b636b240 0x5598b63f1740 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20f8b68740 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598b63626c0 0x5598b638ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20f8b68740 1 --2- 172.21.15.57:0/2245215313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x5598b6361d10 0x5598b638fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 >> 172.21.15.57:0/2245215313 conn(0x5598b635c3e0 msgr2=0x5598b6387ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.751+0000 7f20ceffd640 0 client.26632 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:15.759 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:15.759 INFO:teuthology.orchestra.run.smithi057.stdout: "epoch": 21, 2024-04-21T08:56:15.759 INFO:teuthology.orchestra.run.smithi057.stdout: "leader": 24503, 2024-04-21T08:56:15.759 INFO:teuthology.orchestra.run.smithi057.stdout: "set_version": 3, 2024-04-21T08:56:15.759 INFO:teuthology.orchestra.run.smithi057.stdout: "sets": {} 2024-04-21T08:56:15.759 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:15.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.757+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 shutdown_connections 2024-04-21T08:56:15.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.759+0000 7f20f8b68740 1 -- 172.21.15.57:0/2245215313 wait complete. 2024-04-21T08:56:15.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.759+0000 7f20f8b68740 20 client.26632 trim_cache size 0 max 16384 2024-04-21T08:56:15.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.760+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f20dc0a3e20 msgr2=0x7f20dc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f20dc0a3e20 0x7f20dc0a62e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009bf0 tx=0x7f20e0032da0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 msgr2=0x7f20f01103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 0x7f20f01103b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f20e800b450 tx=0x7f20e800b910 comp rx=0 tx=0).stop 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 shutdown_connections 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f20dc0a3e20 0x7f20dc0a62e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20f0106eb0 0x7f20f01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20f0104970 0x7f20f0109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:15.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 --2- 172.21.15.57:0/4162358458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f20f006a8a0 0x7f20f01103b0 secure :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f20e800b450 tx=0x7f20e800b910 comp rx=0 tx=0).stop 2024-04-21T08:56:15.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.761+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 >> 172.21.15.57:0/4162358458 conn(0x7f20f0077ef0 msgr2=0x7f20f0103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:15.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.762+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 shutdown_connections 2024-04-21T08:56:15.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:15.763+0000 7f20f5ea0640 1 -- 172.21.15.57:0/4162358458 wait complete. 2024-04-21T08:56:15.785 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 310, in task stop_all_quiescers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 263, in stop_all_quiescers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Error releasing root: -124 (Unknown) 2024-04-21T08:56:15.792 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-21T08:56:15.802 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:56:15.815 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-21T08:56:15.825 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-21T08:56:15.826 INFO:teuthology.task.exec:Executing custom commands... 2024-04-21T08:56:15.827 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:56:15.827 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-21T08:56:15.828 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/199242864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:15.904 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-21T08:56:15.952 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-21T08:56:16.169 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.167+0000 7f46d0ab3640 1 -- 172.21.15.57:0/765715124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc1021a0 msgr2=0x7f46cc102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.170 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.167+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/765715124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc1021a0 0x7f46cc102580 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009900 tx=0x7f46c402bc80 comp rx=0 tx=0).stop 2024-04-21T08:56:16.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.175+0000 7f46d0ab3640 1 -- 172.21.15.57:0/765715124 shutdown_connections 2024-04-21T08:56:16.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.175+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/765715124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46cc107270 0x7f46cc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.175+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/765715124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc102b50 0x7f46cc106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.175+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/765715124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc1021a0 0x7f46cc102580 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.177 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.175+0000 7f46d0ab3640 1 -- 172.21.15.57:0/765715124 >> 172.21.15.57:0/765715124 conn(0x7f46cc0fda00 msgr2=0x7f46cc0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:16.178 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.178+0000 7f46d0ab3640 1 -- 172.21.15.57:0/765715124 shutdown_connections 2024-04-21T08:56:16.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.179+0000 7f46d0ab3640 1 -- 172.21.15.57:0/765715124 wait complete. 2024-04-21T08:56:16.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.180+0000 7f46d0ab3640 1 Processor -- start 2024-04-21T08:56:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 -- start start 2024-04-21T08:56:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc1021a0 0x7f46cc10ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 0x7f46cc10bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46cc10c5d0 0x7f46cc1aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc075d70 con 0x7f46cc107270 2024-04-21T08:56:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc075bf0 con 0x7f46cc1021a0 2024-04-21T08:56:16.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.184+0000 7f46d0ab3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc075ef0 con 0x7f46cc10c5d0 2024-04-21T08:56:16.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.187+0000 7f46c9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 0x7f46cc10bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.187+0000 7f46c9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 0x7f46cc10bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52364/0 (socket says 172.21.15.57:52364) 2024-04-21T08:56:16.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.187+0000 7f46c9d9b640 1 -- 172.21.15.57:0/3105256829 learned_addr learned my addr 172.21.15.57:0/3105256829 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:16.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.189+0000 7f46c9d9b640 1 -- 172.21.15.57:0/3105256829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46cc10c5d0 msgr2=0x7f46cc1aafa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:56:16.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.189+0000 7f46c9d9b640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46cc10c5d0 0x7f46cc1aafa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.189+0000 7f46c9d9b640 1 -- 172.21.15.57:0/3105256829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc1021a0 msgr2=0x7f46cc10ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.189+0000 7f46c9d9b640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc1021a0 0x7f46cc10ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.189+0000 7f46c9d9b640 1 -- 172.21.15.57:0/3105256829 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc10cd50 con 0x7f46cc107270 2024-04-21T08:56:16.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.190+0000 7f46c9d9b640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 0x7f46cc10bf30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f46b400cce0 tx=0x7f46b400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:16.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.190+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b400f650 con 0x7f46cc107270 2024-04-21T08:56:16.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.191+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc114b40 con 0x7f46cc107270 2024-04-21T08:56:16.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.191+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc115090 con 0x7f46cc107270 2024-04-21T08:56:16.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.191+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b400fca0 con 0x7f46cc107270 2024-04-21T08:56:16.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.191+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b4018b90 con 0x7f46cc107270 2024-04-21T08:56:16.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.194+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f46b4004b20 con 0x7f46cc107270 2024-04-21T08:56:16.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.195+0000 7f46c37fe640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f46ac0a3f70 0x7f46ac0a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.196+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f46b40d4060 con 0x7f46cc107270 2024-04-21T08:56:16.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.196+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b0005290 con 0x7f46cc107270 2024-04-21T08:56:16.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.198+0000 7f46ca59c640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f46ac0a3f70 0x7f46ac0a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:16.210 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.210+0000 7f46ca59c640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f46ac0a3f70 0x7f46ac0a6430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f46c4005c20 tx=0x7f46c4000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:16.215 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.215+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f46b4014030 con 0x7f46cc107270 2024-04-21T08:56:16.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:16 smithi136 ceph-mon[31791]: pgmap v341: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.1 MiB/s wr, 149 op/s 2024-04-21T08:56:16.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.636+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f46b0000bc0 con 0x7f46ac0a3f70 2024-04-21T08:56:16.779 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.779+0000 7f46c37fe640 1 -- 172.21.15.57:0/3105256829 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f46b0000bc0 con 0x7f46ac0a3f70 2024-04-21T08:56:16.779 INFO:teuthology.orchestra.run.smithi057.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-21T00:00:00", "created": "2024-04-21T08:45:30", "first": "2024-04-21T08:46:00", "last": "2024-04-21T08:56:00", "last_pruned": "2024-04-21T08:56:00", "created_count": 11, "pruned_count": 5, "active": true} 2024-04-21T08:56:16.782 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f46ac0a3f70 msgr2=0x7f46ac0a6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f46ac0a3f70 0x7f46ac0a6430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f46c4005c20 tx=0x7f46c4000bc0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 msgr2=0x7f46cc10bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 0x7f46cc10bf30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f46b400cce0 tx=0x7f46b400a620 comp rx=0 tx=0).stop 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 shutdown_connections 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f46ac0a3f70 0x7f46ac0a6430 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46cc10c5d0 0x7f46cc1aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f46cc107270 0x7f46cc10bf30 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 --2- 172.21.15.57:0/3105256829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc1021a0 0x7f46cc10ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.783 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 >> 172.21.15.57:0/3105256829 conn(0x7f46cc0fda00 msgr2=0x7f46cc0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:16.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 shutdown_connections 2024-04-21T08:56:16.784 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.782+0000 7f46d0ab3640 1 -- 172.21.15.57:0/3105256829 wait complete. 2024-04-21T08:56:16.798 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:16 smithi057 ceph-mon[27968]: pgmap v341: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.1 MiB/s wr, 149 op/s 2024-04-21T08:56:16.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-21T08:56:16.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:16 smithi129 ceph-mon[31272]: pgmap v341: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.1 MiB/s wr, 149 op/s 2024-04-21T08:56:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.985+0000 7fa560b1f640 1 -- 172.21.15.57:0/752024124 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554032950 con 0x7fa55c1150a0 2024-04-21T08:56:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.986+0000 7fa563d85640 1 -- 172.21.15.57:0/752024124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c1150a0 msgr2=0x7fa55c117490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:16.985 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.986+0000 7fa563d85640 1 --2- 172.21.15.57:0/752024124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c1150a0 0x7fa55c117490 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa554009cd0 tx=0x7fa55402c1b0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.991+0000 7fa563d85640 1 -- 172.21.15.57:0/752024124 shutdown_connections 2024-04-21T08:56:16.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.992+0000 7fa563d85640 1 --2- 172.21.15.57:0/752024124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c1150a0 0x7fa55c117490 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.992 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.992+0000 7fa563d85640 1 --2- 172.21.15.57:0/752024124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 0x7fa55c114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.992+0000 7fa563d85640 1 --2- 172.21.15.57:0/752024124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c108f50 0x7fa55c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:16.993 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.992+0000 7fa563d85640 1 -- 172.21.15.57:0/752024124 >> 172.21.15.57:0/752024124 conn(0x7fa55c06d020 msgr2=0x7fa55c06d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:16.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.998+0000 7fa563d85640 1 -- 172.21.15.57:0/752024124 shutdown_connections 2024-04-21T08:56:16.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.998+0000 7fa563d85640 1 -- 172.21.15.57:0/752024124 wait complete. 2024-04-21T08:56:16.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.999+0000 7fa563d85640 1 Processor -- start 2024-04-21T08:56:16.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.999+0000 7fa563d85640 1 -- start start 2024-04-21T08:56:16.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.999+0000 7fa563d85640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c108f50 0x7fa55c10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:16.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:16.999+0000 7fa563d85640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 0x7fa55c10d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:17.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa563d85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c1150a0 0x7fa55c10d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:17.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa563d85640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c11aa80 con 0x7fa55c1150a0 2024-04-21T08:56:17.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa563d85640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c11a900 con 0x7fa55c108f50 2024-04-21T08:56:17.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa563d85640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c11ac00 con 0x7fa55c109870 2024-04-21T08:56:17.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa562322640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c1150a0 0x7fa55c10d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:17.001 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa562322640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c1150a0 0x7fa55c10d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52368/0 (socket says 172.21.15.57:52368) 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa562322640 1 -- 172.21.15.57:0/4251643289 learned_addr learned my addr 172.21.15.57:0/4251643289 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa561320640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 0x7fa55c10d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.000+0000 7fa561b21640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c108f50 0x7fa55c10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.001+0000 7fa561320640 1 -- 172.21.15.57:0/4251643289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c108f50 msgr2=0x7fa55c10cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.001+0000 7fa561320640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c108f50 0x7fa55c10cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.001+0000 7fa561320640 1 -- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c1150a0 msgr2=0x7fa55c10d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.001+0000 7fa561320640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c1150a0 0x7fa55c10d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.001+0000 7fa561320640 1 -- 172.21.15.57:0/4251643289 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55c1a2f20 con 0x7fa55c109870 2024-04-21T08:56:17.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.011+0000 7fa561320640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 0x7fa55c10d110 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa55800d6d0 tx=0x7fa55800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:17.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.014+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558017070 con 0x7fa55c109870 2024-04-21T08:56:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.014+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55c1a3210 con 0x7fa55c109870 2024-04-21T08:56:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.014+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55c1a3760 con 0x7fa55c109870 2024-04-21T08:56:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.014+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa558009820 con 0x7fa55c109870 2024-04-21T08:56:17.014 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.014+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558004c40 con 0x7fa55c109870 2024-04-21T08:56:17.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.016+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa55c06e580 con 0x7fa55c109870 2024-04-21T08:56:17.020 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.020+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa558005220 con 0x7fa55c109870 2024-04-21T08:56:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.021+0000 7fa552ffd640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa5480a3fa0 0x7fa5480a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:17.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.022+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7fa5580d07f0 con 0x7fa55c109870 2024-04-21T08:56:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.023+0000 7fa561b21640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa5480a3fa0 0x7fa5480a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:17.024 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.024+0000 7fa561b21640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa5480a3fa0 0x7fa5480a6460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006fd0 tx=0x7fa54c0090e0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:17.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.028+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa55809c4d0 con 0x7fa55c109870 2024-04-21T08:56:17.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.442+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 --> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa55c071140 con 0x7fa5480a3fa0 2024-04-21T08:56:17.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.490+0000 7fa552ffd640 1 -- 172.21.15.57:0/4251643289 <== mgr.15375 v2:172.21.15.57:6800/3217118060 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa55c071140 con 0x7fa5480a3fa0 2024-04-21T08:56:17.490 INFO:teuthology.orchestra.run.smithi057.stdout:/ 1m 6m3h 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa5480a3fa0 msgr2=0x7fa5480a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa5480a3fa0 0x7fa5480a6460 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006fd0 tx=0x7fa54c0090e0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 msgr2=0x7fa55c10d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 0x7fa55c10d110 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa55800d6d0 tx=0x7fa55800db90 comp rx=0 tx=0).stop 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 shutdown_connections 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fa5480a3fa0 0x7fa5480a6460 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa55c1150a0 0x7fa55c10d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa55c109870 0x7fa55c10d110 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 --2- 172.21.15.57:0/4251643289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa55c108f50 0x7fa55c10cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:17.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 >> 172.21.15.57:0/4251643289 conn(0x7fa55c06d020 msgr2=0x7fa55c070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:17.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 shutdown_connections 2024-04-21T08:56:17.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:17.494+0000 7fa563d85640 1 -- 172.21.15.57:0/4251643289 wait complete. 2024-04-21T08:56:17.511 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-21T08:56:17.522 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-21T08:56:17.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:17 smithi136 ceph-mon[31791]: from='client.17256 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:17.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:17 smithi129 ceph-mon[31272]: from='client.17256 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:17.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:17 smithi057 ceph-mon[27968]: from='client.17256 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:18 smithi136 ceph-mon[31791]: pgmap v342: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-04-21T08:56:18.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:18 smithi136 ceph-mon[31791]: from='client.26638 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:18.777 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:18 smithi057 ceph-mon[27968]: pgmap v342: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-04-21T08:56:18.777 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:18 smithi057 ceph-mon[27968]: from='client.26638 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:18.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:18 smithi129 ceph-mon[31272]: pgmap v342: 129 pgs: 20 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 57 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 2.1 MiB/s wr, 177 op/s 2024-04-21T08:56:18.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:18 smithi129 ceph-mon[31272]: from='client.26638 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-21T08:56:19.589 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-21T08:56:19.589 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-21T08:56:19.790 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.790+0000 7f514fd15640 1 -- 172.21.15.57:0/3314318190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 msgr2=0x7f514810dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.791 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.790+0000 7f514fd15640 1 --2- 172.21.15.57:0/3314318190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514810dc70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5140009860 tx=0x7f514002bb70 comp rx=0 tx=0).stop 2024-04-21T08:56:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.793+0000 7f514fd15640 1 -- 172.21.15.57:0/3314318190 shutdown_connections 2024-04-21T08:56:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.793+0000 7f514fd15640 1 --2- 172.21.15.57:0/3314318190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514810dc70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.793+0000 7f514fd15640 1 --2- 172.21.15.57:0/3314318190 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51481010a0 0x7f5148101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.793+0000 7f514fd15640 1 --2- 172.21.15.57:0/3314318190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5148106ed0 0x7f51481072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.793+0000 7f514fd15640 1 -- 172.21.15.57:0/3314318190 >> 172.21.15.57:0/3314318190 conn(0x7f5148077c40 msgr2=0x7f5148078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.799+0000 7f514fd15640 1 -- 172.21.15.57:0/3314318190 shutdown_connections 2024-04-21T08:56:19.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.799+0000 7f514fd15640 1 -- 172.21.15.57:0/3314318190 wait complete. 2024-04-21T08:56:19.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.800+0000 7f514fd15640 1 Processor -- start 2024-04-21T08:56:19.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.801+0000 7f514fd15640 1 -- start start 2024-04-21T08:56:19.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.801+0000 7f514fd15640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51481010a0 0x7f51480737a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.802+0000 7f514fd15640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514806c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.802+0000 7f514fd15640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5148106ed0 0x7f514806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.802+0000 7f514fd15640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f514810ffa0 con 0x7f5148106ed0 2024-04-21T08:56:19.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.802+0000 7f514fd15640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f514810fe20 con 0x7f51481010a0 2024-04-21T08:56:19.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.802+0000 7f514fd15640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5148110120 con 0x7f5148101a40 2024-04-21T08:56:19.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.804+0000 7f514e2b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5148106ed0 0x7f514806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.804+0000 7f514e2b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5148106ed0 0x7f514806ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52376/0 (socket says 172.21.15.57:52376) 2024-04-21T08:56:19.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.804+0000 7f514e2b2640 1 -- 172.21.15.57:0/3317848035 learned_addr learned my addr 172.21.15.57:0/3317848035 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:19.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.804+0000 7f514d2b0640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514806c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.804+0000 7f514dab1640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51481010a0 0x7f51480737a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.805+0000 7f514d2b0640 1 -- 172.21.15.57:0/3317848035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51481010a0 msgr2=0x7f51480737a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.805+0000 7f514d2b0640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51481010a0 0x7f51480737a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.805+0000 7f514d2b0640 1 -- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5148106ed0 msgr2=0x7f514806ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.805+0000 7f514d2b0640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5148106ed0 0x7f514806ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.805+0000 7f514d2b0640 1 -- 172.21.15.57:0/3317848035 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514806d440 con 0x7f5148101a40 2024-04-21T08:56:19.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.806+0000 7f514d2b0640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514806c8c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f513800ed70 tx=0x7f513800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.806+0000 7f5146ffd640 1 -- 172.21.15.57:0/3317848035 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513801c070 con 0x7f5148101a40 2024-04-21T08:56:19.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.806+0000 7f5146ffd640 1 -- 172.21.15.57:0/3317848035 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5138017af0 con 0x7f5148101a40 2024-04-21T08:56:19.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.806+0000 7f5146ffd640 1 -- 172.21.15.57:0/3317848035 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513800fbb0 con 0x7f5148101a40 2024-04-21T08:56:19.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.807+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51480712c0 con 0x7f5148101a40 2024-04-21T08:56:19.808 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.808+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51480717c0 con 0x7f5148101a40 2024-04-21T08:56:19.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.813+0000 7f51529e2740 1 Processor -- start 2024-04-21T08:56:19.815 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.813+0000 7f5146ffd640 1 -- 172.21.15.57:0/3317848035 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f513801f070 con 0x7f5148101a40 2024-04-21T08:56:19.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.814+0000 7f5146ffd640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51300a6130 0x7f51300a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.816 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.815+0000 7f5146ffd640 1 -- 172.21.15.57:0/3317848035 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f51380d0bb0 con 0x7f5148101a40 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.816+0000 7f51529e2740 1 -- start start 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f51529e2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a2f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f51529e2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 0x559390a2fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f51529e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a435a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f51529e2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x5593909941e0 con 0x559390a2f9a0 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f51529e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559390931390 con 0x559390a411b0 2024-04-21T08:56:19.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f51529e2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5593909901a0 con 0x559390a2eff0 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f514d2b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 0x559390a2fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f514d2b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 0x559390a2fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52392/0 (socket says 172.21.15.57:52392) 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.817+0000 7f514d2b0640 1 -- 172.21.15.57:0/1020568444 learned_addr learned my addr 172.21.15.57:0/1020568444 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.818+0000 7f514d2b0640 1 -- 172.21.15.57:0/1020568444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 msgr2=0x559390a2f3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.818+0000 7f514d2b0640 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a2f3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.818+0000 7f514d2b0640 1 -- 172.21.15.57:0/1020568444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 msgr2=0x559390a435a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.818+0000 7f514d2b0640 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a435a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.818+0000 7f514d2b0640 1 -- 172.21.15.57:0/1020568444 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593907707d0 con 0x559390a2f9a0 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.818+0000 7f514d2b0640 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 0x559390a2fe20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f51380991c0 tx=0x7f513809c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6244cbfd9fb984 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.819+0000 7f512ffff640 1 -- 172.21.15.57:0/1020568444 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51380a5460 con 0x559390a2f9a0 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.819+0000 7f512ffff640 1 -- 172.21.15.57:0/1020568444 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513809aaf0 con 0x559390a2f9a0 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.820+0000 7f514e2b2640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51300a6130 0x7f51300a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.820+0000 7f512ffff640 1 -- 172.21.15.57:0/1020568444 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138098fe0 con 0x559390a2f9a0 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.820+0000 7f51529e2740 1 -- 172.21.15.57:0/1020568444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 msgr2=0x559390a2fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.820+0000 7f51529e2740 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 0x559390a2fe20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f51380991c0 tx=0x7f513809c460 comp rx=0 tx=0).stop 2024-04-21T08:56:19.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.824+0000 7f51529e2740 1 -- 172.21.15.57:0/1020568444 shutdown_connections 2024-04-21T08:56:19.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.824+0000 7f51529e2740 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.824+0000 7f51529e2740 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2f9a0 0x559390a2fe20 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.824+0000 7f51529e2740 1 --2- 172.21.15.57:0/1020568444 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a2f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.824+0000 7f51529e2740 1 -- 172.21.15.57:0/1020568444 >> 172.21.15.57:0/1020568444 conn(0x559390a29cb0 msgr2=0x559390a2a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.825+0000 7f51529e2740 1 -- 172.21.15.57:0/1020568444 shutdown_connections 2024-04-21T08:56:19.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.826+0000 7f51529e2740 1 -- 172.21.15.57:0/1020568444 wait complete. 2024-04-21T08:56:19.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.826+0000 7f51529e2740 1 -- start start 2024-04-21T08:56:19.826 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.826+0000 7f51529e2740 1 Processor -- start 2024-04-21T08:56:19.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.830+0000 7f514e2b2640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51300a6130 0x7f51300a85f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x559390a2b0d0 tx=0x7f514002c090 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a3add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a40af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a411b0 0x559390a3b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x559390a34ce0 con 0x559390a411b0 2024-04-21T08:56:19.831 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55939094d2f0 con 0x559390a2f9a0 2024-04-21T08:56:19.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x559390a34e60 con 0x559390a2eff0 2024-04-21T08:56:19.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.831+0000 7f51529e2740 10 client.0 _ll_register_callbacks cb 0x55939095a580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:19.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514dab1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a3add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a40af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a40af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:34812/0 (socket says 172.21.15.57:34812) 2024-04-21T08:56:19.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 -- 172.21.15.57:0/943804214 learned_addr learned my addr 172.21.15.57:0/943804214 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:19.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 -- 172.21.15.57:0/943804214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 msgr2=0x559390a3add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a3add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 -- 172.21.15.57:0/943804214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a411b0 msgr2=0x559390a3b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a411b0 0x559390a3b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.832+0000 7f514d2b0640 1 -- 172.21.15.57:0/943804214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593906a63f0 con 0x559390a2f9a0 2024-04-21T08:56:19.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.833+0000 7f514d2b0640 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a40af0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f513809c780 tx=0x7f51380a0d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4487099ee8f63034 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.833+0000 7f512d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-04-21T08:56:19.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.833+0000 7f512d7fa640 1 -- 172.21.15.57:0/943804214 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138017ca0 con 0x559390a2f9a0 2024-04-21T08:56:19.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.833+0000 7f512d7fa640 1 -- 172.21.15.57:0/943804214 <== mon.2 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f51380a4fd0 con 0x559390a2f9a0 2024-04-21T08:56:19.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.834+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513809aaf0 con 0x559390a2f9a0 2024-04-21T08:56:19.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.834+0000 7f512d7fa640 1 -- 172.21.15.57:0/943804214 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f51380a5210 con 0x559390a2f9a0 2024-04-21T08:56:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.837+0000 7f51529e2740 10 client.26771 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:19.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.837+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x5593906660f0 con 0x559390a2f9a0 2024-04-21T08:56:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f512d7fa640 1 -- 172.21.15.57:0/943804214 <== mon.2 v2:172.21.15.129:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513809aaf0 con 0x559390a2f9a0 2024-04-21T08:56:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f512d7fa640 1 -- 172.21.15.57:0/943804214 <== mon.2 v2:172.21.15.129:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f513800b030 con 0x559390a2f9a0 2024-04-21T08:56:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f51529e2740 10 client.26771 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f51529e2740 10 client.26771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:19.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f51529e2740 20 client.26771 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:19.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f51529e2740 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390a48b50 0x559390a4afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f51529e2740 4 client.26771 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:19.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.839+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593909a1c00 con 0x559390a48b50 2024-04-21T08:56:19.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.840+0000 7f514dab1640 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390a48b50 0x559390a4afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.840+0000 7f512d7fa640 10 client.26771 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:19.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.841+0000 7f514dab1640 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390a48b50 0x559390a4afd0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=300889efc10e63f2 server_cookie=52d75873a4840775 in_seq=0 out_seq=0 2024-04-21T08:56:19.844 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.845+0000 7f512d7fa640 1 -- 172.21.15.57:0/943804214 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593909a1c00 con 0x559390a48b50 2024-04-21T08:56:19.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.845+0000 7f512d7fa640 10 client.26771 handle_command_reply: tid=0 2024-04-21T08:56:19.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.845+0000 7f51529e2740 1 client.26771 shutdown 2024-04-21T08:56:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.845+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 msgr2=0x559390a40af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.845+0000 7f51529e2740 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a40af0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f513809c780 tx=0x7f51380a0d20 comp rx=0 tx=0).stop 2024-04-21T08:56:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 shutdown_connections 2024-04-21T08:56:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390a48b50 0x559390a4afd0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a411b0 0x559390a3b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a40af0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 --2- 172.21.15.57:0/943804214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2eff0 0x559390a3add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 >> 172.21.15.57:0/943804214 conn(0x559390a29cb0 msgr2=0x559390a2ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f512d7fa640 0 client.26771 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 shutdown_connections 2024-04-21T08:56:19.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.846+0000 7f51529e2740 1 -- 172.21.15.57:0/943804214 wait complete. 2024-04-21T08:56:19.848 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.847+0000 7f51529e2740 20 client.26771 trim_cache size 0 max 16384 2024-04-21T08:56:19.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.894+0000 7f51529e2740 1 Processor -- start 2024-04-21T08:56:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 -- start start 2024-04-21T08:56:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a40a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2f9a0 0x559390a508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a50e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f51481d5b80 con 0x559390a2eff0 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51481d83e0 con 0x559390a411b0 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f51529e2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f51481d61d0 con 0x559390a2f9a0 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f514dab1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a40a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f514dab1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a40a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52410/0 (socket says 172.21.15.57:52410) 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f514dab1640 1 -- 172.21.15.57:0/908538895 learned_addr learned my addr 172.21.15.57:0/908538895 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.896+0000 7f514d2b0640 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2f9a0 0x559390a508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.897+0000 7f514e2b2640 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a50e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.897+0000 7f514dab1640 1 -- 172.21.15.57:0/908538895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2f9a0 msgr2=0x559390a508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.897+0000 7f514dab1640 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2f9a0 0x559390a508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.897+0000 7f514dab1640 1 -- 172.21.15.57:0/908538895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 msgr2=0x559390a50e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.897+0000 7f514dab1640 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a50e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.897+0000 7f514dab1640 1 -- 172.21.15.57:0/908538895 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559390a30380 con 0x559390a2eff0 2024-04-21T08:56:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.901+0000 7f514dab1640 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a40a30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f513c005bf0 tx=0x7f513c005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4d6ef382295fae server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.902+0000 7f512f7fe640 1 -- 172.21.15.57:0/908538895 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c041030 con 0x559390a2eff0 2024-04-21T08:56:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.902+0000 7f512f7fe640 1 -- 172.21.15.57:0/908538895 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513c00ce60 con 0x559390a2eff0 2024-04-21T08:56:19.903 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.902+0000 7f512f7fe640 1 -- 172.21.15.57:0/908538895 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c01a720 con 0x559390a2eff0 2024-04-21T08:56:19.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.905+0000 7f51529e2740 1 -- 172.21.15.57:0/908538895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 msgr2=0x559390a40a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.905+0000 7f51529e2740 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a40a30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f513c005bf0 tx=0x7f513c005d20 comp rx=0 tx=0).stop 2024-04-21T08:56:19.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.906+0000 7f51529e2740 1 -- 172.21.15.57:0/908538895 shutdown_connections 2024-04-21T08:56:19.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.906+0000 7f51529e2740 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a411b0 0x559390a50e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.906+0000 7f51529e2740 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a2f9a0 0x559390a508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.906+0000 7f51529e2740 1 --2- 172.21.15.57:0/908538895 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a40a30 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.906+0000 7f51529e2740 1 -- 172.21.15.57:0/908538895 >> 172.21.15.57:0/908538895 conn(0x559390a29cb0 msgr2=0x559390a33e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.908+0000 7f51529e2740 1 -- 172.21.15.57:0/908538895 shutdown_connections 2024-04-21T08:56:19.907 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.908+0000 7f51529e2740 1 -- 172.21.15.57:0/908538895 wait complete. 2024-04-21T08:56:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.908+0000 7f51529e2740 1 -- start start 2024-04-21T08:56:19.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.908+0000 7f51529e2740 1 Processor -- start 2024-04-21T08:56:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a5bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a5c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a411b0 0x559390a57690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f51481d68a0 con 0x559390a2eff0 2024-04-21T08:56:19.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51481d7d10 con 0x559390a2f9a0 2024-04-21T08:56:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f51481d6f70 con 0x559390a411b0 2024-04-21T08:56:19.910 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.909+0000 7f51529e2740 10 client.0 _ll_register_callbacks cb 0x7f513809c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:19.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.912+0000 7f514dab1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a5bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.912+0000 7f514dab1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a5bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52424/0 (socket says 172.21.15.57:52424) 2024-04-21T08:56:19.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.912+0000 7f514dab1640 1 -- 172.21.15.57:0/1562485796 learned_addr learned my addr 172.21.15.57:0/1562485796 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:19.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.912+0000 7f514d2b0640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a5c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.913+0000 7f514e2b2640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a411b0 0x559390a57690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.914+0000 7f514dab1640 1 -- 172.21.15.57:0/1562485796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a411b0 msgr2=0x559390a57690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.914+0000 7f514dab1640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a411b0 0x559390a57690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.914+0000 7f514dab1640 1 -- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 msgr2=0x559390a5c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.914+0000 7f514dab1640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a5c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.914+0000 7f514dab1640 1 -- 172.21.15.57:0/1562485796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559390a57e60 con 0x559390a2eff0 2024-04-21T08:56:19.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.915+0000 7f514dab1640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a5bef0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f513c00c920 tx=0x7f513c012d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed94f095e458633 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.916+0000 7f512d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3300/0 2024-04-21T08:56:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.916+0000 7f512d7fa640 1 -- 172.21.15.57:0/1562485796 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c01a420 con 0x559390a2eff0 2024-04-21T08:56:19.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.916+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513c00ce60 con 0x559390a2eff0 2024-04-21T08:56:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.916+0000 7f512d7fa640 1 -- 172.21.15.57:0/1562485796 <== mon.0 v2:172.21.15.57:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f513c01a5c0 con 0x559390a2eff0 2024-04-21T08:56:19.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.917+0000 7f512d7fa640 1 -- 172.21.15.57:0/1562485796 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f513c057050 con 0x559390a2eff0 2024-04-21T08:56:19.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.925+0000 7f51529e2740 10 client.17274 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:19.924 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.925+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x559390aae910 con 0x559390a2eff0 2024-04-21T08:56:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f512d7fa640 1 -- 172.21.15.57:0/1562485796 <== mon.0 v2:172.21.15.57:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f513c00ce60 con 0x559390a2eff0 2024-04-21T08:56:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f512d7fa640 1 -- 172.21.15.57:0/1562485796 <== mon.0 v2:172.21.15.57:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f513c004a40 con 0x559390a2eff0 2024-04-21T08:56:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f51529e2740 10 client.17274 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f51529e2740 10 client.17274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f51529e2740 20 client.17274 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:19.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f51529e2740 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390aaef80 0x559390ab13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:19.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f51529e2740 4 client.17274 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-21T08:56:19.927 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.926+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55939099ba70 con 0x559390aaef80 2024-04-21T08:56:19.932 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.931+0000 7f514d2b0640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390aaef80 0x559390ab13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.937+0000 7f514d2b0640 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390aaef80 0x559390ab13a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36a0b20a05ca4b6 server_cookie=b28fcfab34d94cc7 in_seq=0 out_seq=0 2024-04-21T08:56:19.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.938+0000 7f512d7fa640 10 client.17274 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:19.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.942+0000 7f512d7fa640 1 -- 172.21.15.57:0/1562485796 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55939099ba70 con 0x559390aaef80 2024-04-21T08:56:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.942+0000 7f512d7fa640 10 client.17274 handle_command_reply: tid=0 2024-04-21T08:56:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.942+0000 7f51529e2740 1 client.17274 shutdown 2024-04-21T08:56:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.943+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 msgr2=0x559390a5bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.943+0000 7f51529e2740 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a5bef0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f513c00c920 tx=0x7f513c012d20 comp rx=0 tx=0).stop 2024-04-21T08:56:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 shutdown_connections 2024-04-21T08:56:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f51529e2740 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x559390aaef80 0x559390ab13a0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f51529e2740 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559390a411b0 0x559390a57690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f51529e2740 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559390a2f9a0 0x559390a5c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f51529e2740 1 --2- 172.21.15.57:0/1562485796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x559390a2eff0 0x559390a5bef0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 >> 172.21.15.57:0/1562485796 conn(0x559390a29cb0 msgr2=0x559390a308a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.947+0000 7f512d7fa640 0 client.17274 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:19.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.951+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 shutdown_connections 2024-04-21T08:56:19.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.955+0000 7f51529e2740 1 -- 172.21.15.57:0/1562485796 wait complete. 2024-04-21T08:56:19.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.955+0000 7f51529e2740 20 client.17274 trim_cache size 0 max 16384 2024-04-21T08:56:19.959 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:19.959 INFO:teuthology.orchestra.run.smithi057.stdout: "status": "no active scrubs running", 2024-04-21T08:56:19.960 INFO:teuthology.orchestra.run.smithi057.stdout: "scrubs": {} 2024-04-21T08:56:19.960 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.961+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51300a6130 msgr2=0x7f51300a85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.961+0000 7f514fd15640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51300a6130 0x7f51300a85f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x559390a2b0d0 tx=0x7f514002c090 comp rx=0 tx=0).stop 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.961+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 msgr2=0x7f514806c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.961+0000 7f514fd15640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514806c8c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f513800ed70 tx=0x7f513800c6e0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.962+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 shutdown_connections 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.962+0000 7f514fd15640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51300a6130 0x7f51300a85f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.962+0000 7f514fd15640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5148106ed0 0x7f514806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.962+0000 7f514fd15640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5148101a40 0x7f514806c8c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.962+0000 7f514fd15640 1 --2- 172.21.15.57:0/3317848035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51481010a0 0x7f51480737a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:19.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.962+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 >> 172.21.15.57:0/3317848035 conn(0x7f5148077c40 msgr2=0x7f51480ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.967+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 shutdown_connections 2024-04-21T08:56:19.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:19.968+0000 7f514fd15640 1 -- 172.21.15.57:0/3317848035 wait complete. 2024-04-21T08:56:19.986 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-21T08:56:19.987 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:56:20.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.183+0000 7f316fae0640 1 -- 172.21.15.57:0/389713452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168109b70 msgr2=0x7f316811d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.183+0000 7f316fae0640 1 --2- 172.21.15.57:0/389713452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168109b70 0x7f316811d7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3164009860 tx=0x7f316402bb90 comp rx=0 tx=0).stop 2024-04-21T08:56:20.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.187+0000 7f316fae0640 1 -- 172.21.15.57:0/389713452 shutdown_connections 2024-04-21T08:56:20.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.187+0000 7f316fae0640 1 --2- 172.21.15.57:0/389713452 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168109b70 0x7f316811d7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.187+0000 7f316fae0640 1 --2- 172.21.15.57:0/389713452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3168110b30 0x7f3168109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.187+0000 7f316fae0640 1 --2- 172.21.15.57:0/389713452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31681106c0 0x7f3168108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.187+0000 7f316fae0640 1 -- 172.21.15.57:0/389713452 >> 172.21.15.57:0/389713452 conn(0x7f316806e410 msgr2=0x7f31680719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:20.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.188+0000 7f316fae0640 1 -- 172.21.15.57:0/389713452 shutdown_connections 2024-04-21T08:56:20.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.188+0000 7f316fae0640 1 -- 172.21.15.57:0/389713452 wait complete. 2024-04-21T08:56:20.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.189+0000 7f316fae0640 1 Processor -- start 2024-04-21T08:56:20.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 -- start start 2024-04-21T08:56:20.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31681106c0 0x7f316811d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 0x7f3168119480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31681199c0 0x7f3168119e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f31681137b0 con 0x7f31681199c0 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3168113630 con 0x7f31681106c0 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.191+0000 7f316fae0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3168113930 con 0x7f3168110b30 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.193+0000 7f316d07b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 0x7f3168119480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.193+0000 7f316d07b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 0x7f3168119480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43564/0 (socket says 172.21.15.57:43564) 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.193+0000 7f316d07b640 1 -- 172.21.15.57:0/1897783078 learned_addr learned my addr 172.21.15.57:0/1897783078 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.194+0000 7f316d07b640 1 -- 172.21.15.57:0/1897783078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31681106c0 msgr2=0x7f316811d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.194+0000 7f316d07b640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31681106c0 0x7f316811d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.194+0000 7f316d07b640 1 -- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31681199c0 msgr2=0x7f3168119e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.194+0000 7f316d07b640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31681199c0 0x7f3168119e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.194+0000 7f316d07b640 1 -- 172.21.15.57:0/1897783078 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31681ab4d0 con 0x7f3168110b30 2024-04-21T08:56:20.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.194+0000 7f316d07b640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 0x7f3168119480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f316000b9e0 tx=0x7f316000bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:20.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.199+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3160019070 con 0x7f3168110b30 2024-04-21T08:56:20.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.199+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f316000cba0 con 0x7f3168110b30 2024-04-21T08:56:20.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.199+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600125a0 con 0x7f3168110b30 2024-04-21T08:56:20.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.200+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31681ab760 con 0x7f3168110b30 2024-04-21T08:56:20.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.205+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31681abc30 con 0x7f3168110b30 2024-04-21T08:56:20.206 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.207+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f316001c070 con 0x7f3168110b30 2024-04-21T08:56:20.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.207+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3168073e40 con 0x7f3168110b30 2024-04-21T08:56:20.209 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.209+0000 7f315effd640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31540a3f20 0x7f31540a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.209 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.209+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f31600d0c40 con 0x7f3168110b30 2024-04-21T08:56:20.211 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.211+0000 7f316d87c640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31540a3f20 0x7f31540a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.212 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.212+0000 7f316d87c640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31540a3f20 0x7f31540a63e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3158005fd0 tx=0x7f3158005a00 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:20.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.219+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f316009cf80 con 0x7f3168110b30 2024-04-21T08:56:20.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:20 smithi136 ceph-mon[31791]: pgmap v343: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-04-21T08:56:20.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.667+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3168109b70 con 0x7f3168110b30 2024-04-21T08:56:20.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.668+0000 7f315effd640 1 -- 172.21.15.57:0/1897783078 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f316009c880 con 0x7f3168110b30 2024-04-21T08:56:20.668 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:56:20.668 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:56:20.668 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:56:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.671+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31540a3f20 msgr2=0x7f31540a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.671+0000 7f316fae0640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31540a3f20 0x7f31540a63e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3158005fd0 tx=0x7f3158005a00 comp rx=0 tx=0).stop 2024-04-21T08:56:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.671+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 msgr2=0x7f3168119480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.671+0000 7f316fae0640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 0x7f3168119480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f316000b9e0 tx=0x7f316000bea0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 shutdown_connections 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f31540a3f20 0x7f31540a63e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f31681199c0 0x7f3168119e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3168110b30 0x7f3168119480 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 --2- 172.21.15.57:0/1897783078 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31681106c0 0x7f316811d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 >> 172.21.15.57:0/1897783078 conn(0x7f316806e410 msgr2=0x7f316806ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:20.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 shutdown_connections 2024-04-21T08:56:20.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.672+0000 7f316fae0640 1 -- 172.21.15.57:0/1897783078 wait complete. 2024-04-21T08:56:20.690 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:20 smithi057 ceph-mon[27968]: pgmap v343: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-04-21T08:56:20.691 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-21T08:56:20.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:20 smithi129 ceph-mon[31272]: pgmap v343: 129 pgs: 19 active+clean+snaptrim, 52 active+clean+snaptrim_wait, 58 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 1.8 MiB/s wr, 155 op/s 2024-04-21T08:56:20.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.886+0000 7f63b7cc8640 1 -- 172.21.15.57:0/4182270959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0106160 msgr2=0x7f63b010dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.886+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/4182270959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0106160 0x7f63b010dd30 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009ca0 tx=0x7f63a802c140 comp rx=0 tx=0).stop 2024-04-21T08:56:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.887+0000 7f63b7cc8640 1 -- 172.21.15.57:0/4182270959 shutdown_connections 2024-04-21T08:56:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.887+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/4182270959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0106160 0x7f63b010dd30 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.887+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/4182270959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63b0101a40 0x7f63b0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.887+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/4182270959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0101090 0x7f63b0101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.887+0000 7f63b7cc8640 1 -- 172.21.15.57:0/4182270959 >> 172.21.15.57:0/4182270959 conn(0x7f63b0077f50 msgr2=0x7f63b0078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:20.890 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.889+0000 7f63b7cc8640 1 -- 172.21.15.57:0/4182270959 shutdown_connections 2024-04-21T08:56:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.892+0000 7f63b7cc8640 1 -- 172.21.15.57:0/4182270959 wait complete. 2024-04-21T08:56:20.893 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.893+0000 7f63b7cc8640 1 Processor -- start 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 -- start start 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63b0101090 0x7f63b01165d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 0x7f63b0116b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0111730 0x7f63b0111be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0110100 con 0x7f63b0111730 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f63b010ff80 con 0x7f63b0101090 2024-04-21T08:56:20.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.895+0000 7f63b7cc8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f63b0110280 con 0x7f63b0106160 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 0x7f63b0116b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 0x7f63b0116b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43586/0 (socket says 172.21.15.57:43586) 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 -- 172.21.15.57:0/2561708221 learned_addr learned my addr 172.21.15.57:0/2561708221 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 -- 172.21.15.57:0/2561708221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63b0101090 msgr2=0x7f63b01165d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63b0101090 0x7f63b01165d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 -- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0111730 msgr2=0x7f63b0111be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0111730 0x7f63b0111be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.896+0000 7f63b5263640 1 -- 172.21.15.57:0/2561708221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b01123f0 con 0x7f63b0106160 2024-04-21T08:56:20.899 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.897+0000 7f63b5263640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 0x7f63b0116b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b4a0 tx=0x7f63a000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:20.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.899+0000 7f63a6ffd640 1 -- 172.21.15.57:0/2561708221 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0014070 con 0x7f63b0106160 2024-04-21T08:56:20.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.899+0000 7f63a6ffd640 1 -- 172.21.15.57:0/2561708221 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0010ac0 con 0x7f63b0106160 2024-04-21T08:56:20.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.899+0000 7f63a6ffd640 1 -- 172.21.15.57:0/2561708221 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0017b60 con 0x7f63b0106160 2024-04-21T08:56:20.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.901+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b011a2d0 con 0x7f63b0106160 2024-04-21T08:56:20.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.901+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b011a770 con 0x7f63b0106160 2024-04-21T08:56:20.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.907+0000 7f63ba996740 1 Processor -- start 2024-04-21T08:56:20.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.907+0000 7f63a6ffd640 1 -- 172.21.15.57:0/2561708221 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f63a0017070 con 0x7f63b0106160 2024-04-21T08:56:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.908+0000 7f63a6ffd640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f639c0a3f30 0x7f639c0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.909 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.908+0000 7f63a6ffd640 1 -- 172.21.15.57:0/2561708221 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f63a00d01d0 con 0x7f63b0106160 2024-04-21T08:56:20.911 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63b6265640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f639c0a3f30 0x7f639c0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 -- start start 2024-04-21T08:56:20.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.913 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b5ae82e1f0 con 0x55b5ae8c9e80 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b5ae7cb4b0 con 0x55b5ae8c94d0 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.912+0000 7f63ba996740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b5ae82a1b0 con 0x55b5ae8d2a00 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:52462/0 (socket says 172.21.15.57:52462) 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 -- 172.21.15.57:0/2986218173 learned_addr learned my addr 172.21.15.57:0/2986218173 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:20.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 -- 172.21.15.57:0/2986218173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 msgr2=0x55b5ae8d4e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-21T08:56:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 -- 172.21.15.57:0/2986218173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 msgr2=0x55b5ae8c98b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-21T08:56:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8c98b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 -- 172.21.15.57:0/2986218173 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ae7bd1b0 con 0x55b5ae8c9e80 2024-04-21T08:56:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.913+0000 7f63b5263640 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d2430 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f63a00987e0 tx=0x7f63a009b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1b8744602e7cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:20.915 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.915+0000 7f63b6265640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f639c0a3f30 0x7f639c0a63f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63a8006010 tx=0x7f63a8005fa0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:20.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.915+0000 7f639bfff640 1 -- 172.21.15.57:0/2986218173 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a00b5070 con 0x55b5ae8c9e80 2024-04-21T08:56:20.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.915+0000 7f639bfff640 1 -- 172.21.15.57:0/2986218173 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0017d00 con 0x55b5ae8c9e80 2024-04-21T08:56:20.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.915+0000 7f639bfff640 1 -- 172.21.15.57:0/2986218173 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a001e070 con 0x55b5ae8c9e80 2024-04-21T08:56:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.920+0000 7f63ba996740 1 -- 172.21.15.57:0/2986218173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 msgr2=0x55b5ae8d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.921 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.920+0000 7f63ba996740 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d2430 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f63a00987e0 tx=0x7f63a009b8a0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.925 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.926+0000 7f63ba996740 1 -- 172.21.15.57:0/2986218173 shutdown_connections 2024-04-21T08:56:20.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.926+0000 7f63ba996740 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.926+0000 7f63ba996740 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d2430 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.926+0000 7f63ba996740 1 --2- 172.21.15.57:0/2986218173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.926 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.926+0000 7f63ba996740 1 -- 172.21.15.57:0/2986218173 >> 172.21.15.57:0/2986218173 conn(0x55b5ae8c3ba0 msgr2=0x55b5ae8c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:20.933 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.933+0000 7f63ba996740 1 -- 172.21.15.57:0/2986218173 shutdown_connections 2024-04-21T08:56:20.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.936+0000 7f63ba996740 1 -- 172.21.15.57:0/2986218173 wait complete. 2024-04-21T08:56:20.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.936+0000 7f63ba996740 1 -- start start 2024-04-21T08:56:20.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.936+0000 7f63ba996740 1 Processor -- start 2024-04-21T08:56:20.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.937 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x55b5ae8d5970 con 0x55b5ae8c9e80 2024-04-21T08:56:20.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b5ae7e7500 con 0x55b5ae8c94d0 2024-04-21T08:56:20.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b5ae8d5af0 con 0x55b5ae8d2a00 2024-04-21T08:56:20.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63ba996740 10 client.0 _ll_register_callbacks cb 0x55b5ae7b5b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:20.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.937+0000 7f63b6265640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.938+0000 7f63b6265640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43594/0 (socket says 172.21.15.57:43594) 2024-04-21T08:56:20.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.938+0000 7f63b6265640 1 -- 172.21.15.57:0/2573276510 learned_addr learned my addr 172.21.15.57:0/2573276510 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:20.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.940+0000 7f63b5263640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.941+0000 7f63b5a64640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.941+0000 7f63b6265640 1 -- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 msgr2=0x55b5ae8d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:20.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.941+0000 7f63b6265640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.941+0000 7f63b6265640 1 -- 172.21.15.57:0/2573276510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 msgr2=0x55b5ae8cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:20.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.941+0000 7f63b6265640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.941+0000 7f63b6265640 1 -- 172.21.15.57:0/2573276510 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ae5403f0 con 0x55b5ae8d2a00 2024-04-21T08:56:20.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.945+0000 7f63b6265640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cfc50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f63a8032e30 tx=0x7f63a802c140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84f840dcf8f62ad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:20.945 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.945+0000 7f63997fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:56:20.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.947+0000 7f63997fa640 1 -- 172.21.15.57:0/2573276510 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a804f060 con 0x55b5ae8d2a00 2024-04-21T08:56:20.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.947+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a0017d00 con 0x55b5ae8d2a00 2024-04-21T08:56:20.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.947+0000 7f63997fa640 1 -- 172.21.15.57:0/2573276510 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f63a8003b70 con 0x55b5ae8d2a00 2024-04-21T08:56:20.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.948+0000 7f63997fa640 1 -- 172.21.15.57:0/2573276510 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f63a8056050 con 0x55b5ae8d2a00 2024-04-21T08:56:20.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.951+0000 7f63ba996740 10 client.26662 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:20.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.951+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b5ae60a7d0 con 0x55b5ae8d2a00 2024-04-21T08:56:20.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.951+0000 7f63997fa640 1 -- 172.21.15.57:0/2573276510 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0017d00 con 0x55b5ae8d2a00 2024-04-21T08:56:20.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.951+0000 7f63997fa640 1 -- 172.21.15.57:0/2573276510 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f63a803c030 con 0x55b5ae8d2a00 2024-04-21T08:56:20.952 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.953+0000 7f63ba996740 10 client.26662 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.953+0000 7f63ba996740 10 client.26662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.953+0000 7f63ba996740 20 client.26662 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.953+0000 7f63ba996740 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae8e2c70 0x55b5ae8e50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.953+0000 7f63ba996740 4 client.26662 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-21T08:56:20.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.953+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ae7cd930 con 0x55b5ae8e2c70 2024-04-21T08:56:20.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.959+0000 7f63b5a64640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae8e2c70 0x55b5ae8e50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.959+0000 7f63b5a64640 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae8e2c70 0x55b5ae8e50e0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e15d6bf173413ff7 server_cookie=5875b5d34e132af9 in_seq=0 out_seq=0 2024-04-21T08:56:20.959 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.959+0000 7f63997fa640 10 client.26662 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.961+0000 7f63997fa640 1 -- 172.21.15.57:0/2573276510 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5ae7cd930 con 0x55b5ae8e2c70 2024-04-21T08:56:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.961+0000 7f63997fa640 10 client.26662 handle_command_reply: tid=0 2024-04-21T08:56:20.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.961+0000 7f63ba996740 1 client.26662 shutdown 2024-04-21T08:56:20.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 msgr2=0x55b5ae8cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cfc50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f63a8032e30 tx=0x7f63a802c140 comp rx=0 tx=0).stop 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 shutdown_connections 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae8e2c70 0x55b5ae8e50e0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cfc50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 --2- 172.21.15.57:0/2573276510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:20.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.963+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 >> 172.21.15.57:0/2573276510 conn(0x55b5ae8c3ba0 msgr2=0x55b5ae8c58f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:20.964 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.965+0000 7f63997fa640 0 client.26662 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:20.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.966+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 shutdown_connections 2024-04-21T08:56:20.966 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.967+0000 7f63ba996740 1 -- 172.21.15.57:0/2573276510 wait complete. 2024-04-21T08:56:20.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.967+0000 7f63ba996740 20 client.26662 trim_cache size 0 max 16384 2024-04-21T08:56:20.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.997+0000 7f63ba996740 1 Processor -- start 2024-04-21T08:56:20.997 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.997+0000 7f63ba996740 1 -- start start 2024-04-21T08:56:20.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.998+0000 7f63ba996740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8de890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.998+0000 7f63ba996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.998+0000 7f63ba996740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.998+0000 7f63ba996740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f639400e100 con 0x55b5ae8c94d0 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.998+0000 7f63ba996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6394010960 con 0x55b5ae8c9e80 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.998+0000 7f63ba996740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f639400e750 con 0x55b5ae8d2a00 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.999+0000 7f63b6265640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.999+0000 7f63b6265640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43606/0 (socket says 172.21.15.57:43606) 2024-04-21T08:56:20.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:20.999+0000 7f63b6265640 1 -- 172.21.15.57:0/2509645341 learned_addr learned my addr 172.21.15.57:0/2509645341 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:21.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.000+0000 7f63b5263640 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.000+0000 7f63b5a64640 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.004+0000 7f63b6265640 1 -- 172.21.15.57:0/2509645341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c9e80 msgr2=0x55b5ae8d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:21.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.004+0000 7f63b6265640 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.004+0000 7f63b6265640 1 -- 172.21.15.57:0/2509645341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c94d0 msgr2=0x55b5ae8de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:21.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.006+0000 7f63b6265640 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8de890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.006+0000 7f63b6265640 1 -- 172.21.15.57:0/2509645341 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ae8bf470 con 0x55b5ae8d2a00 2024-04-21T08:56:21.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.006+0000 7f63b6265640 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d2320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f63a8037040 tx=0x7f63a80325d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7992497b7fe53525 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:21.006 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.006+0000 7f63b5a64640 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:56:21.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.009+0000 7f639b7fe640 1 -- 172.21.15.57:0/2509645341 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a804f040 con 0x55b5ae8d2a00 2024-04-21T08:56:21.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.009+0000 7f639b7fe640 1 -- 172.21.15.57:0/2509645341 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a8032920 con 0x55b5ae8d2a00 2024-04-21T08:56:21.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.009+0000 7f639b7fe640 1 -- 172.21.15.57:0/2509645341 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8039040 con 0x55b5ae8d2a00 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.010+0000 7f63ba996740 1 -- 172.21.15.57:0/2509645341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 msgr2=0x55b5ae8d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.010+0000 7f63ba996740 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d2320 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f63a8037040 tx=0x7f63a80325d0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.011+0000 7f63ba996740 1 -- 172.21.15.57:0/2509645341 shutdown_connections 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.011+0000 7f63ba996740 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8d2320 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.011+0000 7f63ba996740 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.011+0000 7f63ba996740 1 --2- 172.21.15.57:0/2509645341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.011+0000 7f63ba996740 1 -- 172.21.15.57:0/2509645341 >> 172.21.15.57:0/2509645341 conn(0x55b5ae8c3ba0 msgr2=0x55b5ae8cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:21.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.012+0000 7f63ba996740 1 -- 172.21.15.57:0/2509645341 shutdown_connections 2024-04-21T08:56:21.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.012+0000 7f63ba996740 1 -- 172.21.15.57:0/2509645341 wait complete. 2024-04-21T08:56:21.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 -- start start 2024-04-21T08:56:21.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 Processor -- start 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8db210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f639400ee20 con 0x55b5ae8d2a00 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6394010290 con 0x55b5ae8c94d0 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f639400f4f0 con 0x55b5ae8c9e80 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.013+0000 7f63ba996740 10 client.0 _ll_register_callbacks cb 0x7f63a8032cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-21T08:56:21.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43618/0 (socket says 172.21.15.57:43618) 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 -- 172.21.15.57:0/364880000 learned_addr learned my addr 172.21.15.57:0/364880000 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5a64640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8db210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b6265640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 -- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 msgr2=0x55b5ae8db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8db210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 -- 172.21.15.57:0/364880000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8d2a00 msgr2=0x55b5ae8cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.017 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.014+0000 7f63b5263640 1 -- 172.21.15.57:0/364880000 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ae8cced0 con 0x55b5ae8c9e80 2024-04-21T08:56:21.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.016+0000 7f63b5263640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cc140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63a000be60 tx=0x7f63a009b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8756d68e0ee7056a server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:21.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.016+0000 7f63997fa640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-21T08:56:21.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.016+0000 7f63997fa640 1 -- 172.21.15.57:0/364880000 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a00a9060 con 0x55b5ae8c9e80 2024-04-21T08:56:21.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.016+0000 7f63997fa640 1 -- 172.21.15.57:0/364880000 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f63a0098480 con 0x55b5ae8c9e80 2024-04-21T08:56:21.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.016+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a8032920 con 0x55b5ae8c9e80 2024-04-21T08:56:21.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.017+0000 7f63997fa640 1 -- 172.21.15.57:0/364880000 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f63a001e040 con 0x55b5ae8c9e80 2024-04-21T08:56:21.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.018+0000 7f63ba996740 10 client.26668 fetch_fsmap learned FSMap version 21 2024-04-21T08:56:21.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.018+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=4+}) -- 0x55b5ae96b300 con 0x55b5ae8c9e80 2024-04-21T08:56:21.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.022+0000 7f63997fa640 1 -- 172.21.15.57:0/364880000 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a8032920 con 0x55b5ae8c9e80 2024-04-21T08:56:21.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.022+0000 7f63997fa640 1 -- 172.21.15.57:0/364880000 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 21) ==== 8330+0+0 (secure 0 0 0) 0x7f63a009b260 con 0x55b5ae8c9e80 2024-04-21T08:56:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.027+0000 7f63ba996740 10 client.26668 fetch_fsmap finished waiting for FSMap version 21 2024-04-21T08:56:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.027+0000 7f63ba996740 10 client.26668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-21T08:56:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.027+0000 7f63ba996740 20 client.26668 populate_metadata read hostname 'smithi057' 2024-04-21T08:56:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.027+0000 7f63ba996740 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae96b590 0x55b5ae96d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.027 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.027+0000 7f63ba996740 4 client.26668 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-21T08:56:21.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.027+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 --> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5ae83bc10 con 0x55b5ae96b590 2024-04-21T08:56:21.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.029+0000 7f63b5a64640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae96b590 0x55b5ae96d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.037+0000 7f63b5a64640 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae96b590 0x55b5ae96d980 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8557f21b0c522120 server_cookie=60684c153d166834 in_seq=0 out_seq=0 2024-04-21T08:56:21.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.037+0000 7f63997fa640 10 client.26668 ms_handle_connect on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.037+0000 7f63997fa640 1 -- 172.21.15.57:0/364880000 <== mds.0 v2:172.21.15.129:6838/123336774 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5ae83bc10 con 0x55b5ae96b590 2024-04-21T08:56:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.037+0000 7f63997fa640 10 client.26668 handle_command_reply: tid=0 2024-04-21T08:56:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.038+0000 7f63ba996740 1 client.26668 shutdown 2024-04-21T08:56:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.039+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 msgr2=0x55b5ae8cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.039+0000 7f63ba996740 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cc140 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63a000be60 tx=0x7f63a009b770 comp rx=0 tx=0).stop 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 shutdown_connections 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:6838/123336774,v1:172.21.15.129:6839/123336774] conn(0x55b5ae96b590 0x55b5ae96d980 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x55b5ae8d2a00 0x55b5ae8cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b5ae8c9e80 0x55b5ae8cc140 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 --2- 172.21.15.57:0/364880000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b5ae8c94d0 0x55b5ae8db210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 >> 172.21.15.57:0/364880000 conn(0x55b5ae8c3ba0 msgr2=0x55b5ae8ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:21.040 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63997fa640 0 client.26668 ms_handle_reset on v2:172.21.15.129:6838/123336774 2024-04-21T08:56:21.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 shutdown_connections 2024-04-21T08:56:21.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 1 -- 172.21.15.57:0/364880000 wait complete. 2024-04-21T08:56:21.041 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.040+0000 7f63ba996740 20 client.26668 trim_cache size 0 max 16384 2024-04-21T08:56:21.041 INFO:teuthology.orchestra.run.smithi057.stdout:[] 2024-04-21T08:56:21.042 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.042+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f639c0a3f30 msgr2=0x7f639c0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.042 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f639c0a3f30 0x7f639c0a63f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f63a8006010 tx=0x7f63a8005fa0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 msgr2=0x7f63b0116b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 0x7f63b0116b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b4a0 tx=0x7f63a000b960 comp rx=0 tx=0).stop 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 shutdown_connections 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f639c0a3f30 0x7f639c0a63f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63b0111730 0x7f63b0111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b0106160 0x7f63b0116b10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 --2- 172.21.15.57:0/2561708221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63b0101090 0x7f63b01165d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.043 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.043+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 >> 172.21.15.57:0/2561708221 conn(0x7f63b0077f50 msgr2=0x7f63b00fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:21.045 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.046+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 shutdown_connections 2024-04-21T08:56:21.046 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.046+0000 7f63b7cc8640 1 -- 172.21.15.57:0/2561708221 wait complete. 2024-04-21T08:56:21.067 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/thrasher.py", line 43, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-04-21T08:56:21.071 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 152, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 310, in task stop_all_quiescers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/quiescer.py", line 263, in stop_all_quiescers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Error releasing root: -124 (Unknown) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_leonid-s-usov_ceph-ceph_4f6bc5273643aaf5660909e85400647a8951c10b/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-21T08:56:21.076 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-21T08:56:21.088 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:56:21.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.272+0000 7f495c3e6640 1 -- 172.21.15.57:0/2048054119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4954105e30 msgr2=0x7f4954106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.272 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.272+0000 7f495c3e6640 1 --2- 172.21.15.57:0/2048054119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4954105e30 0x7f4954106210 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f494c009cd0 tx=0x7f494c02c120 comp rx=0 tx=0).stop 2024-04-21T08:56:21.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.276+0000 7f495c3e6640 1 -- 172.21.15.57:0/2048054119 shutdown_connections 2024-04-21T08:56:21.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.276+0000 7f495c3e6640 1 --2- 172.21.15.57:0/2048054119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954101b60 0x7f495410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.277 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.276+0000 7f495c3e6640 1 --2- 172.21.15.57:0/2048054119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49541011a0 0x7f4954101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.276+0000 7f495c3e6640 1 --2- 172.21.15.57:0/2048054119 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4954105e30 0x7f4954106210 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.278 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.276+0000 7f495c3e6640 1 -- 172.21.15.57:0/2048054119 >> 172.21.15.57:0/2048054119 conn(0x7f4954077e30 msgr2=0x7f4954078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:21.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.286+0000 7f495c3e6640 1 -- 172.21.15.57:0/2048054119 shutdown_connections 2024-04-21T08:56:21.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.287+0000 7f495c3e6640 1 -- 172.21.15.57:0/2048054119 wait complete. 2024-04-21T08:56:21.287 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.287+0000 7f495c3e6640 1 Processor -- start 2024-04-21T08:56:21.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 -- start start 2024-04-21T08:56:21.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49541011a0 0x7f495419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4954101b60 0x7f495419fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954105e30 0x7f495410cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4954112350 con 0x7f49541011a0 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f49541121d0 con 0x7f4954101b60 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495c3e6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f49541124d0 con 0x7f4954105e30 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f4959981640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4954101b60 0x7f495419fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f4959981640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4954101b60 0x7f495419fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:34888/0 (socket says 172.21.15.57:34888) 2024-04-21T08:56:21.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f4959981640 1 -- 172.21.15.57:0/3417726131 learned_addr learned my addr 172.21.15.57:0/3417726131 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495a983640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954105e30 0x7f495410cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.290+0000 7f495a182640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49541011a0 0x7f495419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.291+0000 7f495a983640 1 -- 172.21.15.57:0/3417726131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4954101b60 msgr2=0x7f495419fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.291+0000 7f495a983640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4954101b60 0x7f495419fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.291+0000 7f495a983640 1 -- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49541011a0 msgr2=0x7f495419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.291+0000 7f495a983640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49541011a0 0x7f495419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.291+0000 7f495a983640 1 -- 172.21.15.57:0/3417726131 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495410d8b0 con 0x7f4954105e30 2024-04-21T08:56:21.293 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.291+0000 7f495a983640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954105e30 0x7f495410cff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f495000b3e0 tx=0x7f495000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:21.294 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.295+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950015020 con 0x7f4954105e30 2024-04-21T08:56:21.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.295+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4950011ce0 con 0x7f4954105e30 2024-04-21T08:56:21.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.295+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950010e50 con 0x7f4954105e30 2024-04-21T08:56:21.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.295+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495410dba0 con 0x7f4954105e30 2024-04-21T08:56:21.297 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.295+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495419b110 con 0x7f4954105e30 2024-04-21T08:56:21.301 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.298+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4938005290 con 0x7f4954105e30 2024-04-21T08:56:21.302 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.302+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4950018020 con 0x7f4954105e30 2024-04-21T08:56:21.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.303+0000 7f494b7fe640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f49400a40d0 0x7f49400a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.303+0000 7f495a182640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f49400a40d0 0x7f49400a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.303 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.303+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f4950010500 con 0x7f4954105e30 2024-04-21T08:56:21.304 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.304+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f495001f020 con 0x7f4954105e30 2024-04-21T08:56:21.310 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.307+0000 7f495a182640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f49400a40d0 0x7f49400a6590 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f494c006010 tx=0x7f494c000f40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:21.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:21 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1897783078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:21.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.671+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4938005850 con 0x7f4954105e30 2024-04-21T08:56:21.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.675+0000 7f494b7fe640 1 -- 172.21.15.57:0/3417726131 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+25395 (secure 0 0 0) 0x7f495009d510 con 0x7f4954105e30 2024-04-21T08:56:21.675 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:56:21.675 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":107,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:56:07.247129+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:56:21.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f49400a40d0 msgr2=0x7f49400a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f49400a40d0 0x7f49400a6590 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f494c006010 tx=0x7f494c000f40 comp rx=0 tx=0).stop 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954105e30 msgr2=0x7f495410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954105e30 0x7f495410cff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f495000b3e0 tx=0x7f495000b8a0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 shutdown_connections 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f49400a40d0 0x7f49400a6590 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4954105e30 0x7f495410cff0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4954101b60 0x7f495419fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 --2- 172.21.15.57:0/3417726131 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49541011a0 0x7f495419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 >> 172.21.15.57:0/3417726131 conn(0x7f4954077e30 msgr2=0x7f495410e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 shutdown_connections 2024-04-21T08:56:21.680 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.678+0000 7f495c3e6640 1 -- 172.21.15.57:0/3417726131 wait complete. 2024-04-21T08:56:21.694 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:21 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1897783078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:21.696 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:56:21.697 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:56:21.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:21 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1897783078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:56:21.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.880+0000 7fe98bfff640 1 -- 172.21.15.57:0/3034338351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 msgr2=0x7fe9840068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.880+0000 7fe98bfff640 1 --2- 172.21.15.57:0/3034338351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840068d0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe97c01a910 tx=0x7fe97c03cca0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.885+0000 7fe98bfff640 1 -- 172.21.15.57:0/3034338351 shutdown_connections 2024-04-21T08:56:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.885+0000 7fe98bfff640 1 --2- 172.21.15.57:0/3034338351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984098730 0x7fe984098bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.885+0000 7fe98bfff640 1 --2- 172.21.15.57:0/3034338351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe984006e10 0x7fe984007270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.885+0000 7fe98bfff640 1 --2- 172.21.15.57:0/3034338351 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840068d0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.885+0000 7fe98bfff640 1 -- 172.21.15.57:0/3034338351 >> 172.21.15.57:0/3034338351 conn(0x7fe98400a7f0 msgr2=0x7fe98400ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:21.891 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.890+0000 7fe98bfff640 1 -- 172.21.15.57:0/3034338351 shutdown_connections 2024-04-21T08:56:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.894+0000 7fe98bfff640 1 -- 172.21.15.57:0/3034338351 wait complete. 2024-04-21T08:56:21.895 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.894+0000 7fe98bfff640 1 Processor -- start 2024-04-21T08:56:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 -- start start 2024-04-21T08:56:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 0x7fe984140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe984098730 0x7fe9841413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe98409db50 con 0x7fe98409aca0 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe98409d9d0 con 0x7fe984098730 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.899+0000 7fe98bfff640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe98409dcd0 con 0x7fe984006e10 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 0x7fe984140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 0x7fe984140e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:43632/0 (socket says 172.21.15.57:43632) 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 -- 172.21.15.57:0/2864054992 learned_addr learned my addr 172.21.15.57:0/2864054992 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:21.901 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98b7fe640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98a7fc640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe984098730 0x7fe9841413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 -- 172.21.15.57:0/2864054992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe984098730 msgr2=0x7fe9841413d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.905 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe984098730 0x7fe9841413d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 -- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 msgr2=0x7fe9840a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.900+0000 7fe98affd640 1 -- 172.21.15.57:0/2864054992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9840a58b0 con 0x7fe984006e10 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.901+0000 7fe98b7fe640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.901+0000 7fe98affd640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 0x7fe984140e90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe97c03d2b0 tx=0x7fe97c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.904+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c052020 con 0x7fe984006e10 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.904+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9840a5ba0 con 0x7fe984006e10 2024-04-21T08:56:21.906 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.904+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe984147db0 con 0x7fe984006e10 2024-04-21T08:56:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.905+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe97c03dd00 con 0x7fe984006e10 2024-04-21T08:56:21.908 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.905+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c003b50 con 0x7fe984006e10 2024-04-21T08:56:21.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.909+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe978005290 con 0x7fe984006e10 2024-04-21T08:56:21.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.915+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe97c055020 con 0x7fe984006e10 2024-04-21T08:56:21.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.916+0000 7fe973fff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe9740a3f20 0x7fe9740a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:21.916 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.916+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7fe97c103fe0 con 0x7fe984006e10 2024-04-21T08:56:21.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.917+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe97c05c020 con 0x7fe984006e10 2024-04-21T08:56:21.917 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.918+0000 7fe98a7fc640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe9740a3f20 0x7fe9740a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:21.918 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:21.919+0000 7fe98a7fc640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe9740a3f20 0x7fe9740a63e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe980005fd0 tx=0x7fe980005850 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:22.285 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.285+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe978005850 con 0x7fe984006e10 2024-04-21T08:56:22.286 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.286+0000 7fe973fff640 1 -- 172.21.15.57:0/2864054992 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7fe97c0d03a0 con 0x7fe984006e10 2024-04-21T08:56:22.287 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:56:22.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.290+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe9740a3f20 msgr2=0x7fe9740a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.290+0000 7fe98bfff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe9740a3f20 0x7fe9740a63e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe980005fd0 tx=0x7fe980005850 comp rx=0 tx=0).stop 2024-04-21T08:56:22.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.290+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 msgr2=0x7fe984140e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.291 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.290+0000 7fe98bfff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 0x7fe984140e90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe97c03d2b0 tx=0x7fe97c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.291+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 shutdown_connections 2024-04-21T08:56:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.291+0000 7fe98bfff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe9740a3f20 0x7fe9740a63e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.291+0000 7fe98bfff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe98409aca0 0x7fe9840a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.291+0000 7fe98bfff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe984098730 0x7fe9841413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.291+0000 7fe98bfff640 1 --2- 172.21.15.57:0/2864054992 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe984006e10 0x7fe984140e90 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe97c03d2b0 tx=0x7fe97c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.292 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.291+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 >> 172.21.15.57:0/2864054992 conn(0x7fe98400a7f0 msgr2=0x7fe9840974b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.296+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 shutdown_connections 2024-04-21T08:56:22.295 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.296+0000 7fe98bfff640 1 -- 172.21.15.57:0/2864054992 wait complete. 2024-04-21T08:56:22.312 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:56:22.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.490+0000 7f517f550640 1 -- 172.21.15.57:0/2876307954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178101090 msgr2=0x7f5178101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.490+0000 7f517f550640 1 --2- 172.21.15.57:0/2876307954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178101090 0x7f5178101470 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f516c009900 tx=0x7f516c02bc60 comp rx=0 tx=0).stop 2024-04-21T08:56:22.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.491+0000 7f517f550640 1 -- 172.21.15.57:0/2876307954 shutdown_connections 2024-04-21T08:56:22.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.491+0000 7f517f550640 1 --2- 172.21.15.57:0/2876307954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178106160 0x7f517810dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.491+0000 7f517f550640 1 --2- 172.21.15.57:0/2876307954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101a40 0x7f5178105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.491+0000 7f517f550640 1 --2- 172.21.15.57:0/2876307954 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178101090 0x7f5178101470 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.491+0000 7f517f550640 1 -- 172.21.15.57:0/2876307954 >> 172.21.15.57:0/2876307954 conn(0x7f5178077f50 msgr2=0x7f5178078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.492+0000 7f517f550640 1 -- 172.21.15.57:0/2876307954 shutdown_connections 2024-04-21T08:56:22.492 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.492+0000 7f517f550640 1 -- 172.21.15.57:0/2876307954 wait complete. 2024-04-21T08:56:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 Processor -- start 2024-04-21T08:56:22.493 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 -- start start 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101090 0x7f5178116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178106160 0x7f51781108c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178110e00 0x7f51781112b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f517810f180 con 0x7f5178106160 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f517810f000 con 0x7f5178110e00 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.493+0000 7f517f550640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f517810f300 con 0x7f5178101090 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.494+0000 7f517daed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178110e00 0x7f51781112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:22.494 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.494+0000 7f517daed640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178110e00 0x7f51781112b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:34912/0 (socket says 172.21.15.57:34912) 2024-04-21T08:56:22.495 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.494+0000 7f517daed640 1 -- 172.21.15.57:0/2201320122 learned_addr learned my addr 172.21.15.57:0/2201320122 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.494+0000 7f517d2ec640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101090 0x7f5178116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.496+0000 7f517d2ec640 1 -- 172.21.15.57:0/2201320122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178110e00 msgr2=0x7f51781112b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.496+0000 7f517d2ec640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178110e00 0x7f51781112b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.496+0000 7f517d2ec640 1 -- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178106160 msgr2=0x7f51781108c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.496+0000 7f517d2ec640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178106160 0x7f51781108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.496+0000 7f517d2ec640 1 -- 172.21.15.57:0/2201320122 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5178071aa0 con 0x7f5178101090 2024-04-21T08:56:22.497 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.497+0000 7f517d2ec640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101090 0x7f5178116850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f516c009850 tx=0x7f516c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.497+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c043020 con 0x7f5178101090 2024-04-21T08:56:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.497+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5178071d30 con 0x7f5178101090 2024-04-21T08:56:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.497+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51780722a0 con 0x7f5178101090 2024-04-21T08:56:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.497+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f516c02cad0 con 0x7f5178101090 2024-04-21T08:56:22.498 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.497+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c004eb0 con 0x7f5178101090 2024-04-21T08:56:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.499+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5164005290 con 0x7f5178101090 2024-04-21T08:56:22.501 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.501+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f516c046020 con 0x7f5178101090 2024-04-21T08:56:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.501+0000 7f51767fc640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51600a3e60 0x7f51600a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:22.502 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.502+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f516c0f26a0 con 0x7f5178101090 2024-04-21T08:56:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.503+0000 7f517caeb640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51600a3e60 0x7f51600a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:22.504 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.505+0000 7f517caeb640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51600a3e60 0x7f51600a6320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5178112080 tx=0x7f5168009250 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:22.512 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.511+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f516c04d070 con 0x7f5178101090 2024-04-21T08:56:22.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:22 smithi057 ceph-mon[27968]: pgmap v344: 129 pgs: 18 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 65 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-21T08:56:22.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:22 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3417726131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:56:22.590 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:22 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2864054992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:56:22.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:22 smithi136 ceph-mon[31791]: pgmap v344: 129 pgs: 18 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 65 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-21T08:56:22.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:22 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3417726131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:56:22.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:22 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2864054992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:56:22.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:22 smithi129 ceph-mon[31272]: pgmap v344: 129 pgs: 18 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 65 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-04-21T08:56:22.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:22 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3417726131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:56:22.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:22 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2864054992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:56:22.870 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.870+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5164005850 con 0x7f5178101090 2024-04-21T08:56:22.872 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.872+0000 7f51767fc640 1 -- 172.21.15.57:0/2201320122 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7f516c02cc80 con 0x7f5178101090 2024-04-21T08:56:22.872 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:56:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51600a3e60 msgr2=0x7f51600a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51600a3e60 0x7f51600a6320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5178112080 tx=0x7f5168009250 comp rx=0 tx=0).stop 2024-04-21T08:56:22.875 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101090 msgr2=0x7f5178116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101090 0x7f5178116850 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f516c009850 tx=0x7f516c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 shutdown_connections 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f51600a3e60 0x7f51600a6320 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5178110e00 0x7f51781112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5178106160 0x7f51781108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 --2- 172.21.15.57:0/2201320122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5178101090 0x7f5178116850 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 >> 172.21.15.57:0/2201320122 conn(0x7f5178077f50 msgr2=0x7f51780ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.875+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 shutdown_connections 2024-04-21T08:56:22.876 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:22.876+0000 7f517f550640 1 -- 172.21.15.57:0/2201320122 wait complete. 2024-04-21T08:56:22.892 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:56:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.063+0000 7f3e92cba640 1 -- 172.21.15.57:0/3742650066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 msgr2=0x7f3e8c1013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.063+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3742650066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 0x7f3e8c1013a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009900 tx=0x7f3e8002bd90 comp rx=0 tx=0).stop 2024-04-21T08:56:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.064+0000 7f3e92cba640 1 -- 172.21.15.57:0/3742650066 shutdown_connections 2024-04-21T08:56:23.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.064+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3742650066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 0x7f3e8c106770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.064+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3742650066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 0x7f3e8c105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.064+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3742650066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 0x7f3e8c1013a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.064+0000 7f3e92cba640 1 -- 172.21.15.57:0/3742650066 >> 172.21.15.57:0/3742650066 conn(0x7f3e8c077f50 msgr2=0x7f3e8c078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:23.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.067+0000 7f3e92cba640 1 -- 172.21.15.57:0/3742650066 shutdown_connections 2024-04-21T08:56:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.067+0000 7f3e92cba640 1 -- 172.21.15.57:0/3742650066 wait complete. 2024-04-21T08:56:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 Processor -- start 2024-04-21T08:56:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 -- start start 2024-04-21T08:56:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 0x7f3e8c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.068 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 0x7f3e8c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 0x7f3e8c1a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c118910 con 0x7f3e8c100fc0 2024-04-21T08:56:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c118790 con 0x7f3e8c1018e0 2024-04-21T08:56:23.069 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e92cba640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c118a90 con 0x7f3e8c1062e0 2024-04-21T08:56:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.068+0000 7f3e91257640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 0x7f3e8c1a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e91257640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 0x7f3e8c1a37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:41898/0 (socket says 172.21.15.57:41898) 2024-04-21T08:56:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e91257640 1 -- 172.21.15.57:0/3294072716 learned_addr learned my addr 172.21.15.57:0/3294072716 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:23.070 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 0x7f3e8c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e90a56640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 0x7f3e8c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 -- 172.21.15.57:0/3294072716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 msgr2=0x7f3e8c1a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 0x7f3e8c1a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 -- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 msgr2=0x7f3e8c19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 0x7f3e8c19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 -- 172.21.15.57:0/3294072716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c1a3ee0 con 0x7f3e8c1018e0 2024-04-21T08:56:23.071 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.069+0000 7f3e8bfff640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 0x7f3e8c19f480 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400be20 tx=0x7f3e84009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:23.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.074+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8402a070 con 0x7f3e8c1018e0 2024-04-21T08:56:23.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.074+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e84025900 con 0x7f3e8c1018e0 2024-04-21T08:56:23.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.074+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84023990 con 0x7f3e8c1018e0 2024-04-21T08:56:23.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.074+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c1a41d0 con 0x7f3e8c1018e0 2024-04-21T08:56:23.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.074+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c06b970 con 0x7f3e8c1018e0 2024-04-21T08:56:23.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.076+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e8c068d70 con 0x7f3e8c1018e0 2024-04-21T08:56:23.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.077+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3e8402d070 con 0x7f3e8c1018e0 2024-04-21T08:56:23.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.078+0000 7f3e89ffb640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3e740a3f20 0x7f3e740a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.078+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f3e840e1340 con 0x7f3e8c1018e0 2024-04-21T08:56:23.079 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.079+0000 7f3e90a56640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3e740a3f20 0x7f3e740a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.084+0000 7f3e90a56640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3e740a3f20 0x7f3e740a63e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3e80000c00 tx=0x7f3e80005fd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:23.084 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.085+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e840ad030 con 0x7f3e8c1018e0 2024-04-21T08:56:23.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.446+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3e8c06bc30 con 0x7f3e8c1018e0 2024-04-21T08:56:23.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.447+0000 7f3e89ffb640 1 -- 172.21.15.57:0/3294072716 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f3e840b1330 con 0x7f3e8c1018e0 2024-04-21T08:56:23.447 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:56:23.450 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.451+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3e740a3f20 msgr2=0x7f3e740a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.451+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3e740a3f20 0x7f3e740a63e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3e80000c00 tx=0x7f3e80005fd0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.451+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 msgr2=0x7f3e8c19f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.451+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 0x7f3e8c19f480 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400be20 tx=0x7f3e84009f70 comp rx=0 tx=0).stop 2024-04-21T08:56:23.451 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.452+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 shutdown_connections 2024-04-21T08:56:23.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.452+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f3e740a3f20 0x7f3e740a63e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.452+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e8c1062e0 0x7f3e8c1a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.452+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e8c1018e0 0x7f3e8c19f480 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.452+0000 7f3e92cba640 1 --2- 172.21.15.57:0/3294072716 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e8c100fc0 0x7f3e8c19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.452 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.452+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 >> 172.21.15.57:0/3294072716 conn(0x7f3e8c077f50 msgr2=0x7f3e8c10fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:23.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.454+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 shutdown_connections 2024-04-21T08:56:23.454 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.454+0000 7f3e92cba640 1 -- 172.21.15.57:0/3294072716 wait complete. 2024-04-21T08:56:23.471 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:56:23.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:23 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2201320122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:56:23.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.663+0000 7f7100304640 1 -- 172.21.15.57:0/69709825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 msgr2=0x7f70f8100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.664 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.663+0000 7f7100304640 1 --2- 172.21.15.57:0/69709825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 0x7f70f8100ad0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009930 tx=0x7f70f002bc50 comp rx=0 tx=0).stop 2024-04-21T08:56:23.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.668+0000 7f7100304640 1 -- 172.21.15.57:0/69709825 shutdown_connections 2024-04-21T08:56:23.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.668+0000 7f7100304640 1 --2- 172.21.15.57:0/69709825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f8107300 0x7f70f810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.668+0000 7f7100304640 1 --2- 172.21.15.57:0/69709825 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70f81010a0 0x7f70f8106bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.668+0000 7f7100304640 1 --2- 172.21.15.57:0/69709825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 0x7f70f8100ad0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.669+0000 7f7100304640 1 -- 172.21.15.57:0/69709825 >> 172.21.15.57:0/69709825 conn(0x7f70f80773f0 msgr2=0x7f70f80fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:23.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.671+0000 7f7100304640 1 -- 172.21.15.57:0/69709825 shutdown_connections 2024-04-21T08:56:23.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.671+0000 7f7100304640 1 -- 172.21.15.57:0/69709825 wait complete. 2024-04-21T08:56:23.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.672+0000 7f7100304640 1 Processor -- start 2024-04-21T08:56:23.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 -- start start 2024-04-21T08:56:23.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 0x7f70f819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70f8107300 0x7f70f819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 0x7f70f8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f70f81121c0 con 0x7f70f81985c0 2024-04-21T08:56:23.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70f8112040 con 0x7f70f81006f0 2024-04-21T08:56:23.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f7100304640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f70f8112340 con 0x7f70f8107300 2024-04-21T08:56:23.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f70fe8a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 0x7f70f8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.674+0000 7f70fe8a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 0x7f70f8198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:56248/0 (socket says 172.21.15.57:56248) 2024-04-21T08:56:23.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe8a1640 1 -- 172.21.15.57:0/4249518989 learned_addr learned my addr 172.21.15.57:0/4249518989 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe0a0640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 0x7f70f819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fd89f640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70f8107300 0x7f70f819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe8a1640 1 -- 172.21.15.57:0/4249518989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70f8107300 msgr2=0x7f70f819e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe8a1640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70f8107300 0x7f70f819e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe8a1640 1 -- 172.21.15.57:0/4249518989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 msgr2=0x7f70f819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:23.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe8a1640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 0x7f70f819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:23.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.675+0000 7f70fe8a1640 1 -- 172.21.15.57:0/4249518989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f81990e0 con 0x7f70f81985c0 2024-04-21T08:56:23.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.676+0000 7f70fe8a1640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 0x7f70f8198a70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b970 tx=0x7f70e800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:23.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.676+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8019070 con 0x7f70f81985c0 2024-04-21T08:56:23.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.676+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f81993d0 con 0x7f70f81985c0 2024-04-21T08:56:23.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.676+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f8101880 con 0x7f70f81985c0 2024-04-21T08:56:23.678 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.676+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70e8014440 con 0x7f70f81985c0 2024-04-21T08:56:23.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.677+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8019070 con 0x7f70f81985c0 2024-04-21T08:56:23.679 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.679+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e4005290 con 0x7f70f81985c0 2024-04-21T08:56:23.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.681+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f70e800cb40 con 0x7f70f81985c0 2024-04-21T08:56:23.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.682+0000 7f70f77fe640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70e00a4120 0x7f70e00a65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:23.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.682+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7f70e800c940 con 0x7f70f81985c0 2024-04-21T08:56:23.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.684+0000 7f70fe0a0640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70e00a4120 0x7f70e00a65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:23.686 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.686+0000 7f70fe0a0640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70e00a4120 0x7f70e00a65e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f70f0005ab0 tx=0x7f70f0005a20 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:23.693 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:23.693+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f70e804fff0 con 0x7f70f81985c0 2024-04-21T08:56:23.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:23 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2201320122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:56:23.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:23 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2201320122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:56:24.030 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.030+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f70e4005850 con 0x7f70f81985c0 2024-04-21T08:56:24.032 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.032+0000 7f70f77fe640 1 -- 172.21.15.57:0/4249518989 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) ==== 103+0+11 (secure 0 0 0) 0x7f70e809b480 con 0x7f70f81985c0 2024-04-21T08:56:24.032 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:56:24.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.036+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70e00a4120 msgr2=0x7f70e00a65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.036+0000 7f7100304640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70e00a4120 0x7f70e00a65e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f70f0005ab0 tx=0x7f70f0005a20 comp rx=0 tx=0).stop 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.036+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 msgr2=0x7f70f8198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.036+0000 7f7100304640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 0x7f70f8198a70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b970 tx=0x7f70e800be30 comp rx=0 tx=0).stop 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 shutdown_connections 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f70e00a4120 0x7f70e00a65e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f81985c0 0x7f70f8198a70 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70f8107300 0x7f70f819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.037 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 --2- 172.21.15.57:0/4249518989 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f81006f0 0x7f70f819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 >> 172.21.15.57:0/4249518989 conn(0x7f70f80773f0 msgr2=0x7f70f80fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:24.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 shutdown_connections 2024-04-21T08:56:24.038 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.037+0000 7f7100304640 1 -- 172.21.15.57:0/4249518989 wait complete. 2024-04-21T08:56:24.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-21T08:56:24.229 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.229+0000 7fb00ba8c640 1 -- 172.21.15.57:0/2452600860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0041106b0 msgr2=0x7fb00410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:24.230 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.229+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/2452600860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0041106b0 0x7fb00410e290 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faff401a910 tx=0x7faff403cc70 comp rx=0 tx=0).stop 2024-04-21T08:56:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.231+0000 7fb00ba8c640 1 -- 172.21.15.57:0/2452600860 shutdown_connections 2024-04-21T08:56:24.231 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.231+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/2452600860 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb00410eeb0 0x7fb00411d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.231+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/2452600860 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb004110b20 0x7fb00410e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.231+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/2452600860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0041106b0 0x7fb00410e290 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.231+0000 7fb00ba8c640 1 -- 172.21.15.57:0/2452600860 >> 172.21.15.57:0/2452600860 conn(0x7fb00406d070 msgr2=0x7fb00406d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:24.232 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.232+0000 7fb00ba8c640 1 -- 172.21.15.57:0/2452600860 shutdown_connections 2024-04-21T08:56:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.232+0000 7fb00ba8c640 1 -- 172.21.15.57:0/2452600860 wait complete. 2024-04-21T08:56:24.233 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.233+0000 7fb00ba8c640 1 Processor -- start 2024-04-21T08:56:24.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.233+0000 7fb00ba8c640 1 -- start start 2024-04-21T08:56:24.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00ba8c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 0x7fb00411af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:24.235 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00ba8c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb004116fc0 0x7fb004117420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00ba8c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb004117960 0x7fb0041ab4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00ba8c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb0041137e0 con 0x7fb004116fc0 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00ba8c640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb004113660 con 0x7fb004117960 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00ba8c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb004113960 con 0x7fb0041106b0 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 0x7fb00411af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 0x7fb00411af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:41952/0 (socket says 172.21.15.57:41952) 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 -- 172.21.15.57:0/1507610295 learned_addr learned my addr 172.21.15.57:0/1507610295 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:56:24.236 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb00a029640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb004117960 0x7fb0041ab4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009027640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb004116fc0 0x7fb004117420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 -- 172.21.15.57:0/1507610295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb004117960 msgr2=0x7fb0041ab4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb004117960 0x7fb0041ab4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 -- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb004116fc0 msgr2=0x7fb004117420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb004116fc0 0x7fb004117420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.234+0000 7fb009828640 1 -- 172.21.15.57:0/1507610295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0041aba10 con 0x7fb0041106b0 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.235+0000 7fb009828640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 0x7fb00411af30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faff403d280 tx=0x7faff40045c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.235+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff4052020 con 0x7fb0041106b0 2024-04-21T08:56:24.237 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.235+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff40053f0 con 0x7fb0041106b0 2024-04-21T08:56:24.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.236+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff4003be0 con 0x7fb0041106b0 2024-04-21T08:56:24.238 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.236+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0041abc40 con 0x7fb0041106b0 2024-04-21T08:56:24.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.237+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0041ac110 con 0x7fb0041106b0 2024-04-21T08:56:24.239 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.238+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00406e300 con 0x7fb0041106b0 2024-04-21T08:56:24.240 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.240+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faff4055020 con 0x7fb0041106b0 2024-04-21T08:56:24.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.241+0000 7faffaffd640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7faff00a3d00 0x7faff00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:56:24.241 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.241+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9563+0+0 (secure 0 0 0) 0x7faff41038f0 con 0x7fb0041106b0 2024-04-21T08:56:24.245 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.245+0000 7fb009027640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7faff00a3d00 0x7faff00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:56:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.250+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faff405c070 con 0x7fb0041106b0 2024-04-21T08:56:24.250 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.250+0000 7fb009027640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7faff00a3d00 0x7faff00a61c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb000009990 tx=0x7fb000009110 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:56:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:24 smithi136 ceph-mon[31791]: pgmap v345: 129 pgs: 16 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 71 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 846 KiB/s wr, 122 op/s 2024-04-21T08:56:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:24 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3294072716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:56:24.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:24 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4249518989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:56:24.692 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.692+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fb0041184f0 con 0x7fb0041106b0 2024-04-21T08:56:24.705 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.706+0000 7faffaffd640 1 -- 172.21.15.57:0/1507610295 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746061 (secure 0 0 0) 0x7faff40cfd10 con 0x7fb0041106b0 2024-04-21T08:56:24.718 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:56:24.718 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:24.718 INFO:teuthology.orchestra.run.smithi057.stdout: "epoch": 26, 2024-04-21T08:56:24.718 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "active_gid": 15375, 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "active_name": "x", 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "active_addrs": { 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:6800", 2024-04-21T08:56:24.719 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3217118060 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v1", 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:6801", 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3217118060 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.720 INFO:teuthology.orchestra.run.smithi057.stdout: "active_addr": "172.21.15.57:6801/3217118060", 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "active_change": "2024-04-21T08:45:24.897813+0000", 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "active_mgr_features": 4540701547738038271, 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "available": true, 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "standbys": [ 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "gid": 15369, 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "y", 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr_features": 4540701547738038271, 2024-04-21T08:56:24.721 INFO:teuthology.orchestra.run.smithi057.stdout: "available_modules": [ 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "alerts", 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.722 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.723 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.724 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.725 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.726 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.727 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.728 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.729 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.730 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.731 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_destination": { 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_destination", 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.732 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_from_name": { 2024-04-21T08:56:24.733 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_from_name", 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "Ceph", 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.734 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Email From: name", 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_host": { 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_host", 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.735 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP server", 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.736 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_password": { 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_password", 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.737 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Password to authenticate with", 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_port": { 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_port", 2024-04-21T08:56:24.738 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "465", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP port", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.739 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_sender": { 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_sender", 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.740 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.741 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_ssl": { 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_ssl", 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.742 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_user": { 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_user", 2024-04-21T08:56:24.743 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "User to authenticate as", 2024-04-21T08:56:24.744 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "balancer", 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:24.745 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "active": { 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "active", 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.746 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "begin_time": { 2024-04-21T08:56:24.747 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "begin_time", 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0000", 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.748 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "begin_weekday": { 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "begin_weekday", 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:24.749 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0", 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "6", 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.750 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_max_iterations": { 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "25", 2024-04-21T08:56:24.751 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "250", 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_metrics": { 2024-04-21T08:56:24.752 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_metrics", 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.753 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_step": { 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_step", 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:24.754 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0.5", 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0.001", 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "0.999", 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:56:24.755 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "end_time": { 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "end_time", 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "2359", 2024-04-21T08:56:24.756 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "end_weekday": { 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "end_weekday", 2024-04-21T08:56:24.757 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0", 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "6", 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:56:24.758 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.759 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.760 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.761 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.762 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:24.763 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.764 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.765 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.766 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.767 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "min_score": { 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "min_score", 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.768 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": { 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mode", 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.769 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "upmap", 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "crush-compat", 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "none", 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "read", 2024-04-21T08:56:24.770 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap", 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap-read" 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Balancer mode", 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "pool_ids": { 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pool_ids", 2024-04-21T08:56:24.771 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.772 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.773 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap_max_deviation": { 2024-04-21T08:56:24.774 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "upmap_max_deviation", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:56:24.775 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap_max_optimizations": { 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.776 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.777 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephadm", 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:24.778 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_down_multiplier": { 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_down_multiplier", 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3.0", 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.779 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_refresh_rate": { 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_refresh_rate", 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.780 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "20", 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.781 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_starting_port": { 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_starting_port", 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "4721", 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.782 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_ptrace": { 2024-04-21T08:56:24.783 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_ptrace", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:56:24.784 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "autotune_interval": { 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "autotune_interval", 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.785 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.786 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0.7", 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.787 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm_log_destination": { 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephadm_log_destination", 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.788 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "file", 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "file,syslog", 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: "syslog" 2024-04-21T08:56:24.789 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "cgroups_split": { 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cgroups_split", 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.790 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.791 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "config_checks_enabled": { 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_checks_enabled", 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.792 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "config_dashboard": { 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_dashboard", 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.793 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.794 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_alertmanager": { 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_alertmanager", 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.795 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_base": { 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_base", 2024-04-21T08:56:24.796 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:56:24.797 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_elasticsearch": { 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.798 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "elasticsearch container image", 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.799 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_grafana": { 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_grafana", 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.800 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_haproxy": { 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_haproxy", 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.801 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "HAproxy container image", 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.802 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_agent": { 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.803 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.804 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_collector": { 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.805 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_query": { 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.806 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger query container image", 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.807 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_keepalived": { 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_keepalived", 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.808 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Keepalived container image", 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.809 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_loki": { 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_loki", 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.810 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Loki container image", 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_node_exporter": { 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_node_exporter", 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.811 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.812 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_nvmeof": { 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_nvmeof", 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:56:24.813 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Nvme-of container image", 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.814 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_prometheus": { 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_prometheus", 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.815 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_promtail": { 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_promtail", 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.816 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Promtail container image", 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.817 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_samba": { 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_samba", 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.818 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_snmp_gateway": { 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:56:24.819 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.820 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "container_init": { 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_init", 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.821 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "daemon_cache_timeout": { 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:56:24.822 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.823 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "900", 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.824 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "default_registry": { 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "default_registry", 2024-04-21T08:56:24.825 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.826 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "device_cache_timeout": { 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_cache_timeout", 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1800", 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.827 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "device_enhanced_scan": { 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_enhanced_scan", 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.828 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.829 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "facts_cache_timeout": { 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "facts_cache_timeout", 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.830 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "grafana_dashboards_path": { 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.831 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.832 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "host_check_interval": { 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "host_check_interval", 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.833 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "hw_monitoring": { 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "hw_monitoring", 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.834 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.835 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "inventory_list_all": { 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "inventory_list_all", 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.836 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:24.837 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.838 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.839 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "log_refresh_metadata": { 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_refresh_metadata", 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.840 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:24.841 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:56:24.842 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.843 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.844 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:24.845 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.846 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.847 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.848 INFO:teuthology.orchestra.run.smithi057.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "*", 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.849 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "max_count_per_host": { 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_count_per_host", 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.850 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.851 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "max_osd_draining_count": { 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_osd_draining_count", 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.852 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "migration_current": { 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "migration_current", 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.853 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "internal - do not modify", 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.854 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": { 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mode", 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "root", 2024-04-21T08:56:24.855 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm-package", 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "root" 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.856 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "oob_default_addr": { 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "oob_default_addr", 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "169.254.1.1", 2024-04-21T08:56:24.857 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.858 INFO:teuthology.orchestra.run.smithi057.stdout: "prometheus_alerts_path": { 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.859 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_insecure": { 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_insecure", 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.860 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.861 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_password": { 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_password", 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.862 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_url": { 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_url", 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.863 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.865 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.866 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_username": { 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_username", 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.867 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "secure_monitoring_stack": { 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.868 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "service_discovery_port": { 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "service_discovery_port", 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8765", 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "ssh_config_file": { 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssh_config_file", 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.871 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.872 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "use_agent": { 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "use_agent", 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.873 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "use_repo_digest": { 2024-04-21T08:56:24.874 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "use_repo_digest", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:56:24.875 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_failed_host_check": { 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.876 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.877 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_stray_daemons": { 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.878 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_stray_hosts": { 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:56:24.879 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:56:24.880 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crash", 2024-04-21T08:56:24.881 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.882 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.883 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.884 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.885 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.886 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.887 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:24.888 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.889 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "retain_interval": { 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "retain_interval", 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "31536000", 2024-04-21T08:56:24.890 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_recent_interval": { 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_recent_interval", 2024-04-21T08:56:24.891 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1209600", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.892 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "dashboard", 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:24.893 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.894 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.895 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.896 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.897 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:56:24.898 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.899 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.900 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:56:24.901 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.902 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.903 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:56:24.904 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.905 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.906 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.907 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.908 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.909 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.910 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.911 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:56:24.912 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.913 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.914 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.915 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.916 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:56:24.917 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.918 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_URL": { 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.919 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.920 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.921 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:56:24.922 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.923 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.924 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.925 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.926 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:56:24.927 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.928 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.929 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "[]", 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.930 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "{}", 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.931 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:56:24.932 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.933 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.934 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.935 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.936 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:56:24.937 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.938 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.939 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:56:24.940 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.941 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.942 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:56:24.943 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.944 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.945 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.946 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.947 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:56:24.948 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.949 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.950 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.951 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8", 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.952 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.953 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "45", 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.954 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.955 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.956 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.957 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:56:24.958 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.959 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.960 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:56:24.961 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.962 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.963 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.964 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.965 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:56:24.966 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.967 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "cross_origin_url": { 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cross_origin_url", 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.968 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.969 INFO:teuthology.orchestra.run.smithi057.stdout: "crt_file": { 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crt_file", 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.970 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "debug": { 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "debug", 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.971 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.972 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "jwt_token_ttl": { 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "jwt_token_ttl", 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.973 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "28800", 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.974 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "key_file": { 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "key_file", 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.975 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:24.976 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.977 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.978 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.979 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:24.980 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:24.981 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.982 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.983 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "motd": { 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "motd", 2024-04-21T08:56:24.984 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "The message of the day", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.985 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.986 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.987 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "::", 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.988 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.989 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8080", 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.990 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl": { 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl", 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.991 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl_server_port": { 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl_server_port", 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.992 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8443", 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.993 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_behaviour": { 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_behaviour", 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "redirect", 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.994 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "redirect" 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.995 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_error_status_code": { 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "500", 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "400", 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "599", 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.996 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "url_prefix": { 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "url_prefix", 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:24.997 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:24.998 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "devicehealth", 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:24.999 INFO:teuthology.orchestra.run.smithi057.stdout: "enable_monitoring": { 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enable_monitoring", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "monitor device health metrics", 2024-04-21T08:56:25.000 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.001 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.002 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.003 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.004 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.005 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.006 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.007 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.008 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "mark_out_threshold": { 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mark_out_threshold", 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.009 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "2419200", 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.010 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "pool_name": { 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pool_name", 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "device_health_metrics", 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.011 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "retention_period": { 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "retention_period", 2024-04-21T08:56:25.012 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15552000", 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:56:25.013 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "scrape_frequency": { 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "scrape_frequency", 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.014 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "86400", 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.015 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "self_heal": { 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "self_heal", 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.016 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.017 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.018 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_threshold": { 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_threshold", 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "7257600", 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.019 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.020 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "diskprediction_local", 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.021 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.022 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.023 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.024 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.025 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.026 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.027 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.028 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "predict_interval": { 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "predict_interval", 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "86400", 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.029 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "predictor_model": { 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "predictor_model", 2024-04-21T08:56:25.030 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "prophetstor", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.031 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.032 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.033 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "influx", 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": false, 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "batch_size": { 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "batch_size", 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.034 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5000", 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.035 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "database": { 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "database", 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "ceph", 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.036 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": { 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "hostname", 2024-04-21T08:56:25.037 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.038 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "30", 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "5", 2024-04-21T08:56:25.039 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.040 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.041 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.042 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.043 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.044 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.045 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.046 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.047 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "password": { 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "password", 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.048 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "port": { 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "port", 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.049 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8086", 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB server port", 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.050 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl": { 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl", 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "false", 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.051 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "threads": { 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "threads", 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.052 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "32", 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.053 INFO:teuthology.orchestra.run.smithi057.stdout: "username": { 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "username", 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.054 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "verify_ssl": { 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "verify_ssl", 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.055 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "true", 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.056 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "insights", 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.057 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.058 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.059 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.060 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.061 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.062 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.063 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.064 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "iostat", 2024-04-21T08:56:25.065 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.066 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.067 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.067 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.069 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.069 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.069 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.069 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.069 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.070 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.071 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.072 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.073 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.074 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.075 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "k8sevents", 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.076 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_event_retention_days": { 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "7", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.077 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "config_check_secs": { 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_check_secs", 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.078 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "10", 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.079 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.080 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.081 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.082 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.083 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.084 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.085 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.086 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.087 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "localpool", 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "failure_domain": { 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "failure_domain", 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.088 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "host", 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.089 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.090 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.091 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.092 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.093 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.094 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.095 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.096 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "min_size": { 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "min_size", 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.097 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.098 INFO:teuthology.orchestra.run.smithi057.stdout: "num_rep": { 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "num_rep", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:56:25.099 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_num": { 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pg_num", 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.100 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "128", 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.101 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "prefix": { 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prefix", 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.102 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "subtree": { 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "subtree", 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.103 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "rack", 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.104 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mds_autoscaler", 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.105 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.106 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.107 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.108 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.109 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.110 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.111 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.112 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mirroring", 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.113 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.114 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.115 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.116 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.117 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.118 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.119 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.120 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "nfs", 2024-04-21T08:56:25.121 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.122 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.123 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.124 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.125 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.126 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.127 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.128 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.129 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "orchestrator", 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "fail_fs": { 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "fail_fs", 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.130 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.131 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.132 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.133 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.134 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.135 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.136 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.137 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.138 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.139 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator": { 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "orchestrator", 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.140 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm", 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: "rook", 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: "test_orchestrator" 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Orchestrator backend", 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.141 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "osd_perf_query", 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.142 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.143 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.144 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.145 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.146 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.147 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.148 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.149 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.150 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "osd_support", 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.151 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.152 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.153 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.154 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.155 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.156 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.157 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.158 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pg_autoscaler", 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.159 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.160 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.161 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.162 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.163 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.164 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.165 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.166 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.167 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.168 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "threshold": { 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "threshold", 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3.0", 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1.0", 2024-04-21T08:56:25.169 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "scaling threshold", 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.170 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "progress", 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_pg_recovery_event": { 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.171 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.172 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "enabled": { 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enabled", 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.173 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.174 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.175 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.176 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.177 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.178 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.179 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.180 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.181 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.182 INFO:teuthology.orchestra.run.smithi057.stdout: "max_completed_events": { 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_completed_events", 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "50", 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.183 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.184 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.185 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prometheus", 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.186 INFO:teuthology.orchestra.run.smithi057.stdout: "cache": { 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cache", 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.187 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "exclude_perf_counters": { 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "exclude_perf_counters", 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.188 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.189 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.190 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.191 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.192 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.193 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.194 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.195 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.196 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.197 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_stats_pools": { 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_stats_pools", 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.198 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:56:25.199 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "300", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.200 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "scrape_interval": { 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "scrape_interval", 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15.0", 2024-04-21T08:56:25.201 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.202 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "::", 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.203 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.204 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "9283", 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.205 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "stale_cache_strategy": { 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "stale_cache_strategy", 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.206 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "log", 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.207 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_behaviour": { 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_behaviour", 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "default", 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.208 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: "default", 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: "error" 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.209 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_error_status_code": { 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "500", 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "400", 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "599", 2024-04-21T08:56:25.210 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.211 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_support", 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.212 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.213 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.214 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.215 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.216 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.217 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.218 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.219 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "max_concurrent_snap_create": { 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.220 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.221 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.222 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.223 INFO:teuthology.orchestra.run.smithi057.stdout: "trash_purge_schedule": { 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "trash_purge_schedule", 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.224 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.225 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "restful", 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "enable_auth": { 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enable_auth", 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.226 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.227 INFO:teuthology.orchestra.run.smithi057.stdout: "key_file": { 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "key_file", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.228 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.229 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.230 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.231 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.232 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.233 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.234 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.235 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:25.236 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.237 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.238 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.239 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rgw", 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.240 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.241 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.242 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.243 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.244 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.245 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.246 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.247 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.248 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rook", 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.249 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.250 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.251 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.252 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.253 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.254 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.255 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.256 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "storage_class": { 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "storage_class", 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.257 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "local", 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.258 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "selftest", 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.259 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.260 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.261 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.262 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.263 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.264 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.265 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.266 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "roption1": { 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "roption1", 2024-04-21T08:56:25.267 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.268 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "roption2": { 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "roption2", 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "xyz", 2024-04-21T08:56:25.269 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption1": { 2024-04-21T08:56:25.270 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption1", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.271 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption2": { 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption2", 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.272 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption3": { 2024-04-21T08:56:25.273 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption3", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.274 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption4": { 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption4", 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.275 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption5": { 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption5", 2024-04-21T08:56:25.276 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.277 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption6": { 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption6", 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.278 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.279 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption7": { 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption7", 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.280 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "42", 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "testkey": { 2024-04-21T08:56:25.281 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testkey", 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.282 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "testlkey": { 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testlkey", 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.283 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.284 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "testnewline": { 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testnewline", 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.285 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.286 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snap_schedule", 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_m_granularity": { 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_m_granularity", 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.287 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.288 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "dump_on_update": { 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "dump_on_update", 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.289 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.290 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.291 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.292 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.293 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.294 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.295 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.296 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.297 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.298 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "stats", 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.299 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.300 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.301 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.302 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.303 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.304 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.305 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.306 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "status", 2024-04-21T08:56:25.307 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.308 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.309 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.310 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.311 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.312 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.313 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.314 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.315 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "telegraf", 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: "address": { 2024-04-21T08:56:25.316 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "address", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.317 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.318 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15", 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.319 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.320 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.321 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.322 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.323 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.324 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.325 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.326 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.327 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "telemetry", 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_basic": { 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_basic", 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.328 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_crash": { 2024-04-21T08:56:25.329 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_crash", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.330 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_device": { 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_device", 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.331 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_ident": { 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_ident", 2024-04-21T08:56:25.332 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.333 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_perf": { 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_perf", 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.334 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "contact": { 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "contact", 2024-04-21T08:56:25.335 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.336 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "description": { 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "description", 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.337 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "device_url": { 2024-04-21T08:56:25.338 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_url", 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.339 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "enabled": { 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enabled", 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.340 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.341 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "24", 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "8", 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.342 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "last_opt_revision": { 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "last_opt_revision", 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.343 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1", 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.344 INFO:teuthology.orchestra.run.smithi057.stdout: "leaderboard": { 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "leaderboard", 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.345 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "leaderboard_description": { 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "leaderboard_description", 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.346 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.347 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.348 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.349 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.350 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.351 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.352 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.353 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.354 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "organization": { 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "organization", 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.355 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.356 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "proxy": { 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "proxy", 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.357 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "url": { 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "url", 2024-04-21T08:56:25.358 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.359 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "test_orchestrator", 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.360 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.361 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.362 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.363 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.364 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.365 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.366 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.367 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.368 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "volumes", 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.369 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.370 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.371 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.372 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.373 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.374 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.375 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.376 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.377 INFO:teuthology.orchestra.run.smithi057.stdout: "max_concurrent_clones": { 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_concurrent_clones", 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "4", 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.378 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "periodic_async_work": { 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "periodic_async_work", 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.379 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Periodically check for async work", 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.380 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "snapshot_clone_delay": { 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:25.381 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.382 INFO:teuthology.orchestra.run.smithi057.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.383 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix", 2024-04-21T08:56:25.384 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "discovery_interval": { 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "discovery_interval", 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.385 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "100", 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.386 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "identifier": { 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "identifier", 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.387 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:25.388 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.389 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.390 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.391 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.392 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.393 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.394 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.395 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.396 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.397 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_host": { 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_host", 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.398 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_port": { 2024-04-21T08:56:25.399 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_port", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10051", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.400 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_sender": { 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_sender", 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:56:25.401 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.402 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: "gid": 24952, 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "z", 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: "mgr_features": 4540701547738038271, 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: "available_modules": [ 2024-04-21T08:56:25.403 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "alerts", 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.404 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.405 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.406 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.407 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.408 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.409 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.410 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.411 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.412 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.413 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.414 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_destination": { 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_destination", 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.415 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_from_name": { 2024-04-21T08:56:25.416 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_from_name", 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "Ceph", 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.417 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Email From: name", 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_host": { 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_host", 2024-04-21T08:56:25.418 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP server", 2024-04-21T08:56:25.419 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_password": { 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_password", 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.420 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Password to authenticate with", 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.421 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_port": { 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_port", 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "465", 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.422 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP port", 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_sender": { 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_sender", 2024-04-21T08:56:25.423 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:56:25.424 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_ssl": { 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_ssl", 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.425 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.426 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_user": { 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_user", 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.427 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "User to authenticate as", 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.428 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "balancer", 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "active": { 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "active", 2024-04-21T08:56:25.429 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:56:25.430 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "begin_time": { 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "begin_time", 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.431 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0000", 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.432 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "begin_weekday": { 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "begin_weekday", 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:25.433 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0", 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "6", 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.434 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_max_iterations": { 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "25", 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "250", 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.435 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_metrics": { 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_metrics", 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.436 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:56:25.437 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_step": { 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_step", 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.438 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0.5", 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0.001", 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "0.999", 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.439 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "end_time": { 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "end_time", 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "2359", 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.440 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "end_weekday": { 2024-04-21T08:56:25.441 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "end_weekday", 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0", 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "6", 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.442 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.443 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.444 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.445 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.446 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.447 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.448 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.449 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.450 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.451 INFO:teuthology.orchestra.run.smithi057.stdout: "min_score": { 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "min_score", 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.452 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": { 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mode", 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.453 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "upmap", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "crush-compat", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "none", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "read", 2024-04-21T08:56:25.454 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap", 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap-read" 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Balancer mode", 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.455 INFO:teuthology.orchestra.run.smithi057.stdout: "pool_ids": { 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pool_ids", 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.456 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.458 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap_max_deviation": { 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "upmap_max_deviation", 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:25.459 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap_max_optimizations": { 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:56:25.460 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.461 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephadm", 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.462 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_down_multiplier": { 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_down_multiplier", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3.0", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.463 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_refresh_rate": { 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_refresh_rate", 2024-04-21T08:56:25.464 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "20", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_starting_port": { 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_starting_port", 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "4721", 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_ptrace": { 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_ptrace", 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.468 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "autotune_interval": { 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "autotune_interval", 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.469 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.470 INFO:teuthology.orchestra.run.smithi057.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0.7", 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.471 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm_log_destination": { 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephadm_log_destination", 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.472 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "file", 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "file,syslog", 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "syslog" 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:56:25.473 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "cgroups_split": { 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cgroups_split", 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.474 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.475 INFO:teuthology.orchestra.run.smithi057.stdout: "config_checks_enabled": { 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_checks_enabled", 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.476 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "config_dashboard": { 2024-04-21T08:56:25.477 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_dashboard", 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.478 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_alertmanager": { 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_alertmanager", 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.479 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.480 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_base": { 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_base", 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.481 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_elasticsearch": { 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:56:25.482 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "elasticsearch container image", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.483 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_grafana": { 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_grafana", 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.484 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_haproxy": { 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_haproxy", 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "HAproxy container image", 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_agent": { 2024-04-21T08:56:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:56:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_collector": { 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_query": { 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger query container image", 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_keepalived": { 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_keepalived", 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Keepalived container image", 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_loki": { 2024-04-21T08:56:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_loki", 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Loki container image", 2024-04-21T08:56:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_node_exporter": { 2024-04-21T08:56:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_node_exporter", 2024-04-21T08:56:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:56:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_nvmeof": { 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_nvmeof", 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:56:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.500 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.500 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.500 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Nvme-of container image", 2024-04-21T08:56:25.500 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.500 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.500 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_prometheus": { 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_prometheus", 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.501 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.502 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_promtail": { 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_promtail", 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.503 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Promtail container image", 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_samba": { 2024-04-21T08:56:25.504 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_samba", 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.505 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_snmp_gateway": { 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:56:25.506 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:56:25.507 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "container_init": { 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_init", 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.508 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.509 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "daemon_cache_timeout": { 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.510 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:56:25.511 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "900", 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.512 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "default_registry": { 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "default_registry", 2024-04-21T08:56:25.513 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io", 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:56:25.514 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "device_cache_timeout": { 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_cache_timeout", 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.515 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1800", 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.516 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: "device_enhanced_scan": { 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_enhanced_scan", 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.517 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.518 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.518 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.518 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.518 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.518 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:56:25.518 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.519 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.519 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.519 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.519 INFO:teuthology.orchestra.run.smithi057.stdout: "facts_cache_timeout": { 2024-04-21T08:56:25.519 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "facts_cache_timeout", 2024-04-21T08:56:25.519 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.520 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: "grafana_dashboards_path": { 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:56:25.521 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:56:25.522 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "host_check_interval": { 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "host_check_interval", 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.523 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.524 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: "hw_monitoring": { 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "hw_monitoring", 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.525 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.526 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "inventory_list_all": { 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "inventory_list_all", 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.527 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.528 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.529 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.530 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: "log_refresh_metadata": { 2024-04-21T08:56:25.531 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_refresh_metadata", 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.532 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.533 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.534 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.535 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.536 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.537 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.538 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.539 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.540 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "*", 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.541 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "max_count_per_host": { 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_count_per_host", 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.542 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.543 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "max_osd_draining_count": { 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_osd_draining_count", 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.544 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "migration_current": { 2024-04-21T08:56:25.545 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "migration_current", 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.546 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "internal - do not modify", 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": { 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mode", 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.547 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "root", 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm-package", 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "root" 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.548 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "oob_default_addr": { 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "oob_default_addr", 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.549 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "169.254.1.1", 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.550 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "prometheus_alerts_path": { 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:56:25.551 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_insecure": { 2024-04-21T08:56:25.552 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_insecure", 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.553 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_password": { 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_password", 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.554 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.555 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_url": { 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_url", 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.556 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.557 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_username": { 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_username", 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.558 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "secure_monitoring_stack": { 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:56:25.559 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.560 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "service_discovery_port": { 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "service_discovery_port", 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.561 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8765", 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.562 INFO:teuthology.orchestra.run.smithi057.stdout: "ssh_config_file": { 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssh_config_file", 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.563 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "use_agent": { 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "use_agent", 2024-04-21T08:56:25.564 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.565 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "use_repo_digest": { 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "use_repo_digest", 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.566 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.567 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_failed_host_check": { 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.568 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_stray_daemons": { 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.569 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.570 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_stray_hosts": { 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.571 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.572 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crash", 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.573 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.574 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.575 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.576 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.577 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.578 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.579 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.580 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "retain_interval": { 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "retain_interval", 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.581 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "31536000", 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.582 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_recent_interval": { 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_recent_interval", 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.583 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1209600", 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.584 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "dashboard", 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.585 INFO:teuthology.orchestra.run.smithi057.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.586 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:56:25.587 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.588 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.589 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.590 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.591 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.592 INFO:teuthology.orchestra.run.smithi057.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.593 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.594 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.595 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.596 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:56:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.598 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.599 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.600 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.601 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:56:25.602 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.603 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.604 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.605 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.606 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.606 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:56:25.606 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:56:25.606 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.606 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.607 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.608 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.608 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.608 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.608 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.608 INFO:teuthology.orchestra.run.smithi057.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:56:25.608 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:56:25.609 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.609 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.609 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.609 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.609 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.609 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.610 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.610 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.610 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.610 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.610 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.610 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.611 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:56:25.611 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:56:25.611 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.611 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.611 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.611 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.612 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.613 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_URL": { 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.614 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.615 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.616 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.617 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.618 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:56:25.619 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.620 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.621 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.622 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.623 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:56:25.624 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "[]", 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.625 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.626 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "{}", 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.627 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.628 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:56:25.629 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.630 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.631 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.632 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.633 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.634 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.635 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.636 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.637 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.638 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.639 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.640 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:56:25.641 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.642 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.643 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.644 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:56:25.645 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.646 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.647 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.648 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8", 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.649 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "45", 2024-04-21T08:56:25.650 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:56:25.651 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.652 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.653 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.654 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.655 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:56:25.656 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.657 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.658 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.659 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:25.660 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:56:25.661 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.662 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.663 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.664 INFO:teuthology.orchestra.run.smithi057.stdout: "cross_origin_url": { 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cross_origin_url", 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.665 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "crt_file": { 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crt_file", 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.666 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.667 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "debug": { 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "debug", 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.668 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "jwt_token_ttl": { 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "jwt_token_ttl", 2024-04-21T08:56:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "28800", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.670 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "key_file": { 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "key_file", 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.671 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.672 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.673 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.674 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.675 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.676 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.677 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.678 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.679 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.679 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.679 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.679 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.679 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.679 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "motd": { 2024-04-21T08:56:25.681 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "motd", 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.682 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "The message of the day", 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.683 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.684 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "::", 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.685 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:25.686 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8080", 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.687 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl": { 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl", 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.688 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.689 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl_server_port": { 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl_server_port", 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8443", 2024-04-21T08:56:25.690 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_behaviour": { 2024-04-21T08:56:25.691 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_behaviour", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "redirect", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: "redirect" 2024-04-21T08:56:25.692 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_error_status_code": { 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.693 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "500", 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "400", 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "599", 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.694 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "url_prefix": { 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "url_prefix", 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.695 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "devicehealth", 2024-04-21T08:56:25.696 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "enable_monitoring": { 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enable_monitoring", 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.697 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "monitor device health metrics", 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.698 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.699 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.700 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.701 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.702 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.703 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.704 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.705 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "mark_out_threshold": { 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mark_out_threshold", 2024-04-21T08:56:25.706 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "2419200", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.707 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "pool_name": { 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pool_name", 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "device_health_metrics", 2024-04-21T08:56:25.708 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "retention_period": { 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "retention_period", 2024-04-21T08:56:25.709 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15552000", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.710 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "scrape_frequency": { 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "scrape_frequency", 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "86400", 2024-04-21T08:56:25.711 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.712 INFO:teuthology.orchestra.run.smithi057.stdout: "self_heal": { 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "self_heal", 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.713 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.714 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.715 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_threshold": { 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_threshold", 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "7257600", 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.716 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.717 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "diskprediction_local", 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.718 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.719 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.720 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.721 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.722 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.723 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.724 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.725 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "predict_interval": { 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "predict_interval", 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.726 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "86400", 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.727 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "predictor_model": { 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "predictor_model", 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "prophetstor", 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.728 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.729 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.730 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "influx", 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": false, 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.731 INFO:teuthology.orchestra.run.smithi057.stdout: "batch_size": { 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "batch_size", 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5000", 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.732 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "database": { 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "database", 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.733 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "ceph", 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.734 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": { 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "hostname", 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.735 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.736 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "30", 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "5", 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.737 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.738 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.739 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.740 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.741 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.742 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.743 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.744 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.745 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "password": { 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "password", 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.746 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "port": { 2024-04-21T08:56:25.747 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "port", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8086", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB server port", 2024-04-21T08:56:25.748 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl": { 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl", 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "false", 2024-04-21T08:56:25.749 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "threads": { 2024-04-21T08:56:25.750 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "threads", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "32", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:56:25.751 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "username": { 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "username", 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.752 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "verify_ssl": { 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "verify_ssl", 2024-04-21T08:56:25.753 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "true", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.754 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "insights", 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.755 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.756 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.757 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.758 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.759 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.760 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.761 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.762 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.763 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "iostat", 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.764 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.765 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.766 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.767 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.768 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.769 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.770 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.771 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "k8sevents", 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.772 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_event_retention_days": { 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "7", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.773 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "config_check_secs": { 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_check_secs", 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.774 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "10", 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.775 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.776 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.777 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.778 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.779 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.780 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.781 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.782 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.783 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "localpool", 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "failure_domain": { 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "failure_domain", 2024-04-21T08:56:25.784 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "host", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:56:25.785 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.786 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.787 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.788 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.789 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.790 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.791 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.792 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.793 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "min_size": { 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "min_size", 2024-04-21T08:56:25.794 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:56:25.795 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "num_rep": { 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "num_rep", 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3", 2024-04-21T08:56:25.796 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.797 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_num": { 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pg_num", 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "128", 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.798 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "prefix": { 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prefix", 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.799 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.800 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "subtree": { 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "subtree", 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "rack", 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.801 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.802 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mds_autoscaler", 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.803 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.804 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.805 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.806 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.807 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.808 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.809 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.810 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.811 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mirroring", 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.812 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.813 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.814 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.815 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.816 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.817 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.818 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.819 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.820 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "nfs", 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.821 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.822 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.823 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.824 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.825 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.826 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.827 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.828 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.829 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "orchestrator", 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.830 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "fail_fs": { 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "fail_fs", 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.831 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.832 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.833 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.834 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.835 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.836 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.837 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.838 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.839 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.840 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator": { 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "orchestrator", 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.841 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm", 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "rook", 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "test_orchestrator" 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Orchestrator backend", 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.842 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "osd_perf_query", 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.843 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.844 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.845 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.846 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.847 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.848 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.849 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.850 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.851 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "osd_support", 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.852 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.853 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.854 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.855 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.856 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.857 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.858 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.859 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.860 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.861 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pg_autoscaler", 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.862 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.863 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.864 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.865 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.866 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.867 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.868 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.869 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.870 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.871 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "threshold": { 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "threshold", 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3.0", 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1.0", 2024-04-21T08:56:25.872 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "scaling threshold", 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.873 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "progress", 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_pg_recovery_event": { 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.874 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:56:25.875 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "enabled": { 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enabled", 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.876 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.877 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.878 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.879 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.880 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.881 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.882 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.885 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.886 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.886 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.886 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.886 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.886 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.886 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.887 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.888 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "max_completed_events": { 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_completed_events", 2024-04-21T08:56:25.889 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "50", 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:56:25.890 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.891 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.892 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prometheus", 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "cache": { 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cache", 2024-04-21T08:56:25.893 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.894 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "exclude_perf_counters": { 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "exclude_perf_counters", 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.895 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:56:25.896 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.897 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.898 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.899 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.900 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.901 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.902 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.903 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.904 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_stats_pools": { 2024-04-21T08:56:25.905 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_stats_pools", 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.906 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.907 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "300", 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.908 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "scrape_interval": { 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "scrape_interval", 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.909 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15.0", 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.910 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "::", 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.911 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.912 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "9283", 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:56:25.913 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.914 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.914 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.914 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.914 INFO:teuthology.orchestra.run.smithi057.stdout: "stale_cache_strategy": { 2024-04-21T08:56:25.914 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "stale_cache_strategy", 2024-04-21T08:56:25.914 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.915 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.915 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.915 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "log", 2024-04-21T08:56:25.915 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.915 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.915 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.916 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.916 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.916 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.916 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.916 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.916 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_behaviour": { 2024-04-21T08:56:25.917 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_behaviour", 2024-04-21T08:56:25.917 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.917 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.917 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.917 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "default", 2024-04-21T08:56:25.917 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.918 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.918 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.918 INFO:teuthology.orchestra.run.smithi057.stdout: "default", 2024-04-21T08:56:25.918 INFO:teuthology.orchestra.run.smithi057.stdout: "error" 2024-04-21T08:56:25.918 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_error_status_code": { 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:25.919 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "500", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "400", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "599", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.920 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_support", 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.921 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.922 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.923 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.924 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.925 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.926 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.927 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.928 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.929 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "max_concurrent_snap_create": { 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.930 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.931 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.932 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "trash_purge_schedule": { 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "trash_purge_schedule", 2024-04-21T08:56:25.933 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.934 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "restful", 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.935 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "enable_auth": { 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enable_auth", 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.936 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.937 INFO:teuthology.orchestra.run.smithi057.stdout: "key_file": { 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "key_file", 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.938 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.939 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.940 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.941 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.942 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.943 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.944 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.945 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.946 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.947 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.948 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.949 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.950 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rgw", 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.951 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.952 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.953 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.954 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.955 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.956 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.957 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.958 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.959 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rook", 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.960 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.961 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.962 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.963 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.964 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.965 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.966 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.967 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "storage_class": { 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "storage_class", 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.968 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "local", 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.969 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "selftest", 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.970 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.971 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.972 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.973 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.974 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:25.975 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:25.976 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:25.977 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.978 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "roption1": { 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "roption1", 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.979 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "roption2": { 2024-04-21T08:56:25.980 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "roption2", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "xyz", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.981 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption1": { 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption1", 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.982 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.983 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption2": { 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption2", 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.984 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption3": { 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption3", 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:25.985 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.986 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption4": { 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption4", 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.987 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption5": { 2024-04-21T08:56:25.988 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption5", 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.989 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption6": { 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption6", 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.990 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.991 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption7": { 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption7", 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:25.992 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "42", 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "testkey": { 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testkey", 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.993 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.994 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "testlkey": { 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testlkey", 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.995 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.996 INFO:teuthology.orchestra.run.smithi057.stdout: "testnewline": { 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testnewline", 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:25.997 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:25.998 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snap_schedule", 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_m_granularity": { 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_m_granularity", 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:25.999 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.000 INFO:teuthology.orchestra.run.smithi057.stdout: "dump_on_update": { 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "dump_on_update", 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.001 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.002 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.003 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.004 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.005 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.006 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.007 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.008 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.009 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "stats", 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.010 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.011 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.012 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.013 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.014 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.015 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.016 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.017 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.018 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "status", 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.019 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.020 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.021 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.022 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.023 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.024 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.025 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.026 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.027 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "telegraf", 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.028 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "address": { 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "address", 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.029 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.030 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.031 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.032 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.033 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.034 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.035 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.036 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.037 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.038 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.039 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "telemetry", 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.040 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_basic": { 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_basic", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:56:26.041 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_crash": { 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_crash", 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.042 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.043 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_device": { 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_device", 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.044 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_ident": { 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_ident", 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.045 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.046 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_perf": { 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_perf", 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.047 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.048 INFO:teuthology.orchestra.run.smithi057.stdout: "contact": { 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "contact", 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.049 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "description": { 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "description", 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.050 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.051 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "device_url": { 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_url", 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.052 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "enabled": { 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enabled", 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.053 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.054 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "24", 2024-04-21T08:56:26.055 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "8", 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.056 INFO:teuthology.orchestra.run.smithi057.stdout: "last_opt_revision": { 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "last_opt_revision", 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1", 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.057 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "leaderboard": { 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "leaderboard", 2024-04-21T08:56:26.058 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.059 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "leaderboard_description": { 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "leaderboard_description", 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.060 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.061 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.062 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.063 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.064 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.065 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.066 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.067 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.068 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.069 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "organization": { 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "organization", 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.070 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.071 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "proxy": { 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "proxy", 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.072 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.073 INFO:teuthology.orchestra.run.smithi057.stdout: "url": { 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "url", 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.074 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "test_orchestrator", 2024-04-21T08:56:26.075 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.076 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.077 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.078 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.079 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.080 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.081 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.082 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.083 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.084 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "volumes", 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.085 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.086 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.087 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.089 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.089 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.089 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.089 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.089 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.090 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.091 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.092 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.093 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.094 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "max_concurrent_clones": { 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_concurrent_clones", 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "4", 2024-04-21T08:56:26.095 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.096 INFO:teuthology.orchestra.run.smithi057.stdout: "periodic_async_work": { 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "periodic_async_work", 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.097 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Periodically check for async work", 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "snapshot_clone_delay": { 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.098 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.099 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.100 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.101 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix", 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: "discovery_interval": { 2024-04-21T08:56:26.102 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "discovery_interval", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "100", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.103 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "identifier": { 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "identifier", 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.104 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.105 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.106 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.107 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.108 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.109 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.110 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.111 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.112 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.113 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.114 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.115 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_host": { 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_host", 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.116 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.117 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_port": { 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_port", 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10051", 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.118 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_sender": { 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_sender", 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.119 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.120 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: "modules": [ 2024-04-21T08:56:26.121 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm", 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: "dashboard", 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: "iostat", 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: "nfs", 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: "restful", 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: "snap_schedule" 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: "available_modules": [ 2024-04-21T08:56:26.122 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "alerts", 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.123 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How frequently to reexamine health status", 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.124 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.125 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.126 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.127 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.128 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.129 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.130 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.131 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.132 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_destination": { 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_destination", 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.133 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Email address to send alerts to", 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_from_name": { 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_from_name", 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.134 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "Ceph", 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Email From: name", 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.135 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_host": { 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_host", 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.136 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP server", 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_password": { 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_password", 2024-04-21T08:56:26.137 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Password to authenticate with", 2024-04-21T08:56:26.138 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_port": { 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_port", 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "465", 2024-04-21T08:56:26.139 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP port", 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.140 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_sender": { 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_sender", 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.141 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SMTP envelope sender", 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_ssl": { 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_ssl", 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.142 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.143 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "smtp_user": { 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "smtp_user", 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.144 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "User to authenticate as", 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.145 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "balancer", 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "active": { 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "active", 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.146 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "automatically balance PGs across cluster", 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.147 INFO:teuthology.orchestra.run.smithi057.stdout: "begin_time": { 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "begin_time", 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0000", 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.148 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "beginning time of day to automatically balance", 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "begin_weekday": { 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "begin_weekday", 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.149 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0", 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "6", 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:26.150 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_max_iterations": { 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_max_iterations", 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "25", 2024-04-21T08:56:26.151 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "250", 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.152 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_metrics": { 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_metrics", 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "pgs,objects,bytes", 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.153 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "crush_compat_step": { 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crush_compat_step", 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.154 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0.5", 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0.001", 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "0.999", 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "aggressiveness of optimization", 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.155 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "end_time": { 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "end_time", 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "2359", 2024-04-21T08:56:26.156 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "ending time of day to automatically balance", 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "end_weekday": { 2024-04-21T08:56:26.157 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "end_weekday", 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "0", 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "6", 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.158 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.159 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.160 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.161 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.162 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.163 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.164 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.165 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.166 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "min_score": { 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "min_score", 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.167 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.168 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": { 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mode", 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "upmap", 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.169 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "crush-compat", 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "none", 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "read", 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap", 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap-read" 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Balancer mode", 2024-04-21T08:56:26.170 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "pool_ids": { 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pool_ids", 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.171 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.172 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.173 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap_max_deviation": { 2024-04-21T08:56:26.174 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "upmap_max_deviation", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-21T08:56:26.175 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "upmap_max_optimizations": { 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "upmap_max_optimizations", 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.176 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.177 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephadm", 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.178 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_down_multiplier": { 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_down_multiplier", 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3.0", 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.179 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_refresh_rate": { 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_refresh_rate", 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.180 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "20", 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.181 INFO:teuthology.orchestra.run.smithi057.stdout: "agent_starting_port": { 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "agent_starting_port", 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "4721", 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.182 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_ptrace": { 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_ptrace", 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.183 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.184 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "autotune_interval": { 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "autotune_interval", 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.185 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "autotune_memory_target_ratio": { 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "autotune_memory_target_ratio", 2024-04-21T08:56:26.186 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0.7", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.187 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm_log_destination": { 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cephadm_log_destination", 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.188 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "file", 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "file,syslog", 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "syslog" 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.189 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "cgroups_split": { 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cgroups_split", 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.190 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.191 INFO:teuthology.orchestra.run.smithi057.stdout: "config_checks_enabled": { 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_checks_enabled", 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.192 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "config_dashboard": { 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_dashboard", 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.193 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.194 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_alertmanager": { 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_alertmanager", 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-21T08:56:26.195 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_base": { 2024-04-21T08:56:26.196 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_base", 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.197 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Container image name, without the tag", 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_elasticsearch": { 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_elasticsearch", 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.198 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "elasticsearch container image", 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.199 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_grafana": { 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_grafana", 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-21T08:56:26.200 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_haproxy": { 2024-04-21T08:56:26.201 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_haproxy", 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.202 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "HAproxy container image", 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_agent": { 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_agent", 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.203 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger agent container image", 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.204 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_collector": { 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_collector", 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.205 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger collector container image", 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_jaeger_query": { 2024-04-21T08:56:26.206 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_jaeger_query", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Jaeger query container image", 2024-04-21T08:56:26.207 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_keepalived": { 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_keepalived", 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.208 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Keepalived container image", 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.209 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_loki": { 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_loki", 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-21T08:56:26.210 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Loki container image", 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.211 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_node_exporter": { 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_node_exporter", 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.212 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_nvmeof": { 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_nvmeof", 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.213 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Nvme-of container image", 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.214 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_prometheus": { 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_prometheus", 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.215 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Prometheus container image", 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_promtail": { 2024-04-21T08:56:26.216 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_promtail", 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.217 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Promtail container image", 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_samba": { 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_samba", 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.218 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Samba/SMB container image", 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.219 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "container_image_snmp_gateway": { 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_image_snmp_gateway", 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.220 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "SNMP Gateway container image", 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.221 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "container_init": { 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "container_init", 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.222 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Run podman/docker with `--init`", 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "daemon_cache_timeout": { 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "daemon_cache_timeout", 2024-04-21T08:56:26.223 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-21T08:56:26.224 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "default_cephadm_command_timeout": { 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "default_cephadm_command_timeout", 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.225 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "900", 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.226 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "default_registry": { 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "default_registry", 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "docker.io", 2024-04-21T08:56:26.227 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.228 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "device_cache_timeout": { 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_cache_timeout", 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1800", 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.229 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache device inventory", 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "device_enhanced_scan": { 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_enhanced_scan", 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.230 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.231 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "facts_cache_timeout": { 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "facts_cache_timeout", 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.232 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "seconds to cache host facts data", 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.233 INFO:teuthology.orchestra.run.smithi057.stdout: "grafana_dashboards_path": { 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "grafana_dashboards_path", 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.234 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "host_check_interval": { 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "host_check_interval", 2024-04-21T08:56:26.235 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to perform a host check", 2024-04-21T08:56:26.236 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "hw_monitoring": { 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "hw_monitoring", 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.237 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.238 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "inventory_list_all": { 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "inventory_list_all", 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.239 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.240 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.241 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.242 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "log_refresh_metadata": { 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_refresh_metadata", 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.243 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.244 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.245 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.246 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.247 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.248 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.249 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.250 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.251 INFO:teuthology.orchestra.run.smithi057.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "*", 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.252 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "max_count_per_host": { 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_count_per_host", 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.253 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "max number of daemons per service per host", 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.254 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "max_osd_draining_count": { 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_osd_draining_count", 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.255 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "migration_current": { 2024-04-21T08:56:26.256 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "migration_current", 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.257 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "internal - do not modify", 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "mode": { 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mode", 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.258 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "root", 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm-package", 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "root" 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.259 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "mode for remote execution of cephadm", 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "oob_default_addr": { 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "oob_default_addr", 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.260 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "169.254.1.1", 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.261 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "prometheus_alerts_path": { 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prometheus_alerts_path", 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.262 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.263 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_insecure": { 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_insecure", 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.264 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_password": { 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_password", 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.265 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.266 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_url": { 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_url", 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.267 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.268 INFO:teuthology.orchestra.run.smithi057.stdout: "registry_username": { 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "registry_username", 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.269 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "secure_monitoring_stack": { 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "secure_monitoring_stack", 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.270 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.271 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "service_discovery_port": { 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "service_discovery_port", 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8765", 2024-04-21T08:56:26.272 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "cephadm service discovery port", 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.273 INFO:teuthology.orchestra.run.smithi057.stdout: "ssh_config_file": { 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssh_config_file", 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.274 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "use_agent": { 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "use_agent", 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.275 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.276 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "use_repo_digest": { 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "use_repo_digest", 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.277 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.278 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_failed_host_check": { 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_failed_host_check", 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.279 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if the host check fails", 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_stray_daemons": { 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_stray_daemons", 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.280 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.281 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_on_stray_hosts": { 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_on_stray_hosts", 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.282 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.283 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crash", 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.284 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.285 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.286 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.287 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.288 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.289 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.290 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.291 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.293 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "retain_interval": { 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "retain_interval", 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "31536000", 2024-04-21T08:56:26.294 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.295 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_recent_interval": { 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_recent_interval", 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1209600", 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.296 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.297 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "dashboard", 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.298 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.299 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "ALERTMANAGER_API_HOST": { 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.300 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-21T08:56:26.301 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.302 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "AUDIT_API_ENABLED": { 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "AUDIT_API_ENABLED", 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.303 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.304 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.305 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "ENABLE_BROWSABLE_API": { 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-21T08:56:26.306 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.307 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.308 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.309 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.310 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-21T08:56:26.311 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.312 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.313 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.314 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.315 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-21T08:56:26.316 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.317 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.318 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.319 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.320 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_PASSWORD": { 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-21T08:56:26.321 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.322 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.323 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.324 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_URL": { 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_URL", 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.325 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_API_USERNAME": { 2024-04-21T08:56:26.326 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.327 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.328 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.329 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.330 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.331 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.332 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-21T08:56:26.333 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.334 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.335 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "[]", 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.336 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "MULTICLUSTER_CONFIG": { 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "{}", 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.337 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "PROMETHEUS_API_HOST": { 2024-04-21T08:56:26.338 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.339 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.340 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.341 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.342 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.343 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.344 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.345 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.346 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.347 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.348 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.349 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.350 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-21T08:56:26.351 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.352 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_ENABLED": { 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.353 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-21T08:56:26.354 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.355 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.356 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-21T08:56:26.357 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8", 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.358 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "45", 2024-04-21T08:56:26.359 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_ACCESS_KEY": { 2024-04-21T08:56:26.360 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.361 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "admin", 2024-04-21T08:56:26.362 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_SECRET_KEY": { 2024-04-21T08:56:26.363 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.364 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "RGW_API_SSL_VERIFY": { 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.365 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "UNSAFE_TLS_v1_2": { 2024-04-21T08:56:26.366 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.367 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.368 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.369 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.370 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.371 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.372 INFO:teuthology.orchestra.run.smithi057.stdout: "cross_origin_url": { 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cross_origin_url", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.373 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "crt_file": { 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "crt_file", 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.374 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.375 INFO:teuthology.orchestra.run.smithi057.stdout: "debug": { 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "debug", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Enable/disable debug options", 2024-04-21T08:56:26.376 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "jwt_token_ttl": { 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "jwt_token_ttl", 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "28800", 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.378 INFO:teuthology.orchestra.run.smithi057.stdout: "key_file": { 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "key_file", 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.379 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.380 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.381 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.382 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.383 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.384 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.385 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.386 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.387 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "motd": { 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "motd", 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.388 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "The message of the day", 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.389 INFO:teuthology.orchestra.run.smithi057.stdout: "redirect_resolve_ip_addr": { 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "redirect_resolve_ip_addr", 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.390 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.391 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "::", 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.392 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8080", 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.393 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl": { 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl", 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.394 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.395 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl_server_port": { 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl_server_port", 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8443", 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.396 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_behaviour": { 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_behaviour", 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.397 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "redirect", 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "redirect" 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.398 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_error_status_code": { 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "500", 2024-04-21T08:56:26.399 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "400", 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "599", 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "url_prefix": { 2024-04-21T08:56:26.400 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "url_prefix", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.401 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "devicehealth", 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.402 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "enable_monitoring": { 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enable_monitoring", 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.403 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "monitor device health metrics", 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.404 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.405 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.406 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.407 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.408 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.409 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.410 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.411 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "mark_out_threshold": { 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mark_out_threshold", 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.412 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "2419200", 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.413 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "pool_name": { 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pool_name", 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "device_health_metrics", 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.414 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "retention_period": { 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "retention_period", 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.415 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.416 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15552000", 2024-04-21T08:56:26.416 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.416 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.416 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.416 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long to retain device health metrics", 2024-04-21T08:56:26.416 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "scrape_frequency": { 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "scrape_frequency", 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.417 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "86400", 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.418 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "self_heal": { 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "self_heal", 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.419 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:26.420 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how frequently to wake up and check device health", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.421 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "warn_threshold": { 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "warn_threshold", 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.422 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "7257600", 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.423 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "diskprediction_local", 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.424 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.425 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.426 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.427 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.428 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.429 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.430 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.431 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.432 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "predict_interval": { 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "predict_interval", 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.433 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "86400", 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.434 INFO:teuthology.orchestra.run.smithi057.stdout: "predictor_model": { 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "predictor_model", 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "prophetstor", 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.435 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.436 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "600", 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.437 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "influx", 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": false, 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "influxdb python module not found", 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.438 INFO:teuthology.orchestra.run.smithi057.stdout: "batch_size": { 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "batch_size", 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5000", 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.439 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "database": { 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "database", 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.440 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "ceph", 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.441 INFO:teuthology.orchestra.run.smithi057.stdout: "hostname": { 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "hostname", 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.442 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB server hostname", 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.443 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "30", 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "5", 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.444 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.445 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.446 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.447 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.448 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.449 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.450 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.451 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.452 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "password": { 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "password", 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.453 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "password of InfluxDB server user", 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.454 INFO:teuthology.orchestra.run.smithi057.stdout: "port": { 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "port", 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "8086", 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.455 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "InfluxDB server port", 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "ssl": { 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ssl", 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.456 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "false", 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.457 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "threads": { 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "threads", 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:26.458 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "32", 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "username": { 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "username", 2024-04-21T08:56:26.459 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.460 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "username of InfluxDB server user", 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "verify_ssl": { 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "verify_ssl", 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.461 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "true", 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.462 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "insights", 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.463 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.464 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.465 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.466 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.467 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.468 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.469 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.470 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.471 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "iostat", 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.472 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.473 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.474 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.475 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.476 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.477 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.478 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.479 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.480 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "k8sevents", 2024-04-21T08:56:26.481 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "ceph_event_retention_days": { 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "ceph_event_retention_days", 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.482 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "7", 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.483 INFO:teuthology.orchestra.run.smithi057.stdout: "config_check_secs": { 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "config_check_secs", 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "10", 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.484 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.485 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.486 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.487 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.488 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.489 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.490 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.491 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.492 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.493 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "localpool", 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.494 INFO:teuthology.orchestra.run.smithi057.stdout: "failure_domain": { 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "failure_domain", 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "host", 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.495 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "failure domain for any created local pool", 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.496 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.497 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.498 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.499 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.500 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.501 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.502 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.503 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "min_size": { 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "min_size", 2024-04-21T08:56:26.504 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default min_size for any created local pool", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-21T08:56:26.505 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "num_rep": { 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "num_rep", 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.506 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3", 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default replica count for any created local pool", 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.507 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_num": { 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pg_num", 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "128", 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.508 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "default pg_num for any created local pool", 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "prefix": { 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prefix", 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.509 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "name prefix for any created local pool", 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.510 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "subtree": { 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "subtree", 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "rack", 2024-04-21T08:56:26.511 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.512 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mds_autoscaler", 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.513 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.514 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.515 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.516 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.517 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.518 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.519 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.520 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.521 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mirroring", 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.522 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.523 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.524 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.525 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.526 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.527 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.528 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.529 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.530 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "nfs", 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.531 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.532 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.533 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.534 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.535 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.536 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.537 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.538 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.539 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "orchestrator", 2024-04-21T08:56:26.540 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "fail_fs": { 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "fail_fs", 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.541 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.542 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.543 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.544 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.545 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.546 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.547 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.548 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.549 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.550 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator": { 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "orchestrator", 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.551 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "cephadm", 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "rook", 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "test_orchestrator" 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Orchestrator backend", 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.552 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "osd_perf_query", 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.553 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.554 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.555 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.556 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.557 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.558 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.559 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.560 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.561 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.562 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "osd_support", 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.563 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.564 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.565 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.566 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.567 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.568 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.569 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.570 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.571 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "pg_autoscaler", 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.572 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.573 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.574 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.575 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.576 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.577 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.578 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.579 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:26.580 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.581 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "threshold": { 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "threshold", 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "3.0", 2024-04-21T08:56:26.582 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1.0", 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "scaling threshold", 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.583 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "progress", 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_pg_recovery_event": { 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_pg_recovery_event", 2024-04-21T08:56:26.584 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.585 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "enabled": { 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enabled", 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.586 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.587 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.588 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.589 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.590 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.591 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.592 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.593 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.594 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "max_completed_events": { 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_completed_events", 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.595 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "50", 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "number of past completed events to remember", 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.596 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "sleep_interval": { 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "sleep_interval", 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "5", 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.597 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "how long the module is going to sleep", 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.598 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "prometheus", 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "cache": { 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "cache", 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.599 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "exclude_perf_counters": { 2024-04-21T08:56:26.600 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "exclude_perf_counters", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-21T08:56:26.601 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.602 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.603 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.604 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.605 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.606 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.607 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.608 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.609 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_stats_pools": { 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_stats_pools", 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.610 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-21T08:56:26.611 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "300", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.612 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "scrape_interval": { 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "scrape_interval", 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.613 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15.0", 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.614 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "::", 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.615 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:26.616 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "9283", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-21T08:56:26.617 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "stale_cache_strategy": { 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "stale_cache_strategy", 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.618 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "log", 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.619 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_behaviour": { 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_behaviour", 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "default", 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.620 INFO:teuthology.orchestra.run.smithi057.stdout: "default", 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "error" 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "standby_error_status_code": { 2024-04-21T08:56:26.621 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "standby_error_status_code", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "500", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "400", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "599", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.622 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_support", 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.623 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.624 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.625 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.626 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.627 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.628 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.629 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.630 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.631 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "max_concurrent_snap_create": { 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_concurrent_snap_create", 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.632 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10", 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: "mirror_snapshot_schedule": { 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "mirror_snapshot_schedule", 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.634 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "trash_purge_schedule": { 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "trash_purge_schedule", 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.635 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.636 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "restful", 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: "enable_auth": { 2024-04-21T08:56:26.637 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enable_auth", 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.638 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "key_file": { 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "key_file", 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.639 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.640 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.641 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.642 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.643 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.644 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.645 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.646 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.647 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.648 INFO:teuthology.orchestra.run.smithi057.stdout: "server_addr": { 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_addr", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.649 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "server_port": { 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "server_port", 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.650 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.651 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rgw", 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.652 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.653 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.654 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.655 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.656 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.657 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.658 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.659 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rook", 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.660 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.661 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.662 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.663 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.664 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.665 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.666 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.667 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.668 INFO:teuthology.orchestra.run.smithi057.stdout: "storage_class": { 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "storage_class", 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "local", 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.669 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "selftest", 2024-04-21T08:56:26.670 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.671 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.672 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.673 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.674 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.676 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.677 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.678 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "roption1": { 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "roption1", 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.679 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "roption2": { 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "roption2", 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.680 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "xyz", 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.681 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption1": { 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption1", 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.682 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption2": { 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption2", 2024-04-21T08:56:26.683 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.684 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption3": { 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption3", 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "float", 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.685 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption4": { 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption4", 2024-04-21T08:56:26.686 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.687 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption5": { 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption5", 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.688 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.689 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption6": { 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption6", 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.690 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "rwoption7": { 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rwoption7", 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.691 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "1", 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "42", 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.692 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "testkey": { 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testkey", 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.693 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "testlkey": { 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testlkey", 2024-04-21T08:56:26.694 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.695 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "testnewline": { 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "testnewline", 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.696 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.697 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snap_schedule", 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.698 INFO:teuthology.orchestra.run.smithi057.stdout: "allow_m_granularity": { 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "allow_m_granularity", 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.699 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "allow minute scheduled snapshots", 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "dump_on_update": { 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "dump_on_update", 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.700 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "dump database to debug log on update", 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.701 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.702 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.703 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.704 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.705 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.706 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.707 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.708 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.709 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "stats", 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.710 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.711 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.712 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.713 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.714 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.715 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.716 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.717 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.718 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "status", 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.719 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.720 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.721 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.722 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.723 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.724 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.725 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.726 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.727 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "telegraf", 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.728 INFO:teuthology.orchestra.run.smithi057.stdout: "address": { 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "address", 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.729 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.730 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "15", 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.731 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.732 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.733 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.734 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.735 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.736 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.737 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.738 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.739 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.740 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "telemetry", 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_basic": { 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_basic", 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.741 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.742 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_crash": { 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_crash", 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.743 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_device": { 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_device", 2024-04-21T08:56:26.744 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.745 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_ident": { 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_ident", 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.746 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.747 INFO:teuthology.orchestra.run.smithi057.stdout: "channel_perf": { 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "channel_perf", 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.748 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "contact": { 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "contact", 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.749 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.750 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "description": { 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "description", 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.751 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "device_url": { 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "device_url", 2024-04-21T08:56:26.752 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.753 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "enabled": { 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "enabled", 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.754 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.755 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "24", 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "8", 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.756 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "last_opt_revision": { 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "last_opt_revision", 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.757 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "1", 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.758 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "leaderboard": { 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "leaderboard", 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.759 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "leaderboard_description": { 2024-04-21T08:56:26.760 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "leaderboard_description", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.761 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.762 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.763 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.764 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.765 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.766 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.767 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.768 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.769 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "organization": { 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "organization", 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.770 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.771 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "proxy": { 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "proxy", 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.772 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "url": { 2024-04-21T08:56:26.773 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "url", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.774 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "test_orchestrator", 2024-04-21T08:56:26.775 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.776 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.777 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.778 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.779 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.780 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.781 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.782 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.783 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.784 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "volumes", 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.785 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.786 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.787 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.788 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.789 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.790 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.791 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.792 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.793 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "max_concurrent_clones": { 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "max_concurrent_clones", 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "4", 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.794 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "periodic_async_work": { 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "periodic_async_work", 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.795 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Periodically check for async work", 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.796 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "snapshot_clone_delay": { 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snapshot_clone_delay", 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.797 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "0", 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.798 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "snapshot_clone_no_wait": { 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snapshot_clone_no_wait", 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "True", 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.799 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.800 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix", 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "can_run": true, 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "error_string": "", 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "module_options": { 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "discovery_interval": { 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "discovery_interval", 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "uint", 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.801 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "100", 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.802 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "identifier": { 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "identifier", 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.803 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "interval": { 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "interval", 2024-04-21T08:56:26.804 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "secs", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "60", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.805 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: "log_level": { 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_level", 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.806 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.807 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.808 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster": { 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster", 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.809 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_cluster_level": { 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_cluster_level", 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.810 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "info", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [ 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "critical", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "debug", 2024-04-21T08:56:26.811 INFO:teuthology.orchestra.run.smithi057.stdout: "error", 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "info", 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "warning" 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.812 INFO:teuthology.orchestra.run.smithi057.stdout: "log_to_file": { 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "log_to_file", 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "bool", 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 1, 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "False", 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.813 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_host": { 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_host", 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.814 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "", 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.815 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_port": { 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_port", 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "int", 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "10051", 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.816 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "zabbix_sender": { 2024-04-21T08:56:26.817 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "zabbix_sender", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "str", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "level": "advanced", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "flags": 0, 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "min": "", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "max": "", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "enum_allowed": [], 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "desc": "", 2024-04-21T08:56:26.818 INFO:teuthology.orchestra.run.smithi057.stdout: "long_desc": "", 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: "tags": [], 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: "see_also": [] 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: "services": { 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: "dashboard": "https://172.21.15.57:8443/" 2024-04-21T08:56:26.819 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "always_on_modules": { 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "octopus": [ 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "balancer", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "crash", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "devicehealth", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_autoscaler", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "progress", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_support", 2024-04-21T08:56:26.820 INFO:teuthology.orchestra.run.smithi057.stdout: "status", 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "telemetry", 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "volumes" 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "pacific": [ 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "balancer", 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "crash", 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "devicehealth", 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator", 2024-04-21T08:56:26.821 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_autoscaler", 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "progress", 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_support", 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "status", 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "telemetry", 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "volumes" 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "quincy": [ 2024-04-21T08:56:26.822 INFO:teuthology.orchestra.run.smithi057.stdout: "balancer", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "crash", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "devicehealth", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_autoscaler", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "progress", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_support", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "status", 2024-04-21T08:56:26.823 INFO:teuthology.orchestra.run.smithi057.stdout: "telemetry", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "volumes" 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "reef": [ 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "balancer", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "crash", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "devicehealth", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_autoscaler", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "progress", 2024-04-21T08:56:26.824 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_support", 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "status", 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "telemetry", 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "volumes" 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: ], 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "squid": [ 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "balancer", 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "crash", 2024-04-21T08:56:26.825 INFO:teuthology.orchestra.run.smithi057.stdout: "devicehealth", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "orchestrator", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "pg_autoscaler", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "progress", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "rbd_support", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "status", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "telemetry", 2024-04-21T08:56:26.826 INFO:teuthology.orchestra.run.smithi057.stdout: "volumes" 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: "last_failure_osd_epoch": 94, 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: "active_clients": [ 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "devicehealth", 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:26.827 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 276171688 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "libcephsqlite", 2024-04-21T08:56:26.828 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3060699840 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.829 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "rbd_support", 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 2411619731 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.830 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snap_schedule", 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2024-04-21T08:56:26.831 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 194818314 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "snap_schedule", 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.832 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 1970451505 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: "name": "volumes", 2024-04-21T08:56:26.833 INFO:teuthology.orchestra.run.smithi057.stdout: "addrvec": [ 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: { 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: "type": "v2", 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: "addr": "172.21.15.57:0", 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: "nonce": 3317041343 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:26.834 INFO:teuthology.orchestra.run.smithi057.stdout: ] 2024-04-21T08:56:26.835 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:26.836 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:25 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1507610295' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:56:26.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:24 smithi129 ceph-mon[31272]: pgmap v345: 129 pgs: 16 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 71 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 846 KiB/s wr, 122 op/s 2024-04-21T08:56:26.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:24 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3294072716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:56:26.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:24 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4249518989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:56:26.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:25 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1507610295' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:56:26.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:26 smithi129 ceph-mon[31272]: pgmap v346: 129 pgs: 10 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 384 KiB/s wr, 95 op/s 2024-04-21T08:56:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7faff00a3d00 msgr2=0x7faff00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:26.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7faff00a3d00 0x7faff00a61c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb000009990 tx=0x7fb000009110 comp rx=0 tx=0).stop 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 msgr2=0x7fb00411af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 0x7fb00411af30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7faff403d280 tx=0x7faff40045c0 comp rx=0 tx=0).stop 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 shutdown_connections 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7faff00a3d00 0x7faff00a61c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb004117960 0x7fb0041ab4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb004116fc0 0x7fb004117420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 --2- 172.21.15.57:0/1507610295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0041106b0 0x7fb00411af30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:56:26.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.714+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 >> 172.21.15.57:0/1507610295 conn(0x7fb00406d070 msgr2=0x7fb00411c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:56:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.715+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 shutdown_connections 2024-04-21T08:56:26.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:56:24.715+0000 7fb00ba8c640 1 -- 172.21.15.57:0/1507610295 wait complete. 2024-04-21T08:56:26.846 DEBUG:tasks.check_counter:Getting stats from a 2024-04-21T08:56:26.846 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-21T08:56:26.847 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:24 smithi057 ceph-mon[27968]: pgmap v345: 129 pgs: 16 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 71 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 846 KiB/s wr, 122 op/s 2024-04-21T08:56:26.847 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:24 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3294072716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:56:26.847 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:24 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4249518989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:56:26.847 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:25 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1507610295' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-21T08:56:26.848 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:26 smithi057 ceph-mon[27968]: pgmap v346: 129 pgs: 10 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 384 KiB/s wr, 95 op/s 2024-04-21T08:56:26.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:26 smithi136 ceph-mon[31791]: pgmap v346: 129 pgs: 10 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 384 KiB/s wr, 95 op/s 2024-04-21T08:56:27.093 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:28.551 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:28 smithi057 ceph-mon[27968]: pgmap v347: 129 pgs: 7 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 100 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 47 KiB/s wr, 35 op/s 2024-04-21T08:56:28.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:28 smithi129 ceph-mon[31272]: pgmap v347: 129 pgs: 7 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 100 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 47 KiB/s wr, 35 op/s 2024-04-21T08:56:28.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:28 smithi136 ceph-mon[31791]: pgmap v347: 129 pgs: 7 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 100 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 47 KiB/s wr, 35 op/s 2024-04-21T08:56:28.974 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:28.974 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.001776264, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.000045917, 2024-04-21T08:56:28.975 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.000599842, 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 3, 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000705728, 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000235242 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:28.976 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 232, 2024-04-21T08:56:28.977 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 221, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 1592432, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 135744, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.044851048, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.013113855, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.036553064, 2024-04-21T08:56:28.978 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.004726463, 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 220, 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.011642397, 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000052919 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:28.979 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 1592432, 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 135744 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:28.980 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 168, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 185552, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.012600603, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.008402310, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.002003706, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:28.981 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 165, 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.015303089, 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000092745 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:28.982 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 185552 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: "cct": { 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: "total_workers": 0, 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: "unhealthy_workers": 0 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.983 INFO:teuthology.orchestra.run.smithi057.stdout: "mempool": { 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:28.984 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:28.985 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:28.986 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_items": 0, 2024-04-21T08:56:28.987 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_bytes": 148977, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_items": 30, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_bytes": 0, 2024-04-21T08:56:28.988 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_items": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_items": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:28.989 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_items": 0, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_items": 2, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_items": 0, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_items": 0 2024-04-21T08:56:28.990 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "val": 0, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "max": 104857600, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "get_started": 0, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "get": 234, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "get_sum": 1559724, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:28.991 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_success": 234, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "take": 0, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "take_sum": 0, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "put": 234, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "put_sum": 1559724, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "wait": { 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:28.992 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:28.993 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:28.993 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:29.747 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-21T08:56:29.747 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-21T08:56:29.747 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-21T08:56:29.747 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-04-21T08:56:29.748 DEBUG:tasks.check_counter:Getting stats from d 2024-04-21T08:56:29.748 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-21T08:56:30.003 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:30.687 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:30 smithi136 ceph-mon[31791]: pgmap v348: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 11 op/s 2024-04-21T08:56:30.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:30 smithi129 ceph-mon[31272]: pgmap v348: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 11 op/s 2024-04-21T08:56:30.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:30 smithi057 ceph-mon[27968]: pgmap v348: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 11 op/s 2024-04-21T08:56:31.554 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:31 smithi057 ceph-mon[27968]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T08:56:31.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:31 smithi129 ceph-mon[31272]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T08:56:31.841 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:31.841 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:31.841 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:31.841 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.001935091, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.000048370, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.000686369, 2024-04-21T08:56:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 3, 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000814785, 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000271595 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:31.843 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 237, 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 226, 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 1593792, 2024-04-21T08:56:31.844 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 138944, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.046100923, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.013867948, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.037171306, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.004799782, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 225, 2024-04-21T08:56:31.845 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.035859185, 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000159374 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.846 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 1593792, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 138944 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 173, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 186432, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:31.847 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.013706180, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.009252084, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.001870618, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 170, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.017388271, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000102283 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:31.848 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 186432 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "cct": { 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "total_workers": 0, 2024-04-21T08:56:31.849 INFO:teuthology.orchestra.run.smithi057.stdout: "unhealthy_workers": 0 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "mempool": { 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:31.850 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:31.851 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:31.852 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:31.853 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:31.853 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:31.853 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:31.853 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:31.853 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:31.853 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_items": 0, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_bytes": 148977, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_items": 30, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:31.854 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_bytes": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_items": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:31.855 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_items": 0, 2024-04-21T08:56:31.856 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:31.856 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:31.856 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:31.856 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_items": 0, 2024-04-21T08:56:31.856 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_items": 2, 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_items": 0, 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_items": 0 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:31.857 INFO:teuthology.orchestra.run.smithi057.stdout: "val": 0, 2024-04-21T08:56:31.858 INFO:teuthology.orchestra.run.smithi057.stdout: "max": 104857600, 2024-04-21T08:56:31.858 INFO:teuthology.orchestra.run.smithi057.stdout: "get_started": 0, 2024-04-21T08:56:31.858 INFO:teuthology.orchestra.run.smithi057.stdout: "get": 239, 2024-04-21T08:56:31.858 INFO:teuthology.orchestra.run.smithi057.stdout: "get_sum": 1560374, 2024-04-21T08:56:31.858 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:31.858 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_success": 239, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "take": 0, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "take_sum": 0, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "put": 239, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "put_sum": 1560374, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "wait": { 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:31.859 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:31.860 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:31.860 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:31.860 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:31.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:31 smithi136 ceph-mon[31791]: osdmap e108: 12 total, 12 up, 12 in 2024-04-21T08:56:32.717 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:32 smithi057 ceph-mon[27968]: pgmap v350: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-21T08:56:32.719 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-21T08:56:32.719 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-21T08:56:32.719 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-21T08:56:32.719 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-21T08:56:32.719 DEBUG:tasks.check_counter:Getting stats from g 2024-04-21T08:56:32.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-21T08:56:32.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:32 smithi129 ceph-mon[31272]: pgmap v350: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-21T08:56:32.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:32 smithi136 ceph-mon[31791]: pgmap v350: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 27 KiB/s wr, 3 op/s 2024-04-21T08:56:33.024 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:34.787 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:34 smithi057 ceph-mon[27968]: pgmap v351: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 35 KiB/s wr, 3 op/s 2024-04-21T08:56:34.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:34 smithi129 ceph-mon[31272]: pgmap v351: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 35 KiB/s wr, 3 op/s 2024-04-21T08:56:34.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:34 smithi136 ceph-mon[31791]: pgmap v351: 129 pgs: 129 active+clean; 1.4 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 35 KiB/s wr, 3 op/s 2024-04-21T08:56:35.170 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:35.170 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:35.170 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 234, 2024-04-21T08:56:35.170 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 223, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 1592976, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 137024, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.043484037, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.013002974, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.034615679, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.004920353, 2024-04-21T08:56:35.171 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 222, 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.010962101, 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000049378 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:35.172 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 1592976, 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 137024 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 170, 2024-04-21T08:56:35.173 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 185904, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.013918161, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.009669706, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.001784235, 2024-04-21T08:56:35.174 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:35.175 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:35.175 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 167, 2024-04-21T08:56:35.175 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.022400418, 2024-04-21T08:56:35.175 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000134134 2024-04-21T08:56:35.175 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 185904 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.176 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.000738731, 2024-04-21T08:56:35.177 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.000047388, 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.000697777, 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 3, 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000810537, 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000270179 2024-04-21T08:56:35.178 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:35.179 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: "cct": { 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: "total_workers": 0, 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: "unhealthy_workers": 0 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: "mempool": { 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:35.180 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:35.181 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:35.182 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:35.183 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_items": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:35.184 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_bytes": 148995, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_items": 31, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_bytes": 0, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_items": 0, 2024-04-21T08:56:35.185 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_items": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:35.186 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_items": 0, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_items": 2, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_items": 0, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_items": 0 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:35.187 INFO:teuthology.orchestra.run.smithi057.stdout: "val": 0, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "max": 104857600, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "get_started": 0, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "get": 236, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "get_sum": 1559984, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_success": 236, 2024-04-21T08:56:35.188 INFO:teuthology.orchestra.run.smithi057.stdout: "take": 0, 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "take_sum": 0, 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "put": 236, 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "put_sum": 1559984, 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "wait": { 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:35.189 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:35.890 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-21T08:56:35.890 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-21T08:56:35.891 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-21T08:56:35.891 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-04-21T08:56:35.891 DEBUG:tasks.check_counter:Getting stats from j 2024-04-21T08:56:35.891 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-21T08:56:36.148 INFO:teuthology.orchestra.run.smithi057.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.a/config 2024-04-21T08:56:36.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:36 smithi129 ceph-mon[31272]: pgmap v352: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 255 KiB/s wr, 12 op/s 2024-04-21T08:56:36.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:36 smithi129 ceph-mon[31272]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T08:56:36.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:36 smithi057 ceph-mon[27968]: pgmap v352: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 255 KiB/s wr, 12 op/s 2024-04-21T08:56:36.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:36 smithi057 ceph-mon[27968]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T08:56:36.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:36 smithi136 ceph-mon[31791]: pgmap v352: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 255 KiB/s wr, 12 op/s 2024-04-21T08:56:36.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:36 smithi136 ceph-mon[31791]: osdmap e109: 12 total, 12 up, 12 in 2024-04-21T08:56:38.086 INFO:teuthology.orchestra.run.smithi057.stdout:{ 2024-04-21T08:56:38.086 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:38.086 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:38.086 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.001620778, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.000038615, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.000639988, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:38.087 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 3, 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000747945, 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000249315 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:38.088 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 240, 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 229, 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 1594608, 2024-04-21T08:56:38.089 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 140864, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.047919549, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.014449197, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.038236629, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.005027000, 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:38.090 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 228, 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.011017049, 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000048320 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:38.091 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 1594608, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 140864 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages": 175, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_bytes": 186784, 2024-04-21T08:56:38.092 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_total_time": 0.014722536, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_send_time": 0.009835220, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_recv_time": 0.002077368, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 172, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.019593734, 2024-04-21T08:56:38.093 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000113917 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: "msgr_send_encrypted_bytes": 186784 2024-04-21T08:56:38.094 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "cct": { 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "total_workers": 0, 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "unhealthy_workers": 0 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "mempool": { 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:38.095 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:38.096 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:38.097 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:38.098 INFO:teuthology.orchestra.run.smithi057.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_items": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_bytes": 148771, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_anon_items": 32, 2024-04-21T08:56:38.099 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_bytes": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_items": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_items": 0, 2024-04-21T08:56:38.100 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "pgmap_items": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "mds_co_items": 2, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_1_items": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:38.101 INFO:teuthology.orchestra.run.smithi057.stdout: "unittest_2_items": 0 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "val": 0, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "max": 104857600, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "get_started": 0, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "get": 242, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "get_sum": 1560764, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:38.102 INFO:teuthology.orchestra.run.smithi057.stdout: "get_or_fail_success": 242, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "take": 0, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "take_sum": 0, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "put": 242, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "put_sum": 1560764, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "wait": { 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "avgcount": 0, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "sum": 0.000000000, 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: "avgtime": 0.000000000 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:38.103 INFO:teuthology.orchestra.run.smithi057.stdout: } 2024-04-21T08:56:38.104 INFO:teuthology.orchestra.run.smithi057.stdout:} 2024-04-21T08:56:38.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:38 smithi129 ceph-mon[31272]: pgmap v354: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 388 KiB/s wr, 18 op/s 2024-04-21T08:56:38.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:38 smithi057 ceph-mon[27968]: pgmap v354: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 388 KiB/s wr, 18 op/s 2024-04-21T08:56:38.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:38 smithi136 ceph-mon[31791]: pgmap v354: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 388 KiB/s wr, 18 op/s 2024-04-21T08:56:39.246 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-21T08:56:39.246 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-21T08:56:39.246 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-21T08:56:39.246 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-21T08:56:39.247 DEBUG:tasks.check_counter:Getting stats from b 2024-04-21T08:56:39.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-21T08:56:39.664 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:56:40.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:40 smithi129 ceph-mon[31272]: pgmap v355: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 387 KiB/s wr, 20 op/s 2024-04-21T08:56:40.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:40 smithi057 ceph-mon[27968]: pgmap v355: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 387 KiB/s wr, 20 op/s 2024-04-21T08:56:40.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:40 smithi136 ceph-mon[31791]: pgmap v355: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 387 KiB/s wr, 20 op/s 2024-04-21T08:56:41.547 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-04-21T08:56:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 278418, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 277805, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 164959715, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 424670341, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 20, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 5, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 14.893880811, 2024-04-21T08:56:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 4.396951215, 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 2.934347130, 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 6.633737543, 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 277803, 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 136.657010199, 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000491920 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 933, 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000190303, 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000203 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 338768, 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 99632 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 201070, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 200923, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 116380140, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 364538308, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 184, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 6, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 11.638611091, 2024-04-21T08:56:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 3.487728142, 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 2.230535568, 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 5.234780293, 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 200920, 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 123.877283457, 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000616550 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 380, 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000051960, 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000136 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 86448, 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 144464 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 478172, 2024-04-21T08:56:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 663022, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 231012817, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 418097510, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 15, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 6, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 24.815904759, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 9.259538913, 2024-04-21T08:56:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 75.511691930, 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 5.179049528, 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 663022, 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 194.657771109, 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000293591 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 607675, 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.099920761, 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000164 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1699168, 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 531664 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-04-21T08:56:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 1, 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-MDSRank": { 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 357876, 2024-04-21T08:56:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 284.793586500, 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000795788 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "finisher-PurgeQueue": { 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "queue_len": 0, 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "complete_latency": { 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1959, 2024-04-21T08:56:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.231091594, 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000117964 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": { 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 100384, 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "reply": 100384, 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "reply_latency": { 2024-04-21T08:56:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 100384, 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 443.794381541, 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.004420967 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_reply": 0, 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "forward": 0, 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_complete": 0, 2024-04-21T08:56:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_fetch_keys": 0, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_commit": 16331, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_split": 172, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_merge": 0, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes": 100076, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_top": 6787, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_bottom": 93289, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pin_tail": 0, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_pinned": 90379, 2024-04-21T08:56:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_expired": 0, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "inodes_with_caps": 90240, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 90241, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "subtrees": 2, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse": 100859, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_hit": 99863, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_forward": 0, 2024-04-21T08:56:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_discover": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_remote_ino": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "traverse_lock": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "load_cent": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "q": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "exported": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "exported_inodes": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "imported": 0, 2024-04-21T08:56:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "imported_inodes": 0, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_dir_fetch": 0, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_backtrace_fetch": 506957, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "openino_peer_discover": 0, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rfiles": 84949, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rbytes": 1551143202, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "root_rsnaps": 6, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_fetch": 506957, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_set_tag": 0, 2024-04-21T08:56:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_inodes": 29437, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dir_base_inodes": 12, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_dirfrag_rstats": 29425, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_file_inodes": 477520, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_revoke": 27141, 2024-04-21T08:56:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_grant": 95882, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_trunc": 74, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps": 135586, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_caps_dirty": 108284, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_cap_release": 3115, 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "process_request_cap_release": 13289 2024-04-21T08:56:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_cache": { 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update": 0, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_update_receipt": 0, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_try_discover": 0, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_send_discover": 0, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_handle_discover": 0, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays": 0, 2024-04-21T08:56:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_delayed": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_created": 1168, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_enqueued": 1168, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_reintegrated": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "strays_migrated": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_processing": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:56:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_started": 0, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "recovery_completed": 0, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_path": 1, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_quiesce_inode": 21327, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_enqueue_scrub": 12, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_exportdir": 0, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_flush": 0, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragmentdir": 175, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_fragstats": 0, 2024-04-21T08:56:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "ireq_inodestats": 0 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_log": { 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "evlrg": 0, 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "evadd": 218093, 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "evex": 217572, 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "evtrm": 217572, 2024-04-21T08:56:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "ev": 521, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "evexg": 0, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "evexd": 1468, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "segadd": 282, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "segex": 280, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "segtrm": 280, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "seg": 2, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "segmjr": 1, 2024-04-21T08:56:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "segexg": 0, 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "segexd": 2, 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "expos": 539308469, 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "wrpos": 541128564, 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "rdpos": 0, 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "jlat": { 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 14102, 2024-04-21T08:56:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 2576.058038328, 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.182673240 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "replayed": 0 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_mem": { 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 98683, 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "ino+": 99853, 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "ino-": 1170, 2024-04-21T08:56:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "dir": 6515, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "dir+": 6796, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "dir-": 281, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "dn": 100076, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "dn+": 101696, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "dn-": 1620, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "cap": 90241, 2024-04-21T08:56:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "cap+": 112734, 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "cap-": 22493, 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "rss": 561620, 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "heap": 231800 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_server": { 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_client_request": 105648, 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "dispatch_server_request": 0, 2024-04-21T08:56:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_request": 100384, 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_client_session": 36, 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "handle_peer_request": 0, 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "req_create_latency": { 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 85946, 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 383.940496183, 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.004467229 2024-04-21T08:56:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getattr_latency": { 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 251, 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.196278530, 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000781986 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getfilelock_latency": { 2024-04-21T08:56:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "req_link_latency": { 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookup_latency": { 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 62, 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.079748294, 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001286262 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookuphash_latency": { 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupino_latency": { 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 2, 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000317843, 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000158921 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupname_latency": { 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupparent_latency": { 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lookupsnap_latency": { 2024-04-21T08:56:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "req_lssnap_latency": { 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 11, 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.017481948, 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001589268 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mkdir_latency": { 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 5408, 2024-04-21T08:56:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 18.047474237, 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.003337180 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mknod_latency": { 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "req_mksnap_latency": { 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 11, 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 1.326799927, 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.120618175 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "req_open_latency": { 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 713, 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.430220061, 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000603394 2024-04-21T08:56:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "req_readdir_latency": { 2024-04-21T08:56:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 6569, 2024-04-21T08:56:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 26.786842693, 2024-04-21T08:56:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.004077765 2024-04-21T08:56:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rename_latency": { 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 30, 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.055640611, 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001854687 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "req_renamesnap_latency": { 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "req_snapdiff_latency": { 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmdir_latency": { 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 109, 2024-04-21T08:56:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 3.003481260, 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.027554873 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmsnap_latency": { 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 5, 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.840294706, 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.168058941 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "req_rmxattr_latency": { 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setattr_latency": { 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 138, 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 5.348923681, 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.038760316 2024-04-21T08:56:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setdirlayout_latency": { 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setfilelock_latency": { 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setlayout_latency": { 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "req_setxattr_latency": { 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 8, 2024-04-21T08:56:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.021704942, 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.002713117 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "req_symlink_latency": { 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 62, 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.066204782, 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.001067819 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "req_unlink_latency": { 2024-04-21T08:56:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 1059, 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 3.632471843, 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.003430096 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "req_getvxattr_latency": { 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_sessions": { 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "session_count": 2, 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "session_add": 6, 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "session_remove": 4, 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_open": 2, 2024-04-21T08:56:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "sessions_stale": 0, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "total_load": 6812, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "average_load": 3406, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "avg_session_uptime": 660, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "mdthresh_evicted": 0 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 3722794, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 90385, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 88, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 1, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 9408, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 180, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 358489611, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 4568364, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "objecter": { 2024-04-21T08:56:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "op_active": 0, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "op_laggy": 0, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send": 717759, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "op_send_bytes": 760936599, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "op_resend": 32, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "op_reply": 717759, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "op_latency": { 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 717759, 2024-04-21T08:56:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 255096.981759498, 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.355407569 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "op_inflight": 0, 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "oplen_avg": { 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 717759, 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 1010531 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "op": 717759, 2024-04-21T08:56:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "op_r": 507050, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "op_w": 210709, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "op_rmw": 0, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "op_pg": 0, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_stat": 9644, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_create": 176224, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_read": 81, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_write": 14171, 2024-04-21T08:56:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writefull": 225, 2024-04-21T08:56:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_writesame": 0, 2024-04-21T08:56:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_append": 0, 2024-04-21T08:56:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_zero": 0, 2024-04-21T08:56:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_truncate": 149, 2024-04-21T08:56:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_delete": 3017, 2024-04-21T08:56:41.607 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_mapext": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_sparse_read": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_clonerange": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_getxattr": 506969, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_setxattr": 267114, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_rmxattr": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_call": 0, 2024-04-21T08:56:41.608 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_watch": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_notify": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "osdop_other": 74, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_active": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_send": 0, 2024-04-21T08:56:41.609 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_resend": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "linger_ping": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_active": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_send": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "poolop_resend": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_active": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_send": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "poolstat_resend": 0, 2024-04-21T08:56:41.610 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_active": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_send": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "statfs_resend": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "command_active": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "command_send": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "command_resend": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "map_epoch": 109, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "map_full": 0, 2024-04-21T08:56:41.611 INFO:teuthology.orchestra.run.smithi129.stdout: "map_inc": 25, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_sessions": 12, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_open": 12, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_session_close": 0, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_laggy": 0, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_wr": 32306, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_rd": 0, 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_del": 557 2024-04-21T08:56:41.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "oft": { 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_objs": 1, 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_updates": 0, 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "omap_total_removes": 0 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "osd-slow-ops": { 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: "slow_ops_count": 21332 2024-04-21T08:56:41.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "purge_queue": { 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops": 0, 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_ops_high_water": 49, 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing": 0, 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executing_high_water": 40, 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed_ops": 2227, 2024-04-21T08:56:41.614 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_executed": 1168, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "pq_item_in_journal": 0 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 957656, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 426093942, 2024-04-21T08:56:41.615 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 957656, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 957656, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 426093942, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.616 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_bytes": { 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:41.617 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 717759, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 761063191, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 717759, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 761063191, 2024-04-21T08:56:41.618 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-objecter_ops": { 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:41.619 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 1024, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 0, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 0, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 717759, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 717759, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 717759, 2024-04-21T08:56:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 717759, 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:56:41.621 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 1168, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 126592, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 1168, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-04-21T08:56:41.622 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 69, 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 126592, 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:41.623 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-write_buf_throttle-0x563521974500": { 2024-04-21T08:56:41.624 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:41.625 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 3758096384, 2024-04-21T08:56:41.625 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:41.625 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 218093, 2024-04-21T08:56:41.625 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 536934260, 2024-04-21T08:56:41.625 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 218093, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 14102, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 536934260, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:41.627 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-04-21T08:56:42.423 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-21T08:56:42.423 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-04-21T08:56:42.423 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-04-21T08:56:42.424 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=172 2024-04-21T08:56:42.424 DEBUG:tasks.check_counter:Getting stats from e 2024-04-21T08:56:42.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-21T08:56:42.669 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:56:42.793 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:42 smithi129 ceph-mon[31272]: pgmap v356: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 386 KiB/s wr, 21 op/s 2024-04-21T08:56:42.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:42 smithi057 ceph-mon[27968]: pgmap v356: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 386 KiB/s wr, 21 op/s 2024-04-21T08:56:42.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:42 smithi136 ceph-mon[31791]: pgmap v356: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 386 KiB/s wr, 21 op/s 2024-04-21T08:56:44.598 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:44 smithi129 ceph-mon[31272]: pgmap v357: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 384 KiB/s wr, 22 op/s 2024-04-21T08:56:44.658 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 240, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 229, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1594608, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 140864, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:44.659 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.044501576, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.012873340, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.034881580, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.005386604, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 228, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.011468559, 2024-04-21T08:56:44.660 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000050300 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1594608, 2024-04-21T08:56:44.661 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 140864 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 175, 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:44.662 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 186784, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.009469766, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.006196069, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.001197645, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:44.663 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 172, 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.019655828, 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000114278 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:44.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 186784 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:44.665 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.000592053, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000045570, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000517111, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 3, 2024-04-21T08:56:44.666 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000757798, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000252599 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:44.667 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-04-21T08:56:44.668 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:44.669 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:44.670 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:44.671 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:44.672 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 148923, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 29, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:44.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:44.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-04-21T08:56:44.675 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 242, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1560764, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 242, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 242, 2024-04-21T08:56:44.676 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1560764, 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:44.677 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-04-21T08:56:44.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:44 smithi057 ceph-mon[27968]: pgmap v357: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 384 KiB/s wr, 22 op/s 2024-04-21T08:56:44.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:44 smithi136 ceph-mon[31791]: pgmap v357: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 384 KiB/s wr, 22 op/s 2024-04-21T08:56:45.723 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-21T08:56:45.723 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-21T08:56:45.723 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-21T08:56:45.723 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-04-21T08:56:45.724 DEBUG:tasks.check_counter:Getting stats from h 2024-04-21T08:56:45.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-21T08:56:45.974 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:56:46.657 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:46 smithi129 ceph-mon[31272]: pgmap v358: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 138 KiB/s wr, 13 op/s 2024-04-21T08:56:46.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:46 smithi057 ceph-mon[27968]: pgmap v358: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 138 KiB/s wr, 13 op/s 2024-04-21T08:56:46.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:46 smithi136 ceph-mon[31791]: pgmap v358: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 138 KiB/s wr, 13 op/s 2024-04-21T08:56:47.848 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:47.849 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.001332508, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000035504, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000496026, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 3, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000735534, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000245178 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.850 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.851 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 174, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 186608, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.009682238, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.006710438, 2024-04-21T08:56:47.852 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000970325, 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 171, 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.013542969, 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000079198 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:47.853 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 186608 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:47.854 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 239, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 228, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1594336, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 140224, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.041545601, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.012148677, 2024-04-21T08:56:47.855 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.032970535, 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.005427356, 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 227, 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.010448749, 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000046029 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:47.856 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1594336, 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 140224 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-04-21T08:56:47.857 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:47.858 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:47.859 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:47.860 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:47.861 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 148935, 2024-04-21T08:56:47.862 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 28, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:47.863 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:47.864 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 241, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1560634, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 241, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 241, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1560634, 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:47.866 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:47.867 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:47.867 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:47.867 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:47.867 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:47.867 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-04-21T08:56:48.509 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-04-21T08:56:48.509 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-04-21T08:56:48.509 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-04-21T08:56:48.509 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2024-04-21T08:56:48.509 DEBUG:tasks.check_counter:Getting stats from k 2024-04-21T08:56:48.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-21T08:56:48.750 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.b/config 2024-04-21T08:56:48.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:48 smithi129 ceph-mon[31272]: pgmap v359: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 118 KiB/s wr, 12 op/s 2024-04-21T08:56:48.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:48 smithi057 ceph-mon[27968]: pgmap v359: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 118 KiB/s wr, 12 op/s 2024-04-21T08:56:48.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:48 smithi136 ceph-mon[31791]: pgmap v359: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 118 KiB/s wr, 12 op/s 2024-04-21T08:56:50.491 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 0, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 2, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 0, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.000985430, 2024-04-21T08:56:50.492 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.000194677, 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.493 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:50.494 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 244, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 233, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 1595696, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 143424, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.042216279, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.012354910, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.033846326, 2024-04-21T08:56:50.495 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.005365287, 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 232, 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.011148732, 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000048054 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:50.496 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 1595696, 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 143424 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_messages": 6, 2024-04-21T08:56:50.497 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages": 180, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_bytes": 3264, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_bytes": 187760, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_total_time": 0.010644584, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_send_time": 0.006726210, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_recv_time": 0.001766797, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:50.498 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 178, 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.014886644, 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000083632 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "msgr_send_encrypted_bytes": 187760 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "cct": { 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "total_workers": 0, 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "unhealthy_workers": 0 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "mempool": { 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:50.500 INFO:teuthology.orchestra.run.smithi129.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:50.501 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:50.502 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:50.503 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_items": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_bytes": 148714, 2024-04-21T08:56:50.504 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_anon_items": 30, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_bytes": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_items": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:50.505 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_items": 0, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "pgmap_items": 0, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_co_items": 2, 2024-04-21T08:56:50.506 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_1_items": 0, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "unittest_2_items": 0 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "val": 0, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "max": 104857600, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "get_started": 0, 2024-04-21T08:56:50.507 INFO:teuthology.orchestra.run.smithi129.stdout: "get": 246, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "get_sum": 1561284, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "get_or_fail_success": 246, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "take": 0, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "take_sum": 0, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "put": 246, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "put_sum": 1561284, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "wait": { 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "avgcount": 0, 2024-04-21T08:56:50.508 INFO:teuthology.orchestra.run.smithi129.stdout: "sum": 0.000000000, 2024-04-21T08:56:50.509 INFO:teuthology.orchestra.run.smithi129.stdout: "avgtime": 0.000000000 2024-04-21T08:56:50.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:50.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-04-21T08:56:50.509 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-04-21T08:56:50.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:50 smithi129 ceph-mon[31272]: pgmap v360: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-21T08:56:50.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:50 smithi057 ceph-mon[27968]: pgmap v360: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-21T08:56:50.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:50 smithi136 ceph-mon[31791]: pgmap v360: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 69 KiB/s wr, 9 op/s 2024-04-21T08:56:51.284 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-21T08:56:51.284 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-21T08:56:51.285 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-21T08:56:51.285 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-04-21T08:56:51.285 DEBUG:tasks.check_counter:Getting stats from c 2024-04-21T08:56:51.285 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-21T08:56:51.535 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:56:52.803 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:52 smithi136 ceph-mon[31791]: pgmap v361: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2024-04-21T08:56:52.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:52 smithi129 ceph-mon[31272]: pgmap v361: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2024-04-21T08:56:52.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:52 smithi057 ceph-mon[27968]: pgmap v361: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2024-04-21T08:56:53.423 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-21T08:56:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:53.423 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 4, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 4, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 448, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.001355488, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.000042921, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.000508270, 2024-04-21T08:56:53.424 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3, 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000684417, 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000228139 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 241, 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 230, 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 1594880, 2024-04-21T08:56:53.426 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 141504, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.043127905, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.013641673, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.032322362, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.005429022, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 229, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.014459039, 2024-04-21T08:56:53.427 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000063139 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1594880, 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 141504 2024-04-21T08:56:53.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 2, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 176, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 186960, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.011060451, 2024-04-21T08:56:53.429 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.007275597, 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.001562049, 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 173, 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.022462198, 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000129839 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.430 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 186960 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-04-21T08:56:53.431 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 0, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:53.432 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:53.433 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:53.434 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-04-21T08:56:53.435 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 148995, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 31, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-04-21T08:56:53.436 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:53.437 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 2, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:53.438 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 243, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 1560894, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:53.439 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 243, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 243, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1560894, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:53.440 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:53.441 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-21T08:56:54.132 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-04-21T08:56:54.132 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-04-21T08:56:54.132 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-04-21T08:56:54.132 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2024-04-21T08:56:54.132 DEBUG:tasks.check_counter:Getting stats from f 2024-04-21T08:56:54.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-21T08:56:54.384 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:56:54.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:54 smithi129 ceph-mon[31272]: pgmap v362: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.3 KiB/s wr, 4 op/s 2024-04-21T08:56:54.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:54 smithi057 ceph-mon[27968]: pgmap v362: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.3 KiB/s wr, 4 op/s 2024-04-21T08:56:54.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:54 smithi136 ceph-mon[31791]: pgmap v362: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.3 KiB/s wr, 4 op/s 2024-04-21T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:55 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:55 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:55 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:55 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:55 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:55 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:55.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:55 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:55 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:55 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:55 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:55 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-21T08:56:55.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:55 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:55 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:55 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:55.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:55 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.183 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 242, 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 231, 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 1595152, 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 142144, 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 4, 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 0, 2024-04-21T08:56:56.184 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.045685883, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.013611307, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.034562496, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.005749928, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 230, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.009677137, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000042074 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.185 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1595152, 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 142144 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:56.186 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 6, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 179, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 3264, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 187584, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 5, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 3, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.010378385, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.006689189, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.001670898, 2024-04-21T08:56:56.187 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 177, 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.022315659, 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000126077 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 3264, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 187584 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 0, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 2, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 0, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 0, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 2, 2024-04-21T08:56:56.189 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 1, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.000302996, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.000000000, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.000235875, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 0 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.191 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 0, 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:56.192 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:56.193 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:56.194 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-04-21T08:56:56.195 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 148977, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 30, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-04-21T08:56:56.196 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:56.197 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 928, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 2, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:56.198 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 244, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 1561024, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 244, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-04-21T08:56:56.199 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 244, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1561024, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:56.200 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-21T08:56:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:56 smithi129 ceph-mon[31272]: pgmap v363: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-21T08:56:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:56 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:56 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:56 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:56.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:56 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:56.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:56 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:56 smithi057 ceph-mon[27968]: pgmap v363: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-21T08:56:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:56 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:56 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:56 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:56.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:56 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:56.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:56 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:56 smithi136 ceph-mon[31791]: pgmap v363: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-21T08:56:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:56 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:56 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:56 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-21T08:56:56.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:56 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-21T08:56:56.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:56 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' 2024-04-21T08:56:56.991 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-21T08:56:56.992 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-21T08:56:56.992 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-21T08:56:56.992 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-04-21T08:56:56.992 DEBUG:tasks.check_counter:Getting stats from i 2024-04-21T08:56:56.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-21T08:56:57.232 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:56:58.704 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:56:58 smithi136 ceph-mon[31791]: pgmap v364: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-21T08:56:58.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:56:58 smithi129 ceph-mon[31272]: pgmap v364: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-21T08:56:58.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:56:58 smithi057 ceph-mon[27968]: pgmap v364: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-21T08:56:58.968 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-21T08:56:58.968 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:56:58.968 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 1828, 2024-04-21T08:56:58.968 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 1828, 2024-04-21T08:56:58.968 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 193584619, 2024-04-21T08:56:58.968 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 580490, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 8, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 4, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.276965872, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.048273499, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.157493019, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.055709492, 2024-04-21T08:56:58.969 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 1827, 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.073896609, 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000040446 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.970 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 980, 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 968, 2024-04-21T08:56:58.971 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 198960804, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 372498, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 6, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 5, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.215647675, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.029688103, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.163582032, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.026154589, 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:58.972 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 967, 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.044720243, 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000046246 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.973 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1622400, 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 146016 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 608, 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 779, 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 146965041, 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 659209, 2024-04-21T08:56:58.974 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 8, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 6, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.118370228, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.022160310, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.074030267, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.016229888, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 776, 2024-04-21T08:56:58.975 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.040156903, 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000051748 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.976 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 467136 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 1, 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-MDSRank": { 2024-04-21T08:56:58.977 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 4923, 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 1.233052053, 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000250467 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: "finisher-PurgeQueue": { 2024-04-21T08:56:58.978 INFO:teuthology.orchestra.run.smithi136.stdout: "queue_len": 0, 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: "complete_latency": { 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.979 INFO:teuthology.orchestra.run.smithi136.stdout: "mds": { 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: "request": 0, 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: "reply": 0, 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: "reply_latency": { 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_reply": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "forward": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_complete": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_fetch_keys": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_commit": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_split": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_merge": 0, 2024-04-21T08:56:58.981 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes": 100906, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_top": 66828, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_bottom": 34078, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pin_tail": 0, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_pinned": 5547, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_expired": 0, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "inodes_with_caps": 0, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "caps": 0, 2024-04-21T08:56:58.982 INFO:teuthology.orchestra.run.smithi136.stdout: "subtrees": 2, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_hit": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_forward": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_discover": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_dir_fetch": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_remote_ino": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "traverse_lock": 0, 2024-04-21T08:56:58.983 INFO:teuthology.orchestra.run.smithi136.stdout: "load_cent": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "q": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "exported": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "exported_inodes": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "imported": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "imported_inodes": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_dir_fetch": 0, 2024-04-21T08:56:58.985 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_backtrace_fetch": 0, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "openino_peer_discover": 0, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rfiles": 84949, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rbytes": 1553714450, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "root_rsnaps": 6, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_fetch": 0, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_set_tag": 0, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_backtrace_repaired": 0, 2024-04-21T08:56:58.986 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_inotable_repaired": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_inodes": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dir_base_inodes": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_dirfrag_rstats": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "scrub_file_inodes": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_inode_file_caps": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_revoke": 0, 2024-04-21T08:56:58.987 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_grant": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_trunc": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_caps_dirty": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_cap_release": 0, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "process_request_cap_release": 0 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.988 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_cache": { 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_update_receipt": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_try_discover": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_send_discover": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "dir_handle_discover": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_delayed": 0, 2024-04-21T08:56:58.989 INFO:teuthology.orchestra.run.smithi136.stdout: "num_strays_enqueuing": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_created": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_enqueued": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_reintegrated": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "strays_migrated": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_processing": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_enqueued": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "num_recovering_prioritized": 0, 2024-04-21T08:56:58.990 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_started": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "recovery_completed": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_quiesce_path": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_quiesce_inode": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_enqueue_scrub": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_exportdir": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_flush": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragmentdir": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_fragstats": 0, 2024-04-21T08:56:58.991 INFO:teuthology.orchestra.run.smithi136.stdout: "ireq_inodestats": 0 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_log": { 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "evlrg": 0, 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "evadd": 0, 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "evex": 0, 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "evtrm": 0, 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "ev": 218094, 2024-04-21T08:56:58.992 INFO:teuthology.orchestra.run.smithi136.stdout: "evexg": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "evexd": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "segadd": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "segex": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "segtrm": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "seg": 2, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "segmjr": 41, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "segexg": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "segexd": 0, 2024-04-21T08:56:58.993 INFO:teuthology.orchestra.run.smithi136.stdout: "expos": 539308469, 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: "wrpos": 541140421, 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: "rdpos": 541128564, 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: "jlat": { 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "replayed": 218094 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_mem": { 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ino": 98673, 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ino+": 99841, 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "ino-": 1168, 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "dir": 6515, 2024-04-21T08:56:58.995 INFO:teuthology.orchestra.run.smithi136.stdout: "dir+": 6796, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "dir-": 281, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "dn": 100906, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "dn+": 100984, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "dn-": 78, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "cap": 0, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "cap+": 0, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "cap-": 0, 2024-04-21T08:56:58.996 INFO:teuthology.orchestra.run.smithi136.stdout: "rss": 422984, 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: "heap": 215416 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_server": { 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_client_request": 0, 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: "dispatch_server_request": 0, 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_request": 0, 2024-04-21T08:56:58.997 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_client_session": 0, 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "handle_peer_request": 0, 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "req_create_latency": { 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getattr_latency": { 2024-04-21T08:56:58.998 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getfilelock_latency": { 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:58.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "req_link_latency": { 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookup_latency": { 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookuphash_latency": { 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.001 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupino_latency": { 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupname_latency": { 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.002 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupparent_latency": { 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lookupsnap_latency": { 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.003 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.004 INFO:teuthology.orchestra.run.smithi136.stdout: "req_lssnap_latency": { 2024-04-21T08:56:59.004 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.004 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.005 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mkdir_latency": { 2024-04-21T08:56:59.005 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.005 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mknod_latency": { 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.006 INFO:teuthology.orchestra.run.smithi136.stdout: "req_mksnap_latency": { 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "req_open_latency": { 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "req_readdir_latency": { 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rename_latency": { 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.008 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "req_renamesnap_latency": { 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "req_snapdiff_latency": { 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.009 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmdir_latency": { 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.010 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmsnap_latency": { 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "req_rmxattr_latency": { 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setattr_latency": { 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setdirlayout_latency": { 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.012 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setfilelock_latency": { 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setlayout_latency": { 2024-04-21T08:56:59.013 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: "req_setxattr_latency": { 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "req_symlink_latency": { 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "req_unlink_latency": { 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.015 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_revoke_eviction": 0, 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: "cap_acquisition_throttle": 0, 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: "req_getvxattr_latency": { 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_sessions": { 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "session_count": 2, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "session_add": 6, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "session_remove": 4, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_open": 4, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "sessions_stale": 0, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "total_load": 0, 2024-04-21T08:56:59.017 INFO:teuthology.orchestra.run.smithi136.stdout: "average_load": 0, 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "avg_session_uptime": 3019, 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "mdthresh_evicted": 0 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:56:59.018 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:56:59.019 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:56:59.020 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:56:59.021 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 4261193, 2024-04-21T08:56:59.022 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 108775, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-04-21T08:56:59.023 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 9408, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 180, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 319592561, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 3583446, 2024-04-21T08:56:59.024 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "objecter": { 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "op_active": 0, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "op_laggy": 0, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send": 3140, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "op_send_bytes": 26, 2024-04-21T08:56:59.025 INFO:teuthology.orchestra.run.smithi136.stdout: "op_resend": 4, 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: "op_reply": 3140, 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: "op_latency": { 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3140, 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 87.083033308, 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.027733450 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.026 INFO:teuthology.orchestra.run.smithi136.stdout: "op_inflight": 0, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "oplen_avg": { 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3140, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 3141 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "op": 3140, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "op_r": 3140, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "op_w": 0, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "op_rmw": 0, 2024-04-21T08:56:59.027 INFO:teuthology.orchestra.run.smithi136.stdout: "op_pg": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_stat": 1882, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_create": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_read": 1255, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_write": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writefull": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_writesame": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_append": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_zero": 0, 2024-04-21T08:56:59.028 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_truncate": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_delete": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_mapext": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_sparse_read": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_clonerange": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_getxattr": 2, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_setxattr": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_cmpxattr": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_rmxattr": 0, 2024-04-21T08:56:59.029 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_resetxattrs": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_call": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_watch": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_notify": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_src_cmpxattr": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_pgls_filter": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "osdop_other": 0, 2024-04-21T08:56:59.030 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_active": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_send": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_resend": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "linger_ping": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_active": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_send": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "poolop_resend": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_active": 0, 2024-04-21T08:56:59.031 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_send": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "poolstat_resend": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_active": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_send": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "statfs_resend": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "command_active": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "command_send": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "command_resend": 0, 2024-04-21T08:56:59.032 INFO:teuthology.orchestra.run.smithi136.stdout: "map_epoch": 109, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "map_full": 0, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "map_inc": 20, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_sessions": 11, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_open": 11, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_session_close": 0, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_laggy": 0, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_wr": 0, 2024-04-21T08:56:59.033 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_rd": 2, 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_del": 0 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: "oft": { 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_objs": 0, 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_kv_pairs": 0, 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_updates": 0, 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: "omap_total_removes": 0 2024-04-21T08:56:59.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "osd-slow-ops": { 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "slow_ops_count": 0 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "purge_queue": { 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops": 0, 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_ops_high_water": 0, 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing": 0, 2024-04-21T08:56:59.035 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executing_high_water": 0, 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed_ops": 0, 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_executed": 0, 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "pq_item_in_journal": 0 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-04-21T08:56:59.036 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 3412, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 539186775, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 3412, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 3412, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 539186775, 2024-04-21T08:56:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_bytes": { 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:59.038 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 3140, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 537022366, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 3140, 2024-04-21T08:56:59.039 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 537022366, 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-objecter_ops": { 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:59.040 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 1024, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 3140, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 3140, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 3140, 2024-04-21T08:56:59.041 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 3140, 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle": { 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:59.042 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2024-04-21T08:56:59.043 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-write_buf_throttle-0x561e6acce6e0": { 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:56:59.044 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 3758096384, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 0, 2024-04-21T08:56:59.045 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 0, 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:56:59.046 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-21T08:56:59.773 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-21T08:56:59.773 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-21T08:56:59.774 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-21T08:56:59.774 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-21T08:56:59.774 DEBUG:tasks.check_counter:Getting stats from l 2024-04-21T08:56:59.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e shell --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-21T08:57:00.011 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/mon.c/config 2024-04-21T08:57:00.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:00 smithi129 ceph-mon[31272]: pgmap v365: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-21T08:57:00.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:00 smithi057 ceph-mon[27968]: pgmap v365: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-21T08:57:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:00 smithi136 ceph-mon[31791]: pgmap v365: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-21T08:57:01.769 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-21T08:57:01.770 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-0": { 2024-04-21T08:57:01.770 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 4, 2024-04-21T08:57:01.770 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 4, 2024-04-21T08:57:01.770 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 2944, 2024-04-21T08:57:01.770 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 448, 2024-04-21T08:57:01.770 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 4, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 0, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.001302859, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.000027774, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.000438112, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 3, 2024-04-21T08:57:01.771 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000646818, 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000215606 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-21T08:57:01.772 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 448 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-1": { 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 247, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 236, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 1596512, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 145344, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 2, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 1, 2024-04-21T08:57:01.773 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.047758623, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.013998635, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.037193526, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.005698921, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 235, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.016639708, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000070807 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 1596512, 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 145344 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "AsyncMessenger::Worker-2": { 2024-04-21T08:57:01.775 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_messages": 2, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages": 181, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_bytes": 320, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_bytes": 187840, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_created_connections": 5, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_active_connections": 3, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_total_time": 0.011241518, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_send_time": 0.007691913, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_recv_time": 0.001253569, 2024-04-21T08:57:01.776 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_messages_queue_lat": { 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 178, 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.023522757, 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000132150 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_handle_ack_lat": { 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:57:01.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "msgr_send_encrypted_bytes": 187840 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "cct": { 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "total_workers": 0, 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "unhealthy_workers": 0 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "mempool": { 2024-04-21T08:57:01.778 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_bytes": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bloom_filter_items": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_bytes": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_alloc_items": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_bytes": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_data_items": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_onode_items": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-21T08:57:01.779 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_meta_items": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_bytes": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_other_items": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_cache_buffer_items": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_bytes": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_extent_items": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_bytes": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_blob_items": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-21T08:57:01.780 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_shared_blob_items": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_inline_bl_items": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_bytes": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_fsck_items": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_bytes": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_txc_items": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_deferred_items": 0, 2024-04-21T08:57:01.781 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_bytes": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluestore_writing_items": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_bytes": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_items": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_bytes": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_reader_items": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_bytes": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "bluefs_file_writer_items": 0, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_bytes": 148714, 2024-04-21T08:57:01.782 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_anon_items": 30, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_bytes": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "buffer_meta_items": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_bytes": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_items": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_bytes": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_mapbl_items": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_bytes": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osd_pglog_items": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_bytes": 0, 2024-04-21T08:57:01.783 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_items": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_bytes": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "osdmap_mapping_items": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_bytes": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "pgmap_items": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_bytes": 928, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "mds_co_items": 2, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_bytes": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_1_items": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_bytes": 0, 2024-04-21T08:57:01.784 INFO:teuthology.orchestra.run.smithi136.stdout: "unittest_2_items": 0 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "val": 0, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "max": 104857600, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "get_started": 0, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "get": 249, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "get_sum": 1561674, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_fail": 0, 2024-04-21T08:57:01.785 INFO:teuthology.orchestra.run.smithi136.stdout: "get_or_fail_success": 249, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "take": 0, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "take_sum": 0, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "put": 249, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "put_sum": 1561674, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "wait": { 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "avgcount": 0, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "sum": 0.000000000, 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: "avgtime": 0.000000000 2024-04-21T08:57:01.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:57:01.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-04-21T08:57:01.787 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-21T08:57:02.555 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-21T08:57:02.555 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-21T08:57:02.555 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-21T08:57:02.555 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-04-21T08:57:02.556 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-21T08:57:02.567 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T08:57:02.567 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:57:02.567 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:57:02.588 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:57:02.588 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:57:02.646 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:57:02.801 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/550328920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f81031d0 msgr2=0x7fd0f81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/550328920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f81031d0 0x7fd0f81035b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009900 tx=0x7fd0ec02bcd0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/550328920 shutdown_connections 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/550328920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f810c5a0 0x7fd0f810e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/550328920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f8103af0 0x7fd0f810bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/550328920 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f81031d0 0x7fd0f81035b0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.798+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/550328920 >> 172.21.15.57:0/550328920 conn(0x7fd0f80771d0 msgr2=0x7fd0f80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/550328920 shutdown_connections 2024-04-21T08:57:02.802 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/550328920 wait complete. 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 Processor -- start 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 -- start start 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f81031d0 0x7fd0f8198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 0x7fd0f8198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f810c5a0 0x7fd0f819cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f8112220 con 0x7fd0f8103af0 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f81120a0 con 0x7fd0f81031d0 2024-04-21T08:57:02.803 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.799+0000 7fd0ffc3f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f81123a0 con 0x7fd0f810c5a0 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 0x7fd0f8198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 0x7fd0f8198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34754/0 (socket says 172.21.15.57:34754) 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 -- 172.21.15.57:0/856060052 learned_addr learned my addr 172.21.15.57:0/856060052 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd9db640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f81031d0 0x7fd0f8198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fe1dc640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f810c5a0 0x7fd0f819cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 -- 172.21.15.57:0/856060052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f810c5a0 msgr2=0x7fd0f819cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f810c5a0 0x7fd0f819cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 -- 172.21.15.57:0/856060052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f81031d0 msgr2=0x7fd0f8198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:02.804 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f81031d0 0x7fd0f8198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 -- 172.21.15.57:0/856060052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f819d6c0 con 0x7fd0f8103af0 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fe1dc640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f810c5a0 0x7fd0f819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0fd1da640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 0x7fd0f8198c60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400db80 tx=0x7fd0f400b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4017070 con 0x7fd0f8103af0 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0f4003b70 con 0x7fd0f8103af0 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f819d9b0 con 0x7fd0f8103af0 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40106c0 con 0x7fd0f8103af0 2024-04-21T08:57:02.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.800+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f81a5240 con 0x7fd0f8103af0 2024-04-21T08:57:02.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.801+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f8069230 con 0x7fd0f8103af0 2024-04-21T08:57:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.803+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd0f4004020 con 0x7fd0f8103af0 2024-04-21T08:57:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.803+0000 7fd0eaffd640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd0e40a3d20 0x7fd0e40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.803+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9459+0+0 (secure 0 0 0) 0x7fd0f40d0c50 con 0x7fd0f8103af0 2024-04-21T08:57:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.803+0000 7fd0fd9db640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd0e40a3d20 0x7fd0e40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:02.807 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.804+0000 7fd0fd9db640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd0e40a3d20 0x7fd0e40a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec005ab0 tx=0x7fd0ec005a20 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:02.809 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.808+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0f409cee0 con 0x7fd0f8103af0 2024-04-21T08:57:02.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:02 smithi129 ceph-mon[31272]: pgmap v366: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-21T08:57:02.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:02 smithi057 ceph-mon[27968]: pgmap v366: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-21T08:57:02.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:02 smithi136 ceph-mon[31791]: pgmap v366: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-21T08:57:02.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:02.998+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0f802d5c0 con 0x7fd0f8103af0 2024-04-21T08:57:03.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.000+0000 7fd0eaffd640 1 -- 172.21.15.57:0/856060052 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+25237 (secure 0 0 0) 0x7fd0f409c7e0 con 0x7fd0f8103af0 2024-04-21T08:57:03.001 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:57:03.001 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":109,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:56:35.466952+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:57:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd0e40a3d20 msgr2=0x7fd0e40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd0e40a3d20 0x7fd0e40a61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec005ab0 tx=0x7fd0ec005a20 comp rx=0 tx=0).stop 2024-04-21T08:57:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 msgr2=0x7fd0f8198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 0x7fd0f8198c60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400db80 tx=0x7fd0f400b630 comp rx=0 tx=0).stop 2024-04-21T08:57:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 shutdown_connections 2024-04-21T08:57:03.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd0e40a3d20 0x7fd0e40a61e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0f810c5a0 0x7fd0f819cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0f8103af0 0x7fd0f8198c60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 --2- 172.21.15.57:0/856060052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f81031d0 0x7fd0f8198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 >> 172.21.15.57:0/856060052 conn(0x7fd0f80771d0 msgr2=0x7fd0f80fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 shutdown_connections 2024-04-21T08:57:03.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.002+0000 7fd0ffc3f640 1 -- 172.21.15.57:0/856060052 wait complete. 2024-04-21T08:57:03.015 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-21T08:38:08.620595+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-21T08:42:54.854937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-21T08:42:55.981893+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-21T08:42:59.659722+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-21T08:57:03.015 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-21T08:57:03.123 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 -- 172.21.15.57:0/2671614355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0001031d0 msgr2=0x7fa0001035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 --2- 172.21.15.57:0/2671614355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0001031d0 0x7fa0001035b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009900 tx=0x7f9ffc02bd60 comp rx=0 tx=0).stop 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 -- 172.21.15.57:0/2671614355 shutdown_connections 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 --2- 172.21.15.57:0/2671614355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa00010c510 0x7fa00010e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 --2- 172.21.15.57:0/2671614355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa000103af0 0x7fa00010bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 --2- 172.21.15.57:0/2671614355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa0001031d0 0x7fa0001035b0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 -- 172.21.15.57:0/2671614355 >> 172.21.15.57:0/2671614355 conn(0x7fa000077f50 msgr2=0x7fa000078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.124 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 -- 172.21.15.57:0/2671614355 shutdown_connections 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 -- 172.21.15.57:0/2671614355 wait complete. 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 Processor -- start 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.120+0000 7fa008375640 1 -- start start 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa008375640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001031d0 0x7fa000100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa008375640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa000103af0 0x7fa000101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa008375640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 0x7fa0001018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa008375640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa0001121a0 con 0x7fa00010c510 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa008375640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa000112020 con 0x7fa000103af0 2024-04-21T08:57:03.125 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa008375640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa000112320 con 0x7fa0001031d0 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 0x7fa0001018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 0x7fa0001018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34764/0 (socket says 172.21.15.57:34764) 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 -- 172.21.15.57:0/2531866766 learned_addr learned my addr 172.21.15.57:0/2531866766 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006111640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001031d0 0x7fa000100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 -- 172.21.15.57:0/2531866766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001031d0 msgr2=0x7fa000100e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001031d0 0x7fa000100e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.126 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 -- 172.21.15.57:0/2531866766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa000103af0 msgr2=0x7fa000101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa005910640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa000103af0 0x7fa000101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa000103af0 0x7fa000101360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 -- 172.21.15.57:0/2531866766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0001a6c40 con 0x7fa00010c510 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7fa006912640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 0x7fa0001018a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800eaa0 tx=0x7f9ff800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff800cdb0 con 0x7fa00010c510 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.121+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ff8005020 con 0x7fa00010c510 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.122+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0001a6f30 con 0x7fa00010c510 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.122+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80116d0 con 0x7fa00010c510 2024-04-21T08:57:03.127 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.122+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0001a7450 con 0x7fa00010c510 2024-04-21T08:57:03.128 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.123+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa000068d70 con 0x7fa00010c510 2024-04-21T08:57:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.124+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ff80051d0 con 0x7fa00010c510 2024-04-21T08:57:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.124+0000 7f9ff77fe640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9fec0a3d20 0x7f9fec0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.125+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9459+0+0 (secure 0 0 0) 0x7f9ff80d1070 con 0x7fa00010c510 2024-04-21T08:57:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.125+0000 7fa006111640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9fec0a3d20 0x7f9fec0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.125+0000 7fa006111640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9fec0a3d20 0x7f9fec0a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc006010 tx=0x7f9ffc005e40 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.129 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.128+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ff8016030 con 0x7fa00010c510 2024-04-21T08:57:03.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.313+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa000107100 con 0x7fa00010c510 2024-04-21T08:57:03.314 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.313+0000 7f9ff77fe640 1 -- 172.21.15.57:0/2531866766 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7f9ff80bbe20 con 0x7fa00010c510 2024-04-21T08:57:03.315 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2024-04-21T08:57:03.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9fec0a3d20 msgr2=0x7f9fec0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9fec0a3d20 0x7f9fec0a61e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc006010 tx=0x7f9ffc005e40 comp rx=0 tx=0).stop 2024-04-21T08:57:03.316 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 msgr2=0x7fa0001018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 0x7fa0001018a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800eaa0 tx=0x7f9ff800ef60 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 shutdown_connections 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f9fec0a3d20 0x7f9fec0a61e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa00010c510 0x7fa0001018a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa000103af0 0x7fa000101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.317 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 --2- 172.21.15.57:0/2531866766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001031d0 0x7fa000100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 >> 172.21.15.57:0/2531866766 conn(0x7fa000077f50 msgr2=0x7fa0000fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 shutdown_connections 2024-04-21T08:57:03.318 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.315+0000 7fa008375640 1 -- 172.21.15.57:0/2531866766 wait complete. 2024-04-21T08:57:03.327 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-21T08:57:03.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 -- 172.21.15.57:0/4272711786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc1010a0 msgr2=0x7f7dcc101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/4272711786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc1010a0 0x7f7dcc101480 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009900 tx=0x7f7dbc02bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 -- 172.21.15.57:0/4272711786 shutdown_connections 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/4272711786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dcc106170 0x7f7dcc10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/4272711786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7dcc101a50 0x7f7dcc105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/4272711786 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc1010a0 0x7f7dcc101480 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 -- 172.21.15.57:0/4272711786 >> 172.21.15.57:0/4272711786 conn(0x7f7dcc077eb0 msgr2=0x7f7dcc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 -- 172.21.15.57:0/4272711786 shutdown_connections 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 -- 172.21.15.57:0/4272711786 wait complete. 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 Processor -- start 2024-04-21T08:57:03.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.431+0000 7f7dd19f4640 1 -- start start 2024-04-21T08:57:03.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dd19f4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7dcc1010a0 0x7f7dcc1994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dd19f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dcc106170 0x7f7dcc1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dd19f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 0x7f7dcc1a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dd19f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc1110b0 con 0x7f7dcc199f80 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dd19f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc110f30 con 0x7f7dcc106170 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dd19f4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc111230 con 0x7f7dcc1010a0 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 0x7f7dcc1a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 0x7f7dcc1a4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34792/0 (socket says 172.21.15.57:34792) 2024-04-21T08:57:03.436 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 -- 172.21.15.57:0/3617703735 learned_addr learned my addr 172.21.15.57:0/3617703735 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcaffd640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7dcc1010a0 0x7f7dcc1994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 -- 172.21.15.57:0/3617703735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7dcc1010a0 msgr2=0x7f7dcc1994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7dcc1010a0 0x7f7dcc1994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 -- 172.21.15.57:0/3617703735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dcc106170 msgr2=0x7f7dcc1999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dc27fc640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dcc106170 0x7f7dcc1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dcc106170 0x7f7dcc1999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 -- 172.21.15.57:0/3617703735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc1a4d30 con 0x7f7dcc199f80 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.432+0000 7f7dcb7fe640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 0x7f7dcc1a4530 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400eaa0 tx=0x7f7dc400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.433+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc400cdb0 con 0x7f7dcc199f80 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.433+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dc4005020 con 0x7f7dcc199f80 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.433+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dcc1a5020 con 0x7f7dcc199f80 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.433+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40116d0 con 0x7f7dcc199f80 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.433+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dcc1a54f0 con 0x7f7dcc199f80 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.434+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dcc0691c0 con 0x7f7dcc199f80 2024-04-21T08:57:03.438 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.435+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7dc40051d0 con 0x7f7dcc199f80 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.435+0000 7f7dd09f2640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7db80a3d20 0x7f7db80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.436+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9459+0+0 (secure 0 0 0) 0x7f7dc40d1170 con 0x7f7dcc199f80 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.436+0000 7f7dcaffd640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7db80a3d20 0x7f7db80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.439 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.436+0000 7f7dcaffd640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7db80a3d20 0x7f7db80a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc000c00 tx=0x7f7dbc004040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.440 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.439+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7dc4016030 con 0x7f7dcc199f80 2024-04-21T08:57:03.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/856060052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:03.589 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:03 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2531866766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:57:03.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.626+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7dcc101a50 con 0x7f7dcc199f80 2024-04-21T08:57:03.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.626+0000 7f7dd09f2640 1 -- 172.21.15.57:0/3617703735 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7f7dc409d460 con 0x7f7dcc199f80 2024-04-21T08:57:03.627 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:57:03.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7db80a3d20 msgr2=0x7f7db80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7db80a3d20 0x7f7db80a61e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc000c00 tx=0x7f7dbc004040 comp rx=0 tx=0).stop 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 msgr2=0x7f7dcc1a4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 0x7f7dcc1a4530 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400eaa0 tx=0x7f7dc400ef60 comp rx=0 tx=0).stop 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 shutdown_connections 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7db80a3d20 0x7f7db80a61e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dcc199f80 0x7f7dcc1a4530 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7dcc106170 0x7f7dcc1999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 --2- 172.21.15.57:0/3617703735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7dcc1010a0 0x7f7dcc1994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 >> 172.21.15.57:0/3617703735 conn(0x7f7dcc077eb0 msgr2=0x7f7dcc100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 shutdown_connections 2024-04-21T08:57:03.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.628+0000 7f7dd19f4640 1 -- 172.21.15.57:0/3617703735 wait complete. 2024-04-21T08:57:03.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-21T08:57:03.747 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 -- 172.21.15.57:0/286555613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0840102b40 msgr2=0x7f0840106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 --2- 172.21.15.57:0/286555613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0840102b40 0x7f0840106b80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0828009900 tx=0x7f082802bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 -- 172.21.15.57:0/286555613 shutdown_connections 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 --2- 172.21.15.57:0/286555613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840107260 0x7f084010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 --2- 172.21.15.57:0/286555613 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0840102b40 0x7f0840106b80 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 --2- 172.21.15.57:0/286555613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0840102190 0x7f0840102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 -- 172.21.15.57:0/286555613 >> 172.21.15.57:0/286555613 conn(0x7f08400fd990 msgr2=0x7f08400ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 -- 172.21.15.57:0/286555613 shutdown_connections 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.744+0000 7f0844f9f640 1 -- 172.21.15.57:0/286555613 wait complete. 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 Processor -- start 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 -- start start 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840102190 0x7f08401a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0840107260 0x7f08401a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 0x7f084019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f08401121a0 con 0x7f084019a720 2024-04-21T08:57:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0840112020 con 0x7f0840107260 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f0844f9f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0840112320 con 0x7f0840102190 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.745+0000 7f083ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 0x7f084019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 0x7f084019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34820/0 (socket says 172.21.15.57:34820) 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 -- 172.21.15.57:0/1522964796 learned_addr learned my addr 172.21.15.57:0/1522964796 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083e59c640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840102190 0x7f08401a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f0837fff640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0840107260 0x7f08401a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 -- 172.21.15.57:0/1522964796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840102190 msgr2=0x7f08401a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840102190 0x7f08401a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 -- 172.21.15.57:0/1522964796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0840107260 msgr2=0x7f08401a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0840107260 0x7f08401a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 -- 172.21.15.57:0/1522964796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084019b320 con 0x7f084019a720 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083e59c640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840102190 0x7f08401a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f083ed9d640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 0x7f084019aba0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f083800eaa0 tx=0x7f083800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083800cdb0 con 0x7f084019a720 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0838005020 con 0x7f084019a720 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08380116d0 con 0x7f084019a720 2024-04-21T08:57:03.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08401a7400 con 0x7f084019a720 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.746+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08401a7860 con 0x7f084019a720 2024-04-21T08:57:03.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.747+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0840103660 con 0x7f084019a720 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.748+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f08380051d0 con 0x7f084019a720 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.749+0000 7f08377fe640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f082c0a3da0 0x7f082c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.749+0000 7f083e59c640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f082c0a3da0 0x7f082c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.749+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9459+0+0 (secure 0 0 0) 0x7f08380d1170 con 0x7f084019a720 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.750+0000 7f083e59c640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f082c0a3da0 0x7f082c0a6260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f083000abe0 tx=0x7f083000a5c0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:03.753 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.753+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0838016030 con 0x7f084019a720 2024-04-21T08:57:03.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/856060052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:03.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:03 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2531866766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:57:03.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/856060052' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:03.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:03 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2531866766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-21T08:57:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.939+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f084019bc50 con 0x7f084019a720 2024-04-21T08:57:03.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.940+0000 7f08377fe640 1 -- 172.21.15.57:0/1522964796 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f083809d480 con 0x7f084019a720 2024-04-21T08:57:03.940 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 64 2024-04-21T08:57:03.942 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f082c0a3da0 msgr2=0x7f082c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f082c0a3da0 0x7f082c0a6260 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f083000abe0 tx=0x7f083000a5c0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 msgr2=0x7f084019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 0x7f084019aba0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f083800eaa0 tx=0x7f083800ef60 comp rx=0 tx=0).stop 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 shutdown_connections 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f082c0a3da0 0x7f082c0a6260 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f084019a720 0x7f084019aba0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0840107260 0x7f08401a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 --2- 172.21.15.57:0/1522964796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840102190 0x7f08401a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 >> 172.21.15.57:0/1522964796 conn(0x7f08400fd990 msgr2=0x7f08400ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.941+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 shutdown_connections 2024-04-21T08:57:03.944 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:03.942+0000 7f0844f9f640 1 -- 172.21.15.57:0/1522964796 wait complete. 2024-04-21T08:57:03.953 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-21T08:57:04.060 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 -- 172.21.15.57:0/1912653900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb870107270 msgr2=0x7fb87010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 --2- 172.21.15.57:0/1912653900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb870107270 0x7fb87010ee40 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb868009ca0 tx=0x7fb86802c0f0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 -- 172.21.15.57:0/1912653900 shutdown_connections 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 --2- 172.21.15.57:0/1912653900 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb870107270 0x7fb87010ee40 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 --2- 172.21.15.57:0/1912653900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb87006b1f0 0x7fb870106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 --2- 172.21.15.57:0/1912653900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb87006a840 0x7fb87006ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 -- 172.21.15.57:0/1912653900 >> 172.21.15.57:0/1912653900 conn(0x7fb870077eb0 msgr2=0x7fb8700782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.061 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.057+0000 7fb874a50640 1 -- 172.21.15.57:0/1912653900 shutdown_connections 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 -- 172.21.15.57:0/1912653900 wait complete. 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 Processor -- start 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 -- start start 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 0x7fb8701986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb870107270 0x7fb870198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb87019cf70 0x7fb87019d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.062 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb8701121d0 con 0x7fb87006a840 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb870112050 con 0x7fb870107270 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.058+0000 7fb874a50640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb870112350 con 0x7fb87019cf70 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 0x7fb8701986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 0x7fb8701986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34844/0 (socket says 172.21.15.57:34844) 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 -- 172.21.15.57:0/2082780069 learned_addr learned my addr 172.21.15.57:0/2082780069 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86dd9b640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb870107270 0x7fb870198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 -- 172.21.15.57:0/2082780069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb87019cf70 msgr2=0x7fb87019d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86ed9d640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb87019cf70 0x7fb87019d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.063 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb87019cf70 0x7fb87019d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 -- 172.21.15.57:0/2082780069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb870107270 msgr2=0x7fb870198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb870107270 0x7fb870198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 -- 172.21.15.57:0/2082780069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87019da60 con 0x7fb87006a840 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb86e59c640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 0x7fb8701986b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb86000d770 tx=0x7fb86000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860017070 con 0x7fb87006a840 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8600054d0 con 0x7fb87006a840 2024-04-21T08:57:04.064 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87019dd50 con 0x7fb87006a840 2024-04-21T08:57:04.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860004c30 con 0x7fb87006a840 2024-04-21T08:57:04.065 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.059+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8701a5540 con 0x7fb87006a840 2024-04-21T08:57:04.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.060+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8700691c0 con 0x7fb87006a840 2024-04-21T08:57:04.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.061+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb860004dd0 con 0x7fb87006a840 2024-04-21T08:57:04.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.062+0000 7fb85f7fe640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8580a3da0 0x7fb8580a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.062+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9459+0+0 (secure 0 0 0) 0x7fb8600d0b40 con 0x7fb87006a840 2024-04-21T08:57:04.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.062+0000 7fb86dd9b640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8580a3da0 0x7fb8580a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.066 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.063+0000 7fb86dd9b640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8580a3da0 0x7fb8580a6260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb864005e00 tx=0x7fb864005c50 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.067 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.066+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb86009ce50 con 0x7fb87006a840 2024-04-21T08:57:04.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.256+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb87006b1f0 con 0x7fb87006a840 2024-04-21T08:57:04.257 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.256+0000 7fb85f7fe640 1 -- 172.21.15.57:0/2082780069 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v109) ==== 103+0+11 (secure 0 0 0) 0x7fb86009c750 con 0x7fb87006a840 2024-04-21T08:57:04.257 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 32 2024-04-21T08:57:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8580a3da0 msgr2=0x7fb8580a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8580a3da0 0x7fb8580a6260 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb864005e00 tx=0x7fb864005c50 comp rx=0 tx=0).stop 2024-04-21T08:57:04.259 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 msgr2=0x7fb8701986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 0x7fb8701986b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb86000d770 tx=0x7fb86000dc30 comp rx=0 tx=0).stop 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 shutdown_connections 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fb8580a3da0 0x7fb8580a6260 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb87019cf70 0x7fb87019d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb870107270 0x7fb870198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 --2- 172.21.15.57:0/2082780069 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb87006a840 0x7fb8701986b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 >> 172.21.15.57:0/2082780069 conn(0x7fb870077eb0 msgr2=0x7fb8700fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 shutdown_connections 2024-04-21T08:57:04.260 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.258+0000 7fb874a50640 1 -- 172.21.15.57:0/2082780069 wait complete. 2024-04-21T08:57:04.270 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-21T08:57:04.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 -- 172.21.15.57:0/2981645280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 msgr2=0x7fea8010ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 --2- 172.21.15.57:0/2981645280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea8010ee40 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fea70009ca0 tx=0x7fea7002c0f0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 -- 172.21.15.57:0/2981645280 shutdown_connections 2024-04-21T08:57:04.375 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 --2- 172.21.15.57:0/2981645280 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea8010ee40 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 --2- 172.21.15.57:0/2981645280 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8006b1f0 0x7fea80106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 --2- 172.21.15.57:0/2981645280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8006a840 0x7fea8006ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 -- 172.21.15.57:0/2981645280 >> 172.21.15.57:0/2981645280 conn(0x7fea80077eb0 msgr2=0x7fea800782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 -- 172.21.15.57:0/2981645280 shutdown_connections 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 -- 172.21.15.57:0/2981645280 wait complete. 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.372+0000 7fea87ed2640 1 Processor -- start 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 -- start start 2024-04-21T08:57:04.376 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8006a840 0x7fea80198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea80198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8019ce00 0x7fea8019d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fea801121b0 con 0x7fea80107270 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fea80112030 con 0x7fea8019ce00 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fea80112330 con 0x7fea8006a840 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea80198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea80198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34866/0 (socket says 172.21.15.57:34866) 2024-04-21T08:57:04.377 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 -- 172.21.15.57:0/427100354 learned_addr learned my addr 172.21.15.57:0/427100354 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea85c6e640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8006a840 0x7fea80198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 -- 172.21.15.57:0/427100354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8006a840 msgr2=0x7fea80198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8006a840 0x7fea80198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 -- 172.21.15.57:0/427100354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8019ce00 msgr2=0x7fea8019d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8646f640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8019ce00 0x7fea8019d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8019ce00 0x7fea8019d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 -- 172.21.15.57:0/427100354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8019d8f0 con 0x7fea80107270 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8646f640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8019ce00 0x7fea8019d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:04.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea8546d640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea80198ad0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fea7400c970 tx=0x7fea7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74014070 con 0x7fea80107270 2024-04-21T08:57:04.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea74004440 con 0x7fea80107270 2024-04-21T08:57:04.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.373+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8019dbe0 con 0x7fea80107270 2024-04-21T08:57:04.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.374+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74003e50 con 0x7fea80107270 2024-04-21T08:57:04.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.374+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea801a5420 con 0x7fea80107270 2024-04-21T08:57:04.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.374+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea800691c0 con 0x7fea80107270 2024-04-21T08:57:04.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.376+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fea74017070 con 0x7fea80107270 2024-04-21T08:57:04.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.377+0000 7fea7effd640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fea680a3f40 0x7fea680a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:04.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.377+0000 7fea85c6e640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fea680a3f40 0x7fea680a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:04.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.377+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9459+0+0 (secure 0 0 0) 0x7fea740d0e90 con 0x7fea80107270 2024-04-21T08:57:04.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.377+0000 7fea85c6e640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fea680a3f40 0x7fea680a6400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fea78005e10 tx=0x7fea78005da0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:04.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.380+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea7401e030 con 0x7fea80107270 2024-04-21T08:57:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.580+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fea80199890 con 0x7fea80107270 2024-04-21T08:57:04.581 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.580+0000 7fea7effd640 1 -- 172.21.15.57:0/427100354 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v109) ==== 81+0+943 (secure 0 0 0) 0x7fea80199890 con 0x7fea80107270 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3327023287 2024-04-22T08:45:24.897575+0000 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/4242179214 2024-04-22T08:45:24.897575+0000 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/3002879345 2024-04-22T08:45:24.897575+0000 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3363889148 2024-04-22T08:35:44.517439+0000 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/316357528 2024-04-22T08:35:44.517439+0000 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/2928520021 2024-04-22T08:35:44.517439+0000 2024-04-21T08:57:04.582 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/3002879345 2024-04-22T08:45:24.897575+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/2928520021 2024-04-22T08:35:44.517439+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/1907214317 2024-04-22T08:45:24.897575+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2818769380 2024-04-22T08:35:17.446046+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/898712319 2024-04-22T08:35:17.446046+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/2910544903 2024-04-22T08:35:17.446046+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6800/3383203302 2024-04-22T08:35:17.446046+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/608627600 2024-04-22T08:45:24.897575+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:0/3448624546 2024-04-22T08:35:44.517439+0000 2024-04-21T08:57:04.583 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57:6801/3383203302 2024-04-22T08:35:17.446046+0000 2024-04-21T08:57:04.584 INFO:teuthology.orchestra.run.smithi057.stderr:listed 16 entries 2024-04-21T08:57:04.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fea680a3f40 msgr2=0x7fea680a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.584 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fea680a3f40 0x7fea680a6400 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fea78005e10 tx=0x7fea78005da0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 msgr2=0x7fea80198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea80198ad0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fea7400c970 tx=0x7fea7400ce30 comp rx=0 tx=0).stop 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 shutdown_connections 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fea680a3f40 0x7fea680a6400 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea8019ce00 0x7fea8019d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea80107270 0x7fea80198ad0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 --2- 172.21.15.57:0/427100354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8006a840 0x7fea80198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 >> 172.21.15.57:0/427100354 conn(0x7fea80077eb0 msgr2=0x7fea800fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:04.585 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 shutdown_connections 2024-04-21T08:57:04.586 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:04.582+0000 7fea87ed2640 1 -- 172.21.15.57:0/427100354 wait complete. 2024-04-21T08:57:04.594 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-21T08:57:04.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:04.594 DEBUG:teuthology.orchestra.run.smithi057:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-21T08:57:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:04 smithi129 ceph-mon[31272]: pgmap v367: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 3 op/s 2024-04-21T08:57:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:04 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3617703735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:04 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1522964796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:04 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/2082780069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:04 smithi057 ceph-mon[27968]: pgmap v367: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 3 op/s 2024-04-21T08:57:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:04 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3617703735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:04 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1522964796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:04 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/2082780069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:04 smithi136 ceph-mon[31791]: pgmap v367: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 3 op/s 2024-04-21T08:57:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:04 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3617703735' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:04 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1522964796' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-21T08:57:04.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:04 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/2082780069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-21T08:57:05.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:05 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/427100354' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:57:05.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:05 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/427100354' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:57:05.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:05 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/427100354' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-21T08:57:06.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:06 smithi129 ceph-mon[31272]: pgmap v368: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 61 KiB/s wr, 19 op/s 2024-04-21T08:57:06.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:06 smithi057 ceph-mon[27968]: pgmap v368: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 61 KiB/s wr, 19 op/s 2024-04-21T08:57:06.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:06 smithi136 ceph-mon[31791]: pgmap v368: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 61 KiB/s wr, 19 op/s 2024-04-21T08:57:08.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:08 smithi129 ceph-mon[31272]: pgmap v369: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 721 KiB/s wr, 20 op/s 2024-04-21T08:57:08.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:08 smithi057 ceph-mon[27968]: pgmap v369: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 721 KiB/s wr, 20 op/s 2024-04-21T08:57:08.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:08 smithi136 ceph-mon[31791]: pgmap v369: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 721 KiB/s wr, 20 op/s 2024-04-21T08:57:10.310 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T08:57:10.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:10.310 DEBUG:teuthology.orchestra.run.smithi057:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-21T08:57:10.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:10.330 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2024-04-21T08:57:10.388 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-21T08:57:10.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:10.389 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-21T08:57:10.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:10.456 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2024-04-21T08:57:10.554 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-21T08:57:10.555 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-04-21T08:57:10.555 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-21T08:57:10.572 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1278 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/aecd3bd6fb05bb72400d554c2a922ac5fcd22cb15e514f7f8d8bcd525282cd7f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/aecd3bd6fb05bb72400d554c2a922ac5fcd22cb15e514f7f8d8bcd525282cd7f/diff,workdir=/var/lib/containers/storage/overlay/aecd3bd6fb05bb72400d554c2a922ac5fcd22cb15e514f7f8d8bcd525282cd7f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b7d7f881fd0fc570a0089059c4f62a4ff64004b873e64a8bdfe2243468839007/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/b7d7f881fd0fc570a0089059c4f62a4ff64004b873e64a8bdfe2243468839007/diff,workdir=/var/lib/containers/storage/overlay/b7d7f881fd0fc570a0089059c4f62a4ff64004b873e64a8bdfe2243468839007/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/18fdd7414298896357144884b3765c4eb6a931c6d3b3aca3e032139439822a77/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/18fdd7414298896357144884b3765c4eb6a931c6d3b3aca3e032139439822a77/diff,workdir=/var/lib/containers/storage/overlay/18fdd7414298896357144884b3765c4eb6a931c6d3b3aca3e032139439822a77/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/cbe495251376818f18be36cbbff591b680de54cd701db6a16213142227c7778b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/cbe495251376818f18be36cbbff591b680de54cd701db6a16213142227c7778b/diff,workdir=/var/lib/containers/storage/overlay/cbe495251376818f18be36cbbff591b680de54cd701db6a16213142227c7778b/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5f7708329f3271139947a710ebaafb8c9669c1882e4ae9ffeffa205a51f1863a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/5f7708329f3271139947a710ebaafb8c9669c1882e4ae9ffeffa205a51f1863a/diff,workdir=/var/lib/containers/storage/overlay/5f7708329f3271139947a710ebaafb8c9669c1882e4ae9ffeffa205a51f1863a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7caf121605e9fa79fb0030d0812d3b77c54187451561e111abb4022822e74522/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/7caf121605e9fa79fb0030d0812d3b77c54187451561e111abb4022822e74522/diff,workdir=/var/lib/containers/storage/overlay/7caf121605e9fa79fb0030d0812d3b77c54187451561e111abb4022822e74522/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f7f8d40b10efcdbdcfb5beff5c8d73136584a7f59c55ecb1984cdc386f505e3f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/f7f8d40b10efcdbdcfb5beff5c8d73136584a7f59c55ecb1984cdc386f505e3f/diff,workdir=/var/lib/containers/storage/overlay/f7f8d40b10efcdbdcfb5beff5c8d73136584a7f59c55ecb1984cdc386f505e3f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a9013f0f98304d3f64666293554b7c968d893b1b2a05f961386e989eface7656/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/a9013f0f98304d3f64666293554b7c968d893b1b2a05f961386e989eface7656/diff,workdir=/var/lib/containers/storage/overlay/a9013f0f98304d3f64666293554b7c968d893b1b2a05f961386e989eface7656/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6d41099c4b748046550a78bb0c14c1e93c50d8a1e8b532b105c188ffa5e145ba/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/6d41099c4b748046550a78bb0c14c1e93c50d8a1e8b532b105c188ffa5e145ba/diff,workdir=/var/lib/containers/storage/overlay/6d41099c4b748046550a78bb0c14c1e93c50d8a1e8b532b105c188ffa5e145ba/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bde5902fe085e21d41c0534716f85621d7ff92fd408f42e95eb598f7eecd7b9b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/V24ELA3KLODYI6Y3WMZC2VAGVK:/var/lib/containers/storage/overlay/l/ORGTIBWC3JIY7URNV4WGGSTXX2,upperdir=/var/lib/containers/storage/overlay/bde5902fe085e21d41c0534716f85621d7ff92fd408f42e95eb598f7eecd7b9b/diff,workdir=/var/lib/containers/storage/overlay/bde5902fe085e21d41c0534716f85621d7ff92fd408f42e95eb598f7eecd7b9b/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-04-21T08:57:10.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:10.572 DEBUG:teuthology.orchestra.run.smithi057:> ip netns list 2024-04-21T08:57:10.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:10 smithi136 ceph-mon[31791]: pgmap v370: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.0 MiB/s wr, 20 op/s 2024-04-21T08:57:10.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-21T08:57:10.643 DEBUG:teuthology.orchestra.run.smithi057:> sudo ip link delete ceph-brx 2024-04-21T08:57:10.719 INFO:teuthology.orchestra.run.smithi057.stderr:Cannot find device "ceph-brx" 2024-04-21T08:57:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T08:57:10.722 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-21T08:57:10.734 INFO:tasks.cephadm:Teardown begin 2024-04-21T08:57:10.734 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-21T08:57:10.734 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:57:10.813 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:10 smithi057 ceph-mon[27968]: pgmap v370: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.0 MiB/s wr, 20 op/s 2024-04-21T08:57:10.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:10 smithi129 ceph-mon[31272]: pgmap v370: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.0 MiB/s wr, 20 op/s 2024-04-21T08:57:10.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.946+0000 7f7300f3d640 1 -- 172.21.15.57:0/2762550316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc110b30 msgr2=0x7f72fc109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:10.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.946+0000 7f7300f3d640 1 --2- 172.21.15.57:0/2762550316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc110b30 0x7f72fc109490 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009860 tx=0x7f72f002bbb0 comp rx=0 tx=0).stop 2024-04-21T08:57:10.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.949+0000 7f7300f3d640 1 -- 172.21.15.57:0/2762550316 shutdown_connections 2024-04-21T08:57:10.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.949+0000 7f7300f3d640 1 --2- 172.21.15.57:0/2762550316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72fc109b70 0x7f72fc11d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.949+0000 7f7300f3d640 1 --2- 172.21.15.57:0/2762550316 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc110b30 0x7f72fc109490 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.949+0000 7f7300f3d640 1 --2- 172.21.15.57:0/2762550316 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72fc1106c0 0x7f72fc108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:10.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.949+0000 7f7300f3d640 1 -- 172.21.15.57:0/2762550316 >> 172.21.15.57:0/2762550316 conn(0x7f72fc06e410 msgr2=0x7f72fc0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:10.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.953+0000 7f7300f3d640 1 -- 172.21.15.57:0/2762550316 shutdown_connections 2024-04-21T08:57:10.955 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.954+0000 7f7300f3d640 1 -- 172.21.15.57:0/2762550316 wait complete. 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 Processor -- start 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 -- start start 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72fc1106c0 0x7f72fc11aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72fc110b30 0x7f72fc114f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 0x7f72fc115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc111810 con 0x7f72fc115480 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc111690 con 0x7f72fc110b30 2024-04-21T08:57:10.956 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.955+0000 7f7300f3d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f72fc111990 con 0x7f72fc1106c0 2024-04-21T08:57:10.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.956+0000 7f72fad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 0x7f72fc115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:10.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.956+0000 7f72fad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 0x7f72fc115930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34884/0 (socket says 172.21.15.57:34884) 2024-04-21T08:57:10.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.956+0000 7f72fad9d640 1 -- 172.21.15.57:0/573573610 learned_addr learned my addr 172.21.15.57:0/573573610 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:10.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.957+0000 7f72fa59c640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72fc1106c0 0x7f72fc11aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:10.958 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.958+0000 7f72f9d9b640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72fc110b30 0x7f72fc114f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:10.959 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.960+0000 7f72fad9d640 1 -- 172.21.15.57:0/573573610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72fc1106c0 msgr2=0x7f72fc11aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:10.959 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.960+0000 7f72fad9d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72fc1106c0 0x7f72fc11aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:10.960 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.960+0000 7f72fad9d640 1 -- 172.21.15.57:0/573573610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72fc110b30 msgr2=0x7f72fc114f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:10.960 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.960+0000 7f72fad9d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72fc110b30 0x7f72fc114f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:10.960 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.960+0000 7f72fad9d640 1 -- 172.21.15.57:0/573573610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72fc07a280 con 0x7f72fc115480 2024-04-21T08:57:10.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.961+0000 7f72fad9d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 0x7f72fc115930 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00d750 tx=0x7f72ec00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:10.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.961+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec017070 con 0x7f72fc115480 2024-04-21T08:57:10.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.962+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72ec0054d0 con 0x7f72fc115480 2024-04-21T08:57:10.962 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.962+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec004ce0 con 0x7f72fc115480 2024-04-21T08:57:10.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.963+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72fc07a570 con 0x7f72fc115480 2024-04-21T08:57:10.963 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.963+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72fc07aa40 con 0x7f72fc115480 2024-04-21T08:57:10.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.965+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72fc073e40 con 0x7f72fc115480 2024-04-21T08:57:10.967 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.968+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f72ec009bd0 con 0x7f72fc115480 2024-04-21T08:57:10.968 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.969+0000 7f72eb7fe640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72e40a3f20 0x7f72e40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:10.969 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.969+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9563+0+0 (secure 0 0 0) 0x7f72ec0d0990 con 0x7f72fc115480 2024-04-21T08:57:10.976 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.976+0000 7f72fa59c640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72e40a3f20 0x7f72e40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:10.978 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.977+0000 7f72fa59c640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72e40a3f20 0x7f72e40a63e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f72f4005c40 tx=0x7f72f4005bd0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:10.979 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:10.979+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72ec020020 con 0x7f72fc115480 2024-04-21T08:57:11.417 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.417+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72fc068de0 con 0x7f72fc115480 2024-04-21T08:57:11.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.419+0000 7f72eb7fe640 1 -- 172.21.15.57:0/573573610 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7f72ec09cc40 con 0x7f72fc115480 2024-04-21T08:57:11.420 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:57:11.420 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:57:11.420 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:57:11.424 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.425+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72e40a3f20 msgr2=0x7f72e40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.425+0000 7f7300f3d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72e40a3f20 0x7f72e40a63e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f72f4005c40 tx=0x7f72f4005bd0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.425+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 msgr2=0x7f72fc115930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.425+0000 7f7300f3d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 0x7f72fc115930 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00d750 tx=0x7f72ec00dc10 comp rx=0 tx=0).stop 2024-04-21T08:57:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.426+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 shutdown_connections 2024-04-21T08:57:11.425 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.426+0000 7f7300f3d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f72e40a3f20 0x7f72e40a63e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.426+0000 7f7300f3d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f72fc115480 0x7f72fc115930 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.426+0000 7f7300f3d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72fc110b30 0x7f72fc114f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.426 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.426+0000 7f7300f3d640 1 --2- 172.21.15.57:0/573573610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72fc1106c0 0x7f72fc11aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.426+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 >> 172.21.15.57:0/573573610 conn(0x7f72fc06e410 msgr2=0x7f72fc11bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:11.427 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.427+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 shutdown_connections 2024-04-21T08:57:11.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.427+0000 7f7300f3d640 1 -- 172.21.15.57:0/573573610 wait complete. 2024-04-21T08:57:11.445 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-21T08:57:11.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 -- 172.21.15.57:0/309529066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c109b70 msgr2=0x7fac5c11d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:11.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 --2- 172.21.15.57:0/309529066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c109b70 0x7fac5c11d7a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009900 tx=0x7fac4c02bc60 comp rx=0 tx=0).stop 2024-04-21T08:57:11.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 -- 172.21.15.57:0/309529066 shutdown_connections 2024-04-21T08:57:11.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 --2- 172.21.15.57:0/309529066 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c109b70 0x7fac5c11d7a0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 --2- 172.21.15.57:0/309529066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c110b30 0x7fac5c109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 --2- 172.21.15.57:0/309529066 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1106c0 0x7fac5c108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.635+0000 7fac63711640 1 -- 172.21.15.57:0/309529066 >> 172.21.15.57:0/309529066 conn(0x7fac5c06e410 msgr2=0x7fac5c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:11.637 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.637+0000 7fac63711640 1 -- 172.21.15.57:0/309529066 shutdown_connections 2024-04-21T08:57:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.639+0000 7fac63711640 1 -- 172.21.15.57:0/309529066 wait complete. 2024-04-21T08:57:11.639 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.640+0000 7fac63711640 1 Processor -- start 2024-04-21T08:57:11.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 -- start start 2024-04-21T08:57:11.644 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c1106c0 0x7fac5c1a7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c110b30 0x7fac5c1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1abda0 0x7fac5c1ac250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c120ff0 con 0x7fac5c110b30 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c120e70 con 0x7fac5c1abda0 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.642+0000 7fac63711640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c121170 con 0x7fac5c1106c0 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac614ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c1106c0 0x7fac5c1a7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac614ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c1106c0 0x7fac5c1a7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55350/0 (socket says 172.21.15.57:55350) 2024-04-21T08:57:11.645 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac614ad640 1 -- 172.21.15.57:0/4150758772 learned_addr learned my addr 172.21.15.57:0/4150758772 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac61cae640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1abda0 0x7fac5c1ac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac61cae640 1 -- 172.21.15.57:0/4150758772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c1106c0 msgr2=0x7fac5c1a7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac61cae640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c1106c0 0x7fac5c1a7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac61cae640 1 -- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c110b30 msgr2=0x7fac5c1a7a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac61cae640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c110b30 0x7fac5c1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.643+0000 7fac61cae640 1 -- 172.21.15.57:0/4150758772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c1ac890 con 0x7fac5c1abda0 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.644+0000 7fac61cae640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1abda0 0x7fac5c1ac250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac4c036040 tx=0x7fac4c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:11.646 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.645+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c043020 con 0x7fac5c1abda0 2024-04-21T08:57:11.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.645+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac4c02ca40 con 0x7fac5c1abda0 2024-04-21T08:57:11.647 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.645+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c003a20 con 0x7fac5c1abda0 2024-04-21T08:57:11.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.647+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c1acb20 con 0x7fac5c1abda0 2024-04-21T08:57:11.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.647+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c1b4290 con 0x7fac5c1abda0 2024-04-21T08:57:11.649 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.649+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5c068de0 con 0x7fac5c1abda0 2024-04-21T08:57:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.651+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fac4c046020 con 0x7fac5c1abda0 2024-04-21T08:57:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.652+0000 7fac5a7fc640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fac440a3e20 0x7fac440a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.652+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9563+0+0 (secure 0 0 0) 0x7fac4c0f2d60 con 0x7fac5c1abda0 2024-04-21T08:57:11.653 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.653+0000 7fac614ad640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fac440a3e20 0x7fac440a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:11.654 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.654+0000 7fac614ad640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fac440a3e20 0x7fac440a62e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fac54009fb0 tx=0x7fac54009500 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:11.661 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:11.661+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac4c0bf250 con 0x7fac5c1abda0 2024-04-21T08:57:11.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:11 smithi129 ceph-mon[31272]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T08:57:11.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:11 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/573573610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:11.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:11 smithi057 ceph-mon[27968]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T08:57:11.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:11 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/573573610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:11 smithi136 ceph-mon[31791]: osdmap e110: 12 total, 12 up, 12 in 2024-04-21T08:57:11.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:11 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/573573610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:12.101 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.102+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac5c109b70 con 0x7fac5c1abda0 2024-04-21T08:57:12.103 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.103+0000 7fac5a7fc640 1 -- 172.21.15.57:0/4150758772 <== mon.2 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+11818 (secure 0 0 0) 0x7fac4c0beb50 con 0x7fac5c1abda0 2024-04-21T08:57:12.103 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:57:12.104 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6835/2211232297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":2211232297},{"type":"v1","addr":"172.21.15.57:6835","nonce":2211232297}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/2986531851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6836","nonce":2986531851},{"type":"v1","addr":"172.21.15.57:6837","nonce":2986531851}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6839/1393580593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6838","nonce":1393580593},{"type":"v1","addr":"172.21.15.57:6839","nonce":1393580593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6841/177564568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6840","nonce":177564568},{"type":"v1","addr":"172.21.15.57:6841","nonce":177564568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24416,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3055010748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3055010748},{"type":"v1","addr":"172.21.15.129:6833","nonce":3055010748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/3199760269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":3199760269},{"type":"v1","addr":"172.21.15.136:6833","nonce":3199760269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24440,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2158348740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":2158348740},{"type":"v1","addr":"172.21.15.129:6835","nonce":2158348740}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24464,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6837/1005091659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6836","nonce":1005091659},{"type":"v1","addr":"172.21.15.129:6837","nonce":1005091659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/1669978943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":1669978943},{"type":"v1","addr":"172.21.15.136:6835","nonce":1669978943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/1005985303","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":1005985303},{"type":"v1","addr":"172.21.15.136:6837","nonce":1005985303}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-21T08:42:56.988323+0000","modified":"2024-04-21T08:43:22.134494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24496":{"gid":24496,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/4293224778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4293224778},{"type":"v1","addr":"172.21.15.136:6839","nonce":4293224778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6839/123336774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6838","nonce":123336774},{"type":"v1","addr":"172.21.15.129:6839","nonce":123336774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24503,"qdb_cluster":[24503]},"id":1}]} 2024-04-21T08:57:12.104 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 21 2024-04-21T08:57:12.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fac440a3e20 msgr2=0x7fac440a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fac440a3e20 0x7fac440a62e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fac54009fb0 tx=0x7fac54009500 comp rx=0 tx=0).stop 2024-04-21T08:57:12.107 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1abda0 msgr2=0x7fac5c1ac250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1abda0 0x7fac5c1ac250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fac4c036040 tx=0x7fac4c0045e0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 shutdown_connections 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fac440a3e20 0x7fac440a62e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac5c1abda0 0x7fac5c1ac250 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5c110b30 0x7fac5c1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 --2- 172.21.15.57:0/4150758772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c1106c0 0x7fac5c1a7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 >> 172.21.15.57:0/4150758772 conn(0x7fac5c06e410 msgr2=0x7fac5c06fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:12.108 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 shutdown_connections 2024-04-21T08:57:12.109 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.107+0000 7fac63711640 1 -- 172.21.15.57:0/4150758772 wait complete. 2024-04-21T08:57:12.125 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-21T08:57:12.308 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.309+0000 7f4fc9ffb640 1 -- 172.21.15.57:0/1508121778 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4030630 con 0x7f4fcc105e30 2024-04-21T08:57:12.312 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.309+0000 7f4fd15ce640 1 -- 172.21.15.57:0/1508121778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 msgr2=0x7f4fcc106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.309+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/1508121778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc106210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4010910 tx=0x7f4fb402fb10 comp rx=0 tx=0).stop 2024-04-21T08:57:12.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.311+0000 7f4fd15ce640 1 -- 172.21.15.57:0/1508121778 shutdown_connections 2024-04-21T08:57:12.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.311+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/1508121778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fcc101b60 0x7f4fcc10dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.311+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/1508121778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fcc1011a0 0x7f4fcc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.311+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/1508121778 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc106210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.313 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.311+0000 7f4fd15ce640 1 -- 172.21.15.57:0/1508121778 >> 172.21.15.57:0/1508121778 conn(0x7f4fcc077e30 msgr2=0x7f4fcc078240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:12.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.318+0000 7f4fd15ce640 1 -- 172.21.15.57:0/1508121778 shutdown_connections 2024-04-21T08:57:12.319 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.319+0000 7f4fd15ce640 1 -- 172.21.15.57:0/1508121778 wait complete. 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 Processor -- start 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 -- start start 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fcc1011a0 0x7f4fcc1a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fcc101b60 0x7f4fcc1a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc1abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc121090 con 0x7f4fcc1011a0 2024-04-21T08:57:12.323 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc120f10 con 0x7f4fcc101b60 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.320+0000 7f4fd15ce640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4fcc121210 con 0x7f4fcc105e30 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.321+0000 7f4fcb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc1abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.321+0000 7f4fcb7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc1abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:55366/0 (socket says 172.21.15.57:55366) 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.321+0000 7f4fcb7fe640 1 -- 172.21.15.57:0/3143541819 learned_addr learned my addr 172.21.15.57:0/3143541819 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.322+0000 7f4fcaffd640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fcc1011a0 0x7f4fcc1a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.322+0000 7f4fcb7fe640 1 -- 172.21.15.57:0/3143541819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fcc101b60 msgr2=0x7f4fcc1a7b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.322+0000 7f4fcb7fe640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fcc101b60 0x7f4fcc1a7b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.322+0000 7f4fcb7fe640 1 -- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fcc1011a0 msgr2=0x7f4fcc1a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.324 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.322+0000 7f4fcb7fe640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fcc1011a0 0x7f4fcc1a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.322+0000 7f4fcb7fe640 1 -- 172.21.15.57:0/3143541819 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fcc1ac590 con 0x7f4fcc105e30 2024-04-21T08:57:12.325 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.323+0000 7f4fcb7fe640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc1abe60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400eda0 tx=0x7f4fc400c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:12.332 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.327+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4003b70 con 0x7f4fcc105e30 2024-04-21T08:57:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.327+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fc4003d10 con 0x7f4fcc105e30 2024-04-21T08:57:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.327+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4017900 con 0x7f4fcc105e30 2024-04-21T08:57:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.328+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fcc1ac880 con 0x7f4fcc105e30 2024-04-21T08:57:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.328+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fcc1b4110 con 0x7f4fcc105e30 2024-04-21T08:57:12.333 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.330+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fcc102570 con 0x7f4fcc105e30 2024-04-21T08:57:12.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.335+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4fc4017aa0 con 0x7f4fcc105e30 2024-04-21T08:57:12.336 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.336+0000 7f4fc3fff640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4fb80a3ed0 0x7f4fb80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.337 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.336+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9563+0+0 (secure 0 0 0) 0x7f4fc40d0f30 con 0x7f4fcc105e30 2024-04-21T08:57:12.340 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.341+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fc409d2f0 con 0x7f4fcc105e30 2024-04-21T08:57:12.344 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.343+0000 7f4fcaffd640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4fb80a3ed0 0x7f4fb80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:12.353 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.353+0000 7f4fcaffd640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4fb80a3ed0 0x7f4fb80a6390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4039fd0 tx=0x7f4fb400d040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:12.717 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.717+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fcc068de0 con 0x7f4fcc105e30 2024-04-21T08:57:12.720 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.720+0000 7f4fc3fff640 1 -- 172.21.15.57:0/3143541819 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25395 (secure 0 0 0) 0x7f4fcc068de0 con 0x7f4fcc105e30 2024-04-21T08:57:12.721 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-04-21T08:57:12.721 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":110,"fsid":"df0fb3ae-ffb9-11ee-bc93-c7b262605968","created":"2024-04-21T08:34:49.550181+0000","modified":"2024-04-21T08:57:10.561434+0000","last_up_change":"2024-04-21T08:42:02.836978+0000","last_in_change":"2024-04-21T08:41:41.833371+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-21T08:38:08.620595+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-21T08:42:54.854937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-21T08:42:55.981893+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-21T08:42:59.659722+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"773b7403-49ea-4952-8d1e-d30974987af6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6803","nonce":1351441903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6805","nonce":1351441903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6809","nonce":1351441903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":1351441903},{"type":"v1","addr":"172.21.15.57:6807","nonce":1351441903}]},"public_addr":"172.21.15.57:6803/1351441903","cluster_addr":"172.21.15.57:6805/1351441903","heartbeat_back_addr":"172.21.15.57:6809/1351441903","heartbeat_front_addr":"172.21.15.57:6807/1351441903","state":["exists","up"]},{"osd":1,"uuid":"ee1d7ace-7312-4242-aa99-11fcad20dd62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6801","nonce":3548590037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6803","nonce":3548590037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6807","nonce":3548590037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3548590037},{"type":"v1","addr":"172.21.15.129:6805","nonce":3548590037}]},"public_addr":"172.21.15.129:6801/3548590037","cluster_addr":"172.21.15.129:6803/3548590037","heartbeat_back_addr":"172.21.15.129:6807/3548590037","heartbeat_front_addr":"172.21.15.129:6805/3548590037","state":["exists","up"]},{"osd":2,"uuid":"1e30fb84-6e42-4857-9e01-fecda24f1074","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6801","nonce":1481395579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6803","nonce":1481395579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6807","nonce":1481395579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1481395579},{"type":"v1","addr":"172.21.15.136:6805","nonce":1481395579}]},"public_addr":"172.21.15.136:6801/1481395579","cluster_addr":"172.21.15.136:6803/1481395579","heartbeat_back_addr":"172.21.15.136:6807/1481395579","heartbeat_front_addr":"172.21.15.136:6805/1481395579","state":["exists","up"]},{"osd":3,"uuid":"bcf84be5-1210-4e75-86ed-4aac7a0d4dd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6811","nonce":3492557165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6813","nonce":3492557165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6817","nonce":3492557165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3492557165},{"type":"v1","addr":"172.21.15.57:6815","nonce":3492557165}]},"public_addr":"172.21.15.57:6811/3492557165","cluster_addr":"172.21.15.57:6813/3492557165","heartbeat_back_addr":"172.21.15.57:6817/3492557165","heartbeat_front_addr":"172.21.15.57:6815/3492557165","state":["exists","up"]},{"osd":4,"uuid":"d2797f86-8e4f-4d66-ba17-dcd90bb6869e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6809","nonce":757493638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6811","nonce":757493638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6815","nonce":757493638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":757493638},{"type":"v1","addr":"172.21.15.129:6813","nonce":757493638}]},"public_addr":"172.21.15.129:6809/757493638","cluster_addr":"172.21.15.129:6811/757493638","heartbeat_back_addr":"172.21.15.129:6815/757493638","heartbeat_front_addr":"172.21.15.129:6813/757493638","state":["exists","up"]},{"osd":5,"uuid":"f4b316af-e19a-41f1-b552-3e6aa0310bb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6809","nonce":564097587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6811","nonce":564097587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6815","nonce":564097587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":564097587},{"type":"v1","addr":"172.21.15.136:6813","nonce":564097587}]},"public_addr":"172.21.15.136:6809/564097587","cluster_addr":"172.21.15.136:6811/564097587","heartbeat_back_addr":"172.21.15.136:6815/564097587","heartbeat_front_addr":"172.21.15.136:6813/564097587","state":["exists","up"]},{"osd":6,"uuid":"09e7ec78-6803-43af-9a9d-234a5c7d4f4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6819","nonce":961629147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6821","nonce":961629147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6825","nonce":961629147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":961629147},{"type":"v1","addr":"172.21.15.57:6823","nonce":961629147}]},"public_addr":"172.21.15.57:6819/961629147","cluster_addr":"172.21.15.57:6821/961629147","heartbeat_back_addr":"172.21.15.57:6825/961629147","heartbeat_front_addr":"172.21.15.57:6823/961629147","state":["exists","up"]},{"osd":7,"uuid":"6d8a8522-645a-4a69-a6c2-d38b04198b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6817","nonce":1910048774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6819","nonce":1910048774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6823","nonce":1910048774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1910048774},{"type":"v1","addr":"172.21.15.129:6821","nonce":1910048774}]},"public_addr":"172.21.15.129:6817/1910048774","cluster_addr":"172.21.15.129:6819/1910048774","heartbeat_back_addr":"172.21.15.129:6823/1910048774","heartbeat_front_addr":"172.21.15.129:6821/1910048774","state":["exists","up"]},{"osd":8,"uuid":"5a0b1a0a-5937-4691-ac89-25c328fabf77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6817","nonce":94445430}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6819","nonce":94445430}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6823","nonce":94445430}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":94445430},{"type":"v1","addr":"172.21.15.136:6821","nonce":94445430}]},"public_addr":"172.21.15.136:6817/94445430","cluster_addr":"172.21.15.136:6819/94445430","heartbeat_back_addr":"172.21.15.136:6823/94445430","heartbeat_front_addr":"172.21.15.136:6821/94445430","state":["exists","up"]},{"osd":9,"uuid":"ed5fefc0-93b7-4cb0-9b64-ccd879d2f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6827","nonce":4116200547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6829","nonce":4116200547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6833","nonce":4116200547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":4116200547},{"type":"v1","addr":"172.21.15.57:6831","nonce":4116200547}]},"public_addr":"172.21.15.57:6827/4116200547","cluster_addr":"172.21.15.57:6829/4116200547","heartbeat_back_addr":"172.21.15.57:6833/4116200547","heartbeat_front_addr":"172.21.15.57:6831/4116200547","state":["exists","up"]},{"osd":10,"uuid":"4319594b-8e0c-4120-a5d9-40c9e1536cac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6825","nonce":1559005411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6827","nonce":1559005411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6831","nonce":1559005411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1559005411},{"type":"v1","addr":"172.21.15.129:6829","nonce":1559005411}]},"public_addr":"172.21.15.129:6825/1559005411","cluster_addr":"172.21.15.129:6827/1559005411","heartbeat_back_addr":"172.21.15.129:6831/1559005411","heartbeat_front_addr":"172.21.15.129:6829/1559005411","state":["exists","up"]},{"osd":11,"uuid":"b88b4078-cb27-4bfd-a17a-bf0a05dcc291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6825","nonce":4048591529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6827","nonce":4048591529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6831","nonce":4048591529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":4048591529},{"type":"v1","addr":"172.21.15.136:6829","nonce":4048591529}]},"public_addr":"172.21.15.136:6825/4048591529","cluster_addr":"172.21.15.136:6827/4048591529","heartbeat_back_addr":"172.21.15.136:6831/4048591529","heartbeat_front_addr":"172.21.15.136:6829/4048591529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:18.884168+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:37:43.314493+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:07.033447+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:32.010774+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:38:57.202079+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:22.510182+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:39:48.640406+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:14.925714+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:40:40.612950+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:07.495914+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:41:34.200482+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-21T08:42:00.436797+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8},{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:0/3327023287":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/4242179214":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3363889148":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/316357528":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3002879345":"2024-04-22T08:45:24.897575+0000","172.21.15.57:6800/2928520021":"2024-04-22T08:35:44.517439+0000","172.21.15.57:0/1907214317":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/2818769380":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/898712319":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/2910544903":"2024-04-22T08:35:17.446046+0000","172.21.15.57:6800/3383203302":"2024-04-22T08:35:17.446046+0000","172.21.15.57:0/608627600":"2024-04-22T08:45:24.897575+0000","172.21.15.57:0/3448624546":"2024-04-22T08:35:44.517439+0000","172.21.15.57:6801/3383203302":"2024-04-22T08:35:17.446046+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-21T08:57:12.724 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.725+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4fb80a3ed0 msgr2=0x7f4fb80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.725+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4fb80a3ed0 0x7f4fb80a6390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4039fd0 tx=0x7f4fb400d040 comp rx=0 tx=0).stop 2024-04-21T08:57:12.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.725+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 msgr2=0x7f4fcc1abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.725 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.725+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc1abe60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4fc400eda0 tx=0x7f4fc400c6e0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.726+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 shutdown_connections 2024-04-21T08:57:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.726+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4fb80a3ed0 0x7f4fb80a6390 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.726+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fcc105e30 0x7f4fcc1abe60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.726+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fcc101b60 0x7f4fcc1a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.726 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.726+0000 7f4fd15ce640 1 --2- 172.21.15.57:0/3143541819 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4fcc1011a0 0x7f4fcc1a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.726+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 >> 172.21.15.57:0/3143541819 conn(0x7f4fcc077e30 msgr2=0x7f4fcc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:12.727 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.727+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 shutdown_connections 2024-04-21T08:57:12.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.731+0000 7f4fd15ce640 1 -- 172.21.15.57:0/3143541819 wait complete. 2024-04-21T08:57:12.750 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-21T08:57:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:12 smithi057 ceph-mon[27968]: pgmap v372: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 24 op/s 2024-04-21T08:57:12.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:12 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/4150758772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:12.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:12 smithi136 ceph-mon[31791]: pgmap v372: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 24 op/s 2024-04-21T08:57:12.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:12 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/4150758772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:12.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.938+0000 7f4a688d2640 1 -- 172.21.15.57:0/306771070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a64106160 msgr2=0x7f4a6410dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.938 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.938+0000 7f4a688d2640 1 --2- 172.21.15.57:0/306771070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a64106160 0x7f4a6410dd30 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009ca0 tx=0x7f4a5c030200 comp rx=0 tx=0).stop 2024-04-21T08:57:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.940+0000 7f4a688d2640 1 -- 172.21.15.57:0/306771070 shutdown_connections 2024-04-21T08:57:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.940+0000 7f4a688d2640 1 --2- 172.21.15.57:0/306771070 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a64106160 0x7f4a6410dd30 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.940+0000 7f4a688d2640 1 --2- 172.21.15.57:0/306771070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64101a40 0x7f4a64105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.940+0000 7f4a688d2640 1 --2- 172.21.15.57:0/306771070 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 0x7f4a64101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.940 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.940+0000 7f4a688d2640 1 -- 172.21.15.57:0/306771070 >> 172.21.15.57:0/306771070 conn(0x7f4a64077f50 msgr2=0x7f4a64078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:12.941 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.942+0000 7f4a688d2640 1 -- 172.21.15.57:0/306771070 shutdown_connections 2024-04-21T08:57:12.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.942+0000 7f4a688d2640 1 -- 172.21.15.57:0/306771070 wait complete. 2024-04-21T08:57:12.943 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.943+0000 7f4a688d2640 1 Processor -- start 2024-04-21T08:57:12.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.946+0000 7f4a688d2640 1 -- start start 2024-04-21T08:57:12.946 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.946+0000 7f4a688d2640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 0x7f4a640737f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.947+0000 7f4a688d2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 0x7f4a6406c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.947+0000 7f4a688d2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a6406ce00 0x7f4a6406d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.947+0000 7f4a688d2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64118910 con 0x7f4a6406ce00 2024-04-21T08:57:12.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.947+0000 7f4a688d2640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64118790 con 0x7f4a64101090 2024-04-21T08:57:12.947 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.947+0000 7f4a688d2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a64118a90 con 0x7f4a64106160 2024-04-21T08:57:12.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.948+0000 7f4a61d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 0x7f4a6406c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:12.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.948+0000 7f4a61d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 0x7f4a6406c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.57:37922/0 (socket says 172.21.15.57:37922) 2024-04-21T08:57:12.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.948+0000 7f4a61d9b640 1 -- 172.21.15.57:0/1880780242 learned_addr learned my addr 172.21.15.57:0/1880780242 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:12.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.948+0000 7f4a6259c640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 0x7f4a640737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a61d9b640 1 -- 172.21.15.57:0/1880780242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 msgr2=0x7f4a640737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a61d9b640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 0x7f4a640737f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a61d9b640 1 -- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a6406ce00 msgr2=0x7f4a6406d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:12.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a61d9b640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a6406ce00 0x7f4a6406d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a61d9b640 1 -- 172.21.15.57:0/1880780242 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a64071340 con 0x7f4a64106160 2024-04-21T08:57:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a6259c640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 0x7f4a640737f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-21T08:57:12.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.949+0000 7f4a61d9b640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 0x7f4a6406c8c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400b450 tx=0x7f4a5400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:12.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.954+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54015020 con 0x7f4a64106160 2024-04-21T08:57:12.953 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.954+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a54011d40 con 0x7f4a64106160 2024-04-21T08:57:12.954 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.954+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5400f340 con 0x7f4a64106160 2024-04-21T08:57:12.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.954+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a64071630 con 0x7f4a64106160 2024-04-21T08:57:12.957 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.955+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a64071b30 con 0x7f4a64106160 2024-04-21T08:57:12.960 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.960+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4a54018020 con 0x7f4a64106160 2024-04-21T08:57:12.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.961+0000 7f4a537fe640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a4c0a3f30 0x7f4a4c0a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:12.961 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.962+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9563+0+0 (secure 0 0 0) 0x7f4a540d0510 con 0x7f4a64106160 2024-04-21T08:57:12.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.964+0000 7f4a6259c640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a4c0a3f30 0x7f4a4c0a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:12.965 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.965+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a64102560 con 0x7f4a64106160 2024-04-21T08:57:12.974 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.974+0000 7f4a6259c640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a4c0a3f30 0x7f4a4c0a63f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a58007900 tx=0x7f4a58008040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:12.982 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:12.980+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a5401f020 con 0x7f4a64106160 2024-04-21T08:57:13.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:12 smithi129 ceph-mon[31272]: pgmap v372: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 24 op/s 2024-04-21T08:57:13.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:12 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/4150758772' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-21T08:57:13.428 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.428+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4a6406dda0 con 0x7f4a64106160 2024-04-21T08:57:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.442+0000 7f4a537fe640 1 -- 172.21.15.57:0/1880780242 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) ==== 161+0+0 (secure 0 0 0) 0x7f4a5400d040 con 0x7f4a64106160 2024-04-21T08:57:13.442 INFO:teuthology.orchestra.run.smithi057.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-21T08:57:13.445 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.445+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a4c0a3f30 msgr2=0x7f4a4c0a63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.445+0000 7f4a688d2640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a4c0a3f30 0x7f4a4c0a63f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a58007900 tx=0x7f4a58008040 comp rx=0 tx=0).stop 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 msgr2=0x7f4a6406c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 0x7f4a6406c8c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400b450 tx=0x7f4a5400b910 comp rx=0 tx=0).stop 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 shutdown_connections 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f4a4c0a3f30 0x7f4a4c0a63f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a6406ce00 0x7f4a6406d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a64106160 0x7f4a6406c8c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 --2- 172.21.15.57:0/1880780242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a64101090 0x7f4a640737f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 >> 172.21.15.57:0/1880780242 conn(0x7f4a64077f50 msgr2=0x7f4a640ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:13.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 shutdown_connections 2024-04-21T08:57:13.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.446+0000 7f4a688d2640 1 -- 172.21.15.57:0/1880780242 wait complete. 2024-04-21T08:57:13.464 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-21T08:57:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.656+0000 7fda0ecd3640 1 -- 172.21.15.57:0/3578385624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 msgr2=0x7fda081093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.656+0000 7fda077fe640 1 -- 172.21.15.57:0/3578385624 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8005020 con 0x7fda08108f50 2024-04-21T08:57:13.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.656+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/3578385624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda081093d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800af30 tx=0x7fd9f802fe60 comp rx=0 tx=0).stop 2024-04-21T08:57:13.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.664+0000 7fda0ecd3640 1 -- 172.21.15.57:0/3578385624 shutdown_connections 2024-04-21T08:57:13.663 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.664+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/3578385624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 0x7fda081157f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.664 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.664+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/3578385624 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda081093d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.664 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.664+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/3578385624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda0810e930 0x7fda0810ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.664 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.664+0000 7fda0ecd3640 1 -- 172.21.15.57:0/3578385624 >> 172.21.15.57:0/3578385624 conn(0x7fda0806d4d0 msgr2=0x7fda0806d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:13.665 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.665+0000 7fda0ecd3640 1 -- 172.21.15.57:0/3578385624 shutdown_connections 2024-04-21T08:57:13.665 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.666+0000 7fda0ecd3640 1 -- 172.21.15.57:0/3578385624 wait complete. 2024-04-21T08:57:13.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.666+0000 7fda0ecd3640 1 Processor -- start 2024-04-21T08:57:13.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.666+0000 7fda0ecd3640 1 -- start start 2024-04-21T08:57:13.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ecd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda0819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:13.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ecd3640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 0x7fda0819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:13.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ecd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda0810e930 0x7fda081a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:13.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ecd3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fda081188b0 con 0x7fda0810e930 2024-04-21T08:57:13.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ecd3640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda08118730 con 0x7fda08109910 2024-04-21T08:57:13.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ecd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fda08118a30 con 0x7fda08108f50 2024-04-21T08:57:13.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda07fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 0x7fda0819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:13.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda07fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 0x7fda0819f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:34050/0 (socket says 172.21.15.57:34050) 2024-04-21T08:57:13.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda07fff640 1 -- 172.21.15.57:0/201388995 learned_addr learned my addr 172.21.15.57:0/201388995 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:13.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0ca6f640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda0819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:13.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.667+0000 7fda0d270640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda0810e930 0x7fda081a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:13.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.668+0000 7fda0ca6f640 1 -- 172.21.15.57:0/201388995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 msgr2=0x7fda0819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:13.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.668+0000 7fda0ca6f640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 0x7fda0819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.668+0000 7fda0ca6f640 1 -- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda0810e930 msgr2=0x7fda081a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:13.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.668+0000 7fda0ca6f640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda0810e930 0x7fda081a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:13.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.669+0000 7fda0ca6f640 1 -- 172.21.15.57:0/201388995 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda081a3fc0 con 0x7fda08108f50 2024-04-21T08:57:13.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.669+0000 7fda0ca6f640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda0819efd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009fd0 tx=0x7fd9fc00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:13.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.672+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc015020 con 0x7fda08108f50 2024-04-21T08:57:13.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.672+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9fc011ce0 con 0x7fda08108f50 2024-04-21T08:57:13.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.672+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc010e40 con 0x7fda08108f50 2024-04-21T08:57:13.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.672+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda081a42b0 con 0x7fda08108f50 2024-04-21T08:57:13.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.672+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda081abb40 con 0x7fda08108f50 2024-04-21T08:57:13.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.676+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda08068d70 con 0x7fda08108f50 2024-04-21T08:57:13.682 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.682+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd9fc018020 con 0x7fda08108f50 2024-04-21T08:57:13.683 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.683+0000 7fda05ffb640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd9f00a3f20 0x7fd9f00a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:13.683 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.684+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9561+0+0 (secure 0 0 0) 0x7fd9fc0d0140 con 0x7fda08108f50 2024-04-21T08:57:13.686 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.687+0000 7fda07fff640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd9f00a3f20 0x7fd9f00a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:13.690 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.690+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9fc01f030 con 0x7fda08108f50 2024-04-21T08:57:13.702 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:13.699+0000 7fda07fff640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd9f00a3f20 0x7fd9f00a63e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fda08072260 tx=0x7fd9f8032040 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:13.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a[27964]: 2024-04-21T08:57:13.432+0000 7f483a245700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3143541819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1880780242' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:57:13.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T08:57:13.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:13 smithi057 ceph-mon[27968]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T08:57:13.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i[62065]: ignoring --setuser ceph since I am not root 2024-04-21T08:57:13.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i[62065]: ignoring --setgroup ceph since I am not root 2024-04-21T08:57:13.885 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mds[62069]: -- 172.21.15.136:0/3624124482 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635a6c19a40 con 0x5635a6becc00 2024-04-21T08:57:13.886 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i[62065]: starting mds.i at 2024-04-21T08:57:13.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3143541819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:13.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1880780242' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:13.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:13.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:57:13.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:13.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:57:13.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T08:57:13.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:13 smithi136 ceph-mon[31791]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3143541819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1880780242' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-21T08:57:14.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: osdmap e111: 12 total, 12 up, 12 in 2024-04-21T08:57:14.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:13 smithi129 ceph-mon[31272]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-04-21T08:57:14.116 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.115+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fda08110e90 con 0x7fda08108f50 2024-04-21T08:57:14.442 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.442+0000 7fda05ffb640 1 -- 172.21.15.57:0/201388995 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7fd9fc09c480 con 0x7fda08108f50 2024-04-21T08:57:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.446+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd9f00a3f20 msgr2=0x7fd9f00a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.446+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd9f00a3f20 0x7fd9f00a63e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fda08072260 tx=0x7fd9f8032040 comp rx=0 tx=0).stop 2024-04-21T08:57:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.446+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 msgr2=0x7fda0819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:14.446 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.446+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda0819efd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc009fd0 tx=0x7fd9fc00b870 comp rx=0 tx=0).stop 2024-04-21T08:57:14.447 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.447+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 shutdown_connections 2024-04-21T08:57:14.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.447+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fd9f00a3f20 0x7fd9f00a63e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.447+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda0810e930 0x7fda081a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.447+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda08109910 0x7fda0819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.447+0000 7fda0ecd3640 1 --2- 172.21.15.57:0/201388995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda08108f50 0x7fda0819efd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.448 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.448+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 >> 172.21.15.57:0/201388995 conn(0x7fda0806d4d0 msgr2=0x7fda080729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:14.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.449+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 shutdown_connections 2024-04-21T08:57:14.449 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.449+0000 7fda0ecd3640 1 -- 172.21.15.57:0/201388995 wait complete. 2024-04-21T08:57:14.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-21T08:57:14.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: ignoring --setuser ceph since I am not root 2024-04-21T08:57:14.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: ignoring --setgroup ceph since I am not root 2024-04-21T08:57:14.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mds[61097]: -- 172.21.15.129:0/1078893230 <== mon.2 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a0cc32fa40 con 0x55a0cc303000 2024-04-21T08:57:14.574 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: starting mds.b at 2024-04-21T08:57:14.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.655+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3488690854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2781106d0 msgr2=0x7fe27810e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:14.655 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.655+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3488690854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2781106d0 0x7fe27810e270 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe270009930 tx=0x7fe27002bc60 comp rx=0 tx=0).stop 2024-04-21T08:57:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.659+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3488690854 shutdown_connections 2024-04-21T08:57:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.659+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3488690854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 0x7fe27811db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.659+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3488690854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe27810e7b0 0x7fe27810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.659+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3488690854 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2781106d0 0x7fe27810e270 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.660 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.659+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3488690854 >> 172.21.15.57:0/3488690854 conn(0x7fe27806d070 msgr2=0x7fe27806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:14.665 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.666+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3488690854 shutdown_connections 2024-04-21T08:57:14.666 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.667+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3488690854 wait complete. 2024-04-21T08:57:14.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.668+0000 7fe27e2eb640 1 Processor -- start 2024-04-21T08:57:14.667 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.668+0000 7fe27e2eb640 1 -- start start 2024-04-21T08:57:14.668 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.668+0000 7fe27e2eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe27810e7b0 0x7fe2781a7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:14.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe27e2eb640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2781106d0 0x7fe2781a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:14.669 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe277fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe27810e7b0 0x7fe2781a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe277fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe27810e7b0 0x7fe2781a7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47302/0 (socket says 172.21.15.57:47302) 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe277fff640 1 -- 172.21.15.57:0/3095502755 learned_addr learned my addr 172.21.15.57:0/3095502755 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 0x7fe2781abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe278121100 con 0x7fe27810e7b0 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe278120f80 con 0x7fe2781106d0 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe278121280 con 0x7fe27811b730 2024-04-21T08:57:14.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.669+0000 7fe27c888640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 0x7fe2781abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:14.671 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27c888640 1 -- 172.21.15.57:0/3095502755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2781106d0 msgr2=0x7fe2781a7c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27c888640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2781106d0 0x7fe2781a7c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27c888640 1 -- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe27810e7b0 msgr2=0x7fe2781a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27c888640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe27810e7b0 0x7fe2781a7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27c888640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2781ac6f0 con 0x7fe27811b730 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27c888640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 0x7fe2781abfc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe27806b640 tx=0x7fe26c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c015020 con 0x7fe27811b730 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.671+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2781ac9e0 con 0x7fe27811b730 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.672+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2781b4270 con 0x7fe27811b730 2024-04-21T08:57:14.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.672+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe26c011ce0 con 0x7fe27811b730 2024-04-21T08:57:14.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.673+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c010d30 con 0x7fe27811b730 2024-04-21T08:57:14.675 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.676+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe26c018020 con 0x7fe27811b730 2024-04-21T08:57:14.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.677+0000 7fe2757fa640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe2600a3d40 0x7fe2600a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:14.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.677+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9561+0+0 (secure 0 0 0) 0x7fe26c0d0300 con 0x7fe27811b730 2024-04-21T08:57:14.681 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.682+0000 7fe277fff640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe2600a3d40 0x7fe2600a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:14.683 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.683+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe264005290 con 0x7fe27811b730 2024-04-21T08:57:14.684 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.684+0000 7fe277fff640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe2600a3d40 0x7fe2600a6200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe270006010 tx=0x7fe270000bc0 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:14.691 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:14.692+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe26c01f020 con 0x7fe27811b730 2024-04-21T08:57:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: pgmap v373: 129 pgs: 8 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 105 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 27 op/s 2024-04-21T08:57:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/201388995' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:14.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: Cluster is now healthy 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.136:6838/3119239737,v1:172.21.15.136:6839/3119239737] up:boot 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: mds.? [v2:172.21.15.129:6838/192662750,v1:172.21.15.129:6839/192662750] up:boot 2024-04-21T08:57:14.840 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: fsmap 12 up:standby 2024-04-21T08:57:14.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:57:14.841 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:14 smithi057 ceph-mon[27968]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:57:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: pgmap v373: 129 pgs: 8 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 105 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 27 op/s 2024-04-21T08:57:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/201388995' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:14.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: Cluster is now healthy 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.136:6838/3119239737,v1:172.21.15.136:6839/3119239737] up:boot 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: mds.? [v2:172.21.15.129:6838/192662750,v1:172.21.15.129:6839/192662750] up:boot 2024-04-21T08:57:14.886 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: fsmap 12 up:standby 2024-04-21T08:57:14.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:57:14.887 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:14 smithi136 ceph-mon[31791]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:57:15.059 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.060+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe264005580 con 0x7fe27811b730 2024-04-21T08:57:15.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: pgmap v373: 129 pgs: 8 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 105 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 27 op/s 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/201388995' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: Cluster is now healthy 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.136:6838/3119239737,v1:172.21.15.136:6839/3119239737] up:boot 2024-04-21T08:57:15.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: mds.? [v2:172.21.15.129:6838/192662750,v1:172.21.15.129:6839/192662750] up:boot 2024-04-21T08:57:15.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: fsmap 12 up:standby 2024-04-21T08:57:15.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-21T08:57:15.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:14 smithi129 ceph-mon[31272]: from='mgr.15375 172.21.15.57:0/557797553' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-21T08:57:15.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.617+0000 7fe2757fa640 1 -- 172.21.15.57:0/3095502755 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) ==== 181+0+0 (secure 0 0 0) 0x7fe26c09c640 con 0x7fe27811b730 2024-04-21T08:57:15.618 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' removed 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.620+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe2600a3d40 msgr2=0x7fe2600a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.620+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe2600a3d40 0x7fe2600a6200 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe270006010 tx=0x7fe270000bc0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 msgr2=0x7fe2781abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 0x7fe2781abfc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe27806b640 tx=0x7fe26c00b870 comp rx=0 tx=0).stop 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 shutdown_connections 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fe2600a3d40 0x7fe2600a6200 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe27811b730 0x7fe2781abfc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2781106d0 0x7fe2781a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 --2- 172.21.15.57:0/3095502755 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe27810e7b0 0x7fe2781a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 >> 172.21.15.57:0/3095502755 conn(0x7fe27806d070 msgr2=0x7fe27811a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:15.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 shutdown_connections 2024-04-21T08:57:15.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.621+0000 7fe27e2eb640 1 -- 172.21.15.57:0/3095502755 wait complete. 2024-04-21T08:57:15.638 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-21T08:57:15.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.816+0000 7fc5027f4640 1 -- 172.21.15.57:0/3380766508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 msgr2=0x7fc4fc10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:15.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.816+0000 7fc5027f4640 1 --2- 172.21.15.57:0/3380766508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc10ec50 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009930 tx=0x7fc4f002bc30 comp rx=0 tx=0).stop 2024-04-21T08:57:15.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 -- 172.21.15.57:0/3380766508 shutdown_connections 2024-04-21T08:57:15.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 --2- 172.21.15.57:0/3380766508 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4fc11b730 0x7fc4fc11db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 --2- 172.21.15.57:0/3380766508 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc10ec50 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.817 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 --2- 172.21.15.57:0/3380766508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4fc1107a0 0x7fc4fc10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 -- 172.21.15.57:0/3380766508 >> 172.21.15.57:0/3380766508 conn(0x7fc4fc06d060 msgr2=0x7fc4fc06d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:15.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 -- 172.21.15.57:0/3380766508 shutdown_connections 2024-04-21T08:57:15.818 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 -- 172.21.15.57:0/3380766508 wait complete. 2024-04-21T08:57:15.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 Processor -- start 2024-04-21T08:57:15.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.817+0000 7fc5027f4640 1 -- start start 2024-04-21T08:57:15.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc5027f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc1a75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:15.819 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc5027f4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4fc1107a0 0x7fc4fc1a7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc5027f4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4fc11b730 0x7fc4fc1abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc5027f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc4fc120fa0 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc5027f4640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc4fc120e20 con 0x7fc4fc11b730 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc5027f4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc4fc121120 con 0x7fc4fc1107a0 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc4fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc1a75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc4fbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc1a75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:47330/0 (socket says 172.21.15.57:47330) 2024-04-21T08:57:15.820 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc4fbfff640 1 -- 172.21.15.57:0/546062408 learned_addr learned my addr 172.21.15.57:0/546062408 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.818+0000 7fc4fb7fe640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4fc1107a0 0x7fc4fc1a7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4fbfff640 1 -- 172.21.15.57:0/546062408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4fc1107a0 msgr2=0x7fc4fc1a7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4fbfff640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4fc1107a0 0x7fc4fc1a7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4fbfff640 1 -- 172.21.15.57:0/546062408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4fc11b730 msgr2=0x7fc4fc1abe60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4fbfff640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4fc11b730 0x7fc4fc1abe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4fbfff640 1 -- 172.21.15.57:0/546062408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4fc1ac590 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.821 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4fbfff640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc1a75a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00d950 tx=0x7fc4ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:15.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec017070 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.819+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4ec009c30 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.822 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.820+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec00f620 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.824+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4fc1ac880 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.824+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4fc1b4110 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.834+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4fc06e300 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.835+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc4ec010460 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.836 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.836+0000 7fc4f97fa640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc4e40a3ef0 0x7fc4e40a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:15.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.836+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9036+0+0 (secure 0 0 0) 0x7fc4ec0d0330 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.837+0000 7fc4fb7fe640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc4e40a3ef0 0x7fc4e40a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:15.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.840+0000 7fc4fb7fe640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc4e40a3ef0 0x7fc4e40a63b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009900 tx=0x7fc4f0005a20 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:15.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:15.845+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4ec020070 con 0x7fc4fc10e7d0 2024-04-21T08:57:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:15 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/3095502755' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:15.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:15 smithi136 ceph-mon[31791]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:16.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:15 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/3095502755' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:16.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:15 smithi129 ceph-mon[31272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:16.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:15 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/3095502755' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:16.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:15 smithi057 ceph-mon[27968]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:16.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.205+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc4fc1b42e0 con 0x7fc4fc10e7d0 2024-04-21T08:57:16.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.613+0000 7fc4f97fa640 1 -- 172.21.15.57:0/546062408 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v113) ==== 169+0+0 (secure 0 0 0) 0x7fc4ec09c7f0 con 0x7fc4fc10e7d0 2024-04-21T08:57:16.613 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' removed 2024-04-21T08:57:16.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc4e40a3ef0 msgr2=0x7fc4e40a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:16.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc4e40a3ef0 0x7fc4e40a63b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0009900 tx=0x7fc4f0005a20 comp rx=0 tx=0).stop 2024-04-21T08:57:16.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 msgr2=0x7fc4fc1a75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:16.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc1a75a0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00d950 tx=0x7fc4ec00de10 comp rx=0 tx=0).stop 2024-04-21T08:57:16.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 shutdown_connections 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7fc4e40a3ef0 0x7fc4e40a63b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4fc11b730 0x7fc4fc1abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4fc1107a0 0x7fc4fc1a7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 --2- 172.21.15.57:0/546062408 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4fc10e7d0 0x7fc4fc1a75a0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 >> 172.21.15.57:0/546062408 conn(0x7fc4fc06d060 msgr2=0x7fc4fc11a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 shutdown_connections 2024-04-21T08:57:16.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.616+0000 7fc5027f4640 1 -- 172.21.15.57:0/546062408 wait complete. 2024-04-21T08:57:16.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-21T08:57:16.792 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.790+0000 7f7c45edf640 1 -- 172.21.15.57:0/1579724160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 msgr2=0x7f7c40101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.790+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1579724160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c40101500 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009860 tx=0x7f7c3402bb70 comp rx=0 tx=0).stop 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 -- 172.21.15.57:0/1579724160 shutdown_connections 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1579724160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c40101a40 0x7f7c4010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1579724160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c40101500 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1579724160 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40106ed0 0x7f7c401072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 -- 172.21.15.57:0/1579724160 >> 172.21.15.57:0/1579724160 conn(0x7f7c40077c40 msgr2=0x7f7c40078050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:16.793 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 -- 172.21.15.57:0/1579724160 shutdown_connections 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 -- 172.21.15.57:0/1579724160 wait complete. 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.791+0000 7f7c45edf640 1 Processor -- start 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c45edf640 1 -- start start 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c45edf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c4019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c45edf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40101a40 0x7f7c4019aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c45edf640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c40106ed0 0x7f7c4019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:16.794 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c45edf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40111080 con 0x7f7c40101a40 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c45edf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40110f00 con 0x7f7c401010a0 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.792+0000 7f7c3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c4019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c4019a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.57:34088/0 (socket says 172.21.15.57:34088) 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3f7fe640 1 -- 172.21.15.57:0/1255031709 learned_addr learned my addr 172.21.15.57:0/1255031709 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3ffff640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c40106ed0 0x7f7c4019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7c40111200 con 0x7f7c40106ed0 2024-04-21T08:57:16.795 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3effd640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40101a40 0x7f7c4019aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:16.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3effd640 1 -- 172.21.15.57:0/1255031709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c40106ed0 msgr2=0x7f7c4019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:16.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3effd640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c40106ed0 0x7f7c4019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3effd640 1 -- 172.21.15.57:0/1255031709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 msgr2=0x7f7c4019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:16.797 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3effd640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c4019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3effd640 1 -- 172.21.15.57:0/1255031709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4019f750 con 0x7f7c40101a40 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.793+0000 7f7c3f7fe640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c4019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.794+0000 7f7c3effd640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40101a40 0x7f7c4019aca0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7c3402c180 tx=0x7f7c34035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.794+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34041020 con 0x7f7c40101a40 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.794+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c340035f0 con 0x7f7c40101a40 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.794+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4019f9e0 con 0x7f7c40101a40 2024-04-21T08:57:16.798 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.794+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34039040 con 0x7f7c40101a40 2024-04-21T08:57:16.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.794+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c401a72d0 con 0x7f7c40101a40 2024-04-21T08:57:16.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.797+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c400684e0 con 0x7f7c40101a40 2024-04-21T08:57:16.799 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.798+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7c34044020 con 0x7f7c40101a40 2024-04-21T08:57:16.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.799+0000 7f7c3cff9640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7c280a3de0 0x7f7c280a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-21T08:57:16.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.799+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8477+0+0 (secure 0 0 0) 0x7f7c340f1fd0 con 0x7f7c40101a40 2024-04-21T08:57:16.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.800+0000 7f7c3f7fe640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7c280a3de0 0x7f7c280a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-21T08:57:16.800 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.800+0000 7f7c3f7fe640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7c280a3de0 0x7f7c280a62a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c30005fd0 tx=0x7f7c3000b480 comp rx=0 tx=0).ready entity=mgr.15375 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-21T08:57:16.806 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:16.806+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c3404b020 con 0x7f7c40101a40 2024-04-21T08:57:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:16 smithi136 ceph-mon[31791]: pgmap v375: 129 pgs: 17 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 70 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 12 op/s 2024-04-21T08:57:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:16 smithi136 ceph-mon[31791]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:16 smithi136 ceph-mon[31791]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T08:57:16.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:16 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/546062408' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:17.073 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:16 smithi129 ceph-mon[31272]: pgmap v375: 129 pgs: 17 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 70 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 12 op/s 2024-04-21T08:57:17.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:16 smithi129 ceph-mon[31272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:17.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:16 smithi129 ceph-mon[31272]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T08:57:17.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:16 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/546062408' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:17.086 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.086+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7c401a7530 con 0x7f7c40101a40 2024-04-21T08:57:17.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:16 smithi057 ceph-mon[27968]: pgmap v375: 129 pgs: 17 active+clean+snaptrim, 42 active+clean+snaptrim_wait, 70 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.9 MiB/s wr, 12 op/s 2024-04-21T08:57:17.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:16 smithi057 ceph-mon[27968]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:17.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:16 smithi057 ceph-mon[27968]: osdmap e112: 12 total, 12 up, 12 in 2024-04-21T08:57:17.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:16 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/546062408' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:17.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.625+0000 7f7c3cff9640 1 -- 172.21.15.57:0/1255031709 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v114) ==== 178+0+0 (secure 0 0 0) 0x7f7c340be7b0 con 0x7f7c40101a40 2024-04-21T08:57:17.625 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data_ec' removed 2024-04-21T08:57:17.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7c280a3de0 msgr2=0x7f7c280a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:17.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7c280a3de0 0x7f7c280a62a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c30005fd0 tx=0x7f7c3000b480 comp rx=0 tx=0).stop 2024-04-21T08:57:17.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40101a40 msgr2=0x7f7c4019aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40101a40 0x7f7c4019aca0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7c3402c180 tx=0x7f7c34035d10 comp rx=0 tx=0).stop 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 shutdown_connections 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:6800/3217118060,v1:172.21.15.57:6801/3217118060] conn(0x7f7c280a3de0 0x7f7c280a62a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c40106ed0 0x7f7c4019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c40101a40 0x7f7c4019aca0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 --2- 172.21.15.57:0/1255031709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c401010a0 0x7f7c4019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-21T08:57:17.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 >> 172.21.15.57:0/1255031709 conn(0x7f7c40077c40 msgr2=0x7f7c4006a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-21T08:57:17.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 shutdown_connections 2024-04-21T08:57:17.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-04-21T08:57:17.627+0000 7f7c45edf640 1 -- 172.21.15.57:0/1255031709 wait complete. 2024-04-21T08:57:17.643 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:57:17.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:57:17.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T08:57:17.739 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-21T08:57:17.740 DEBUG:teuthology.orchestra.run.smithi057:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:57:17.757 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:57:17.773 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-21T08:57:17.797 INFO:tasks.cephadm:Stopping all daemons... 2024-04-21T08:57:17.797 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-21T08:57:17.798 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a 2024-04-21T08:57:17.822 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:17 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/546062408' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:17.823 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:17 smithi057 ceph-mon[27968]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T08:57:17.823 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:17 smithi057 ceph-mon[27968]: from='client.? 172.21.15.57:0/1255031709' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:17.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:17 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/546062408' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:17.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:17 smithi136 ceph-mon[31791]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T08:57:17.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:17 smithi136 ceph-mon[31791]: from='client.? 172.21.15.57:0/1255031709' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:18.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:17 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/546062408' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-21T08:57:18.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:17 smithi129 ceph-mon[31272]: osdmap e113: 12 total, 12 up, 12 in 2024-04-21T08:57:18.074 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:17 smithi129 ceph-mon[31272]: from='client.? 172.21.15.57:0/1255031709' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-21T08:57:18.089 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:17 smithi057 systemd[1]: Stopping Ceph mon.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:18.527 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a[27964]: 2024-04-21T08:57:18.183+0000 7f483fa50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:18.528 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a[27964]: 2024-04-21T08:57:18.183+0000 7f483fa50700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-21T08:57:18.528 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:18 smithi057 podman[108123]: 2024-04-21 08:57:18.216177785 +0000 UTC m=+0.158556439 container died 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T08:57:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:18 smithi057 podman[108123]: 2024-04-21 08:57:18.634033407 +0000 UTC m=+0.576412070 container cleanup 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True) 2024-04-21T08:57:18.839 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:18 smithi057 bash[108123]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a 2024-04-21T08:57:19.228 INFO:journalctl@ceph.mon.a.smithi057.stdout:Apr 21 08:57:18 smithi057 podman[108136]: 2024-04-21 08:57:18.869312278 +0000 UTC m=+0.650631000 container remove 585910ff514b3ddd5cdfddcc4e14e377b9a2cfcb1c0542f0d0935d7f38db78e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-21T08:57:19.244 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.a.service' 2024-04-21T08:57:19.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:19.346 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-21T08:57:19.346 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-21T08:57:19.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.b 2024-04-21T08:57:19.717 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:19 smithi129 systemd[1]: Stopping Ceph mon.b for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:19.993 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b[31268]: 2024-04-21T08:57:19.716+0000 7fe5d56f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:19.993 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b[31268]: 2024-04-21T08:57:19.716+0000 7fe5d56f2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-21T08:57:20.293 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:19 smithi129 podman[66568]: 2024-04-21 08:57:19.993990621 +0000 UTC m=+0.406763051 container died a066933c3cdf93e877a3f064fa2419d87d792e95b84c82b2d80250a56954c483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:57:20.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:20 smithi129 podman[66568]: 2024-04-21 08:57:20.392720052 +0000 UTC m=+0.805492484 container cleanup a066933c3cdf93e877a3f064fa2419d87d792e95b84c82b2d80250a56954c483 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-21T08:57:20.574 INFO:journalctl@ceph.mon.b.smithi129.stdout:Apr 21 08:57:20 smithi129 bash[66568]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-b 2024-04-21T08:57:20.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.b.service' 2024-04-21T08:57:20.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:20.905 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-21T08:57:20.905 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-21T08:57:20.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.c 2024-04-21T08:57:21.220 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:20 smithi136 systemd[1]: Stopping Ceph mon.c for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:21.635 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-c[31774]: 2024-04-21T08:57:21.220+0000 7f6a14b5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:21.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-c[31774]: 2024-04-21T08:57:21.220+0000 7f6a14b5e700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-21T08:57:21.636 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:21 smithi136 podman[67930]: 2024-04-21 08:57:21.414452198 +0000 UTC m=+0.310805887 container died 830ca3f1e5998ad44ae7ef3c82db1011086b4f880c4aced7335d49f29987c05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:57:22.609 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:22 smithi136 podman[67930]: 2024-04-21 08:57:22.296970974 +0000 UTC m=+1.193324671 container cleanup 830ca3f1e5998ad44ae7ef3c82db1011086b4f880c4aced7335d49f29987c05d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:57:22.610 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 21 08:57:22 smithi136 bash[67930]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mon-c 2024-04-21T08:57:22.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mon.c.service' 2024-04-21T08:57:22.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:22.792 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-21T08:57:22.792 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-21T08:57:22.792 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x 2024-04-21T08:57:23.089 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:57:22 smithi057 systemd[1]: Stopping Ceph mgr.x for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:23.506 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:57:23 smithi057 podman[108239]: 2024-04-21 08:57:23.236875637 +0000 UTC m=+0.251368732 container died c79614f5e3d1fdfb7d099c8c5e5b15097877079c6f0737c3a476007379aa07a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-21T08:57:23.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:57:23 smithi057 podman[108239]: 2024-04-21 08:57:23.606383673 +0000 UTC m=+0.620876764 container cleanup c79614f5e3d1fdfb7d099c8c5e5b15097877079c6f0737c3a476007379aa07a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a630765c1f93c6e61d9b40c4b76437611d5ef29e, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux ) 2024-04-21T08:57:23.839 INFO:journalctl@ceph.mgr.x.smithi057.stdout:Apr 21 08:57:23 smithi057 bash[108239]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-x 2024-04-21T08:57:24.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.x.service' 2024-04-21T08:57:24.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:24.111 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-21T08:57:24.112 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-21T08:57:24.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.y 2024-04-21T08:57:24.443 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:57:24 smithi129 systemd[1]: Stopping Ceph mgr.y for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:24.748 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:57:24 smithi129 podman[66693]: 2024-04-21 08:57:24.44344881 +0000 UTC m=+0.143801287 container died b3b453020d0251078ac6efed76621a272319b93568d0fd0204bb996649cbebd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-21T08:57:25.074 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:57:24 smithi129 podman[66693]: 2024-04-21 08:57:24.856731487 +0000 UTC m=+0.557084004 container cleanup b3b453020d0251078ac6efed76621a272319b93568d0fd0204bb996649cbebd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-21T08:57:25.074 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Apr 21 08:57:24 smithi129 bash[66693]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-y 2024-04-21T08:57:25.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.y.service' 2024-04-21T08:57:25.294 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:25.294 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-21T08:57:25.295 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-21T08:57:25.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.z 2024-04-21T08:57:25.635 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:25 smithi136 systemd[1]: Stopping Ceph mgr.z for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:27.135 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:26 smithi136 podman[68053]: 2024-04-21 08:57:26.684265619 +0000 UTC m=+1.192103926 container died b1b5b8418733e7ef9e9f14a430309fc1d9be97ce69b117b1ac2e7079d42cf883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T08:57:27.553 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 podman[68053]: 2024-04-21 08:57:27.302989298 +0000 UTC m=+1.810827607 container cleanup b1b5b8418733e7ef9e9f14a430309fc1d9be97ce69b117b1ac2e7079d42cf883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-21T08:57:27.554 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 bash[68053]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z 2024-04-21T08:57:27.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.z.service' 2024-04-21T08:57:27.809 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 podman[68066]: 2024-04-21 08:57:27.554481497 +0000 UTC m=+0.873093853 container remove b1b5b8418733e7ef9e9f14a430309fc1d9be97ce69b117b1ac2e7079d42cf883 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-21T08:57:27.809 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-21T08:57:27.809 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-04-21T08:57:27.809 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 systemd[1]: Stopped Ceph mgr.z for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:57:27.809 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 21 08:57:27 smithi136 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mgr.z.service: Consumed 20.568s CPU time. 2024-04-21T08:57:27.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:27.821 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-21T08:57:27.822 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-21T08:57:27.822 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.0 2024-04-21T08:57:28.186 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:27 smithi057 systemd[1]: Stopping Ceph osd.0 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:28.589 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0[42757]: 2024-04-21T08:57:28.186+0000 7fd19df61700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:28.589 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0[42757]: 2024-04-21T08:57:28.186+0000 7fd19df61700 -1 osd.0 114 *** Got signal Terminated *** 2024-04-21T08:57:28.590 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0[42757]: 2024-04-21T08:57:28.186+0000 7fd19df61700 -1 osd.0 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:57:33.929 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:33 smithi057 podman[108359]: 2024-04-21 08:57:33.643528733 +0000 UTC m=+5.620516270 container died 788dcec6d3f711a1a4b07962a75c2b7eb30067e8d7c006267ee67287fd5d0652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:57:34.329 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 podman[108359]: 2024-04-21 08:57:34.02066557 +0000 UTC m=+5.997653113 container cleanup 788dcec6d3f711a1a4b07962a75c2b7eb30067e8d7c006267ee67287fd5d0652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T08:57:34.330 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 bash[108359]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0 2024-04-21T08:57:34.589 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 podman[108376]: 2024-04-21 08:57:34.330616487 +0000 UTC m=+0.686924773 container remove 788dcec6d3f711a1a4b07962a75c2b7eb30067e8d7c006267ee67287fd5d0652 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-21T08:57:34.975 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 podman[108467]: 2024-04-21 08:57:34.601616252 +0000 UTC m=+0.018785053 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:57:34.975 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 podman[108467]: 2024-04-21 08:57:34.72349679 +0000 UTC m=+0.140665604 container create bd745feb2f9926d8aaa5ab19cb7bc8f0ea2190e7c5e53b4f86c5f8edcc724252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-21T08:57:35.291 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 podman[108467]: 2024-04-21 08:57:34.976230284 +0000 UTC m=+0.393399059 container init bd745feb2f9926d8aaa5ab19cb7bc8f0ea2190e7c5e53b4f86c5f8edcc724252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T08:57:35.291 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:34 smithi057 podman[108467]: 2024-04-21 08:57:34.979679999 +0000 UTC m=+0.396848789 container start bd745feb2f9926d8aaa5ab19cb7bc8f0ea2190e7c5e53b4f86c5f8edcc724252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-21T08:57:35.291 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:35 smithi057 podman[108467]: 2024-04-21 08:57:35.092468407 +0000 UTC m=+0.509637190 container attach bd745feb2f9926d8aaa5ab19cb7bc8f0ea2190e7c5e53b4f86c5f8edcc724252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-21T08:57:35.291 INFO:journalctl@ceph.osd.0.smithi057.stdout:Apr 21 08:57:35 smithi057 podman[108467]: 2024-04-21 08:57:35.291735539 +0000 UTC m=+0.708904318 container died bd745feb2f9926d8aaa5ab19cb7bc8f0ea2190e7c5e53b4f86c5f8edcc724252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-21T08:57:35.859 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.0.service' 2024-04-21T08:57:35.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:35.899 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-21T08:57:35.899 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-21T08:57:35.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.1 2024-04-21T08:57:36.220 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:35 smithi129 systemd[1]: Stopping Ceph osd.1 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:36.573 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1[35872]: 2024-04-21T08:57:36.220+0000 7fd000e23700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:36.574 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1[35872]: 2024-04-21T08:57:36.220+0000 7fd000e23700 -1 osd.1 114 *** Got signal Terminated *** 2024-04-21T08:57:36.574 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1[35872]: 2024-04-21T08:57:36.220+0000 7fd000e23700 -1 osd.1 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:57:42.073 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:41 smithi129 podman[66832]: 2024-04-21 08:57:41.784339716 +0000 UTC m=+5.691912508 container died 7b0cf388cccdeaeb9b07a1fe3afe63c8bdd46c53e46a0f87e473bc2fc6ece5ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-21T08:57:42.573 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:42 smithi129 podman[66832]: 2024-04-21 08:57:42.230594885 +0000 UTC m=+6.138167678 container cleanup 7b0cf388cccdeaeb9b07a1fe3afe63c8bdd46c53e46a0f87e473bc2fc6ece5ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-21T08:57:42.574 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:42 smithi129 bash[66832]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1 2024-04-21T08:57:42.902 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:42 smithi129 podman[66849]: 2024-04-21 08:57:42.588209507 +0000 UTC m=+0.829018710 container remove 7b0cf388cccdeaeb9b07a1fe3afe63c8bdd46c53e46a0f87e473bc2fc6ece5ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:57:43.200 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:42 smithi129 podman[66940]: 2024-04-21 08:57:42.806547454 +0000 UTC m=+0.018285958 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:57:43.200 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:42 smithi129 podman[66940]: 2024-04-21 08:57:42.939182976 +0000 UTC m=+0.150921447 container create 67254f69884b87c47af783abf6784b7db12991e96fdb98fef5532ef1ab85538c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-21T08:57:43.573 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:43 smithi129 podman[66940]: 2024-04-21 08:57:43.200520161 +0000 UTC m=+0.412258633 container init 67254f69884b87c47af783abf6784b7db12991e96fdb98fef5532ef1ab85538c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T08:57:43.574 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:43 smithi129 podman[66940]: 2024-04-21 08:57:43.204063902 +0000 UTC m=+0.415802381 container start 67254f69884b87c47af783abf6784b7db12991e96fdb98fef5532ef1ab85538c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-21T08:57:43.574 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:43 smithi129 podman[66940]: 2024-04-21 08:57:43.308911906 +0000 UTC m=+0.520650396 container attach 67254f69884b87c47af783abf6784b7db12991e96fdb98fef5532ef1ab85538c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T08:57:44.253 INFO:journalctl@ceph.osd.1.smithi129.stdout:Apr 21 08:57:43 smithi129 podman[66940]: 2024-04-21 08:57:43.889158425 +0000 UTC m=+1.100896897 container died 67254f69884b87c47af783abf6784b7db12991e96fdb98fef5532ef1ab85538c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T08:57:44.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.1.service' 2024-04-21T08:57:44.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:44.607 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-21T08:57:44.607 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-21T08:57:44.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.2 2024-04-21T08:57:44.885 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:44 smithi136 systemd[1]: Stopping Ceph osd.2 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:45.248 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2[36946]: 2024-04-21T08:57:44.947+0000 7eff38ce3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:45.249 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2[36946]: 2024-04-21T08:57:44.947+0000 7eff38ce3700 -1 osd.2 114 *** Got signal Terminated *** 2024-04-21T08:57:45.249 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2[36946]: 2024-04-21T08:57:44.947+0000 7eff38ce3700 -1 osd.2 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:57:50.800 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:50 smithi136 podman[68187]: 2024-04-21 08:57:50.502784605 +0000 UTC m=+5.695307736 container died bcf0f6e7c212af9be6cf2e144d95cdb04d2eee057c00b772e22941fa0908832e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:57:51.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:50 smithi136 podman[68187]: 2024-04-21 08:57:50.900488951 +0000 UTC m=+6.093012081 container cleanup bcf0f6e7c212af9be6cf2e144d95cdb04d2eee057c00b772e22941fa0908832e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True) 2024-04-21T08:57:51.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:50 smithi136 bash[68187]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2 2024-04-21T08:57:51.385 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:51 smithi136 podman[68204]: 2024-04-21 08:57:51.135962343 +0000 UTC m=+0.628001984 container remove bcf0f6e7c212af9be6cf2e144d95cdb04d2eee057c00b772e22941fa0908832e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-21T08:57:51.862 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:51 smithi136 podman[68294]: 2024-04-21 08:57:51.464787349 +0000 UTC m=+0.018118824 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:57:51.862 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:51 smithi136 podman[68294]: 2024-04-21 08:57:51.745350105 +0000 UTC m=+0.298681550 container create 81e9809939370bc4fdb9d49516e3039165e5453021e40fa7b417b396c7d64f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-21T08:57:52.135 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:51 smithi136 podman[68294]: 2024-04-21 08:57:51.982008062 +0000 UTC m=+0.535339514 container init 81e9809939370bc4fdb9d49516e3039165e5453021e40fa7b417b396c7d64f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-21T08:57:52.136 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:51 smithi136 podman[68294]: 2024-04-21 08:57:51.985484387 +0000 UTC m=+0.538815833 container start 81e9809939370bc4fdb9d49516e3039165e5453021e40fa7b417b396c7d64f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:57:52.136 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:52 smithi136 podman[68294]: 2024-04-21 08:57:52.10114848 +0000 UTC m=+0.654479934 container attach 81e9809939370bc4fdb9d49516e3039165e5453021e40fa7b417b396c7d64f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:57:52.567 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 21 08:57:52 smithi136 podman[68294]: 2024-04-21 08:57:52.285631926 +0000 UTC m=+0.838963375 container died 81e9809939370bc4fdb9d49516e3039165e5453021e40fa7b417b396c7d64f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-21T08:57:53.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.2.service' 2024-04-21T08:57:53.048 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:57:53.048 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-21T08:57:53.049 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-21T08:57:53.049 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.3 2024-04-21T08:57:53.339 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:53 smithi057 systemd[1]: Stopping Ceph osd.3 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:57:53.839 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:53 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3[49735]: 2024-04-21T08:57:53.367+0000 7f67a4df2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:57:53.839 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:53 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3[49735]: 2024-04-21T08:57:53.367+0000 7f67a4df2700 -1 osd.3 114 *** Got signal Terminated *** 2024-04-21T08:57:53.839 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:53 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3[49735]: 2024-04-21T08:57:53.367+0000 7f67a4df2700 -1 osd.3 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:57:54.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:54 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:54.011+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:57:54.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:57:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:57:54.397+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:57:55.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:55 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:55.050+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:57:55.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:57:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:57:55.410+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:57:56.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:56 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:56.031+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:57:56.859 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:57:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:57:56.448+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:57:57.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:57:56 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:57:56.677+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:57:57.135 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:57:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:57:56.860+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:57:57.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:57 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:57.064+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:57:57.881 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:57:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:57:57.475+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:57:58.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:57:57 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:57:57.707+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:57:58.135 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:57:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:57:57.881+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:57:58.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:58 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:58.050+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:57:58.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:57:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:57:58.441+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:57:59.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:57:58 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:57:58.717+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:57:59.327 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:59 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:59.001+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:57:59.327 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:59 smithi057 podman[108586]: 2024-04-21 08:57:59.030137601 +0000 UTC m=+5.772537337 container died be3da817f3a480202ef2267623ad796b399a458c06dcbbca64a160b3d881f89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:57:59.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:57:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:57:58.886+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:57:59.589 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:59 smithi057 podman[108586]: 2024-04-21 08:57:59.426739817 +0000 UTC m=+6.169139553 container cleanup be3da817f3a480202ef2267623ad796b399a458c06dcbbca64a160b3d881f89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:57:59.589 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:59 smithi057 bash[108586]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3 2024-04-21T08:57:59.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:57:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:57:59.441+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:57:59.983 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:57:59 smithi057 podman[108603]: 2024-04-21 08:57:59.64497789 +0000 UTC m=+0.611324772 container remove be3da817f3a480202ef2267623ad796b399a458c06dcbbca64a160b3d881f89b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T08:58:00.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:57:59 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:57:59.701+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:00.337 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:57:59 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:57:59.983+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:00.337 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:58:00 smithi057 podman[108692]: 2024-04-21 08:57:59.96809219 +0000 UTC m=+0.018699836 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:00.337 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:58:00 smithi057 podman[108692]: 2024-04-21 08:58:00.104309702 +0000 UTC m=+0.154917329 container create 76e619c41d1323384372023199fe19f262685d59b1f8797d9a86d8b4666976ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True) 2024-04-21T08:58:00.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:57:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:57:59.915+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:00.589 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:58:00 smithi057 podman[108692]: 2024-04-21 08:58:00.491057282 +0000 UTC m=+0.541664910 container init 76e619c41d1323384372023199fe19f262685d59b1f8797d9a86d8b4666976ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-21T08:58:00.589 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:58:00 smithi057 podman[108692]: 2024-04-21 08:58:00.495401797 +0000 UTC m=+0.546009427 container start 76e619c41d1323384372023199fe19f262685d59b1f8797d9a86d8b4666976ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-21T08:58:00.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:00.457+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:00.984 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:58:00 smithi057 podman[108692]: 2024-04-21 08:58:00.591082856 +0000 UTC m=+0.641690507 container attach 76e619c41d1323384372023199fe19f262685d59b1f8797d9a86d8b4666976ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T08:58:00.984 INFO:journalctl@ceph.osd.3.smithi057.stdout:Apr 21 08:58:00 smithi057 podman[108692]: 2024-04-21 08:58:00.802169844 +0000 UTC m=+0.852777472 container died 76e619c41d1323384372023199fe19f262685d59b1f8797d9a86d8b4666976ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:01.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:00 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:00.734+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:01.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:00 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:00.984+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:01.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:00.912+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:01.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.3.service' 2024-04-21T08:58:01.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:01.483 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-21T08:58:01.483 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-21T08:58:01.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.4 2024-04-21T08:58:01.766 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:01 smithi129 systemd[1]: Stopping Ceph osd.4 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:01.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:01.458+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:02.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:01.765+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:02.074 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4[42512]: 2024-04-21T08:58:01.798+0000 7f97e0e11700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:02.074 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4[42512]: 2024-04-21T08:58:01.798+0000 7f97e0e11700 -1 osd.4 114 *** Got signal Terminated *** 2024-04-21T08:58:02.074 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:01 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4[42512]: 2024-04-21T08:58:01.798+0000 7f97e0e11700 -1 osd.4 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:02.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:02.029+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:02.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:01.904+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:02.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:02.465+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:03.073 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:02.721+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:03.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:02.718+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:03.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:02 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:02.718+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:03.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:03 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:03.038+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:03.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:02 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:02.987+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:03.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:02.934+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:03.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:02.934+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:03.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:03 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:03.479+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:03.975 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:03 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:03.753+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:03.975 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:03 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:03.708+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:03.975 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:03 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:03.708+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:04.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:04 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:04.018+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:04.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:04 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:04.029+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:04.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:04 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:04.029+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:04.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:03 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:03.957+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:04.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:03 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:03.957+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:04.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:04 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:04.472+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:04.988 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:04 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:04.763+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:04.989 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:04 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:04.723+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:04.989 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:04 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:04.723+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:05.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:05 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:05.000+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:05.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:05 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:05.046+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:05.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:05 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:05.046+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:05.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:04 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:04.990+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:05.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:04 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:04.990+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:05.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:05 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:05.456+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:06.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:05 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:05.769+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:06.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:05 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:05.752+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:06.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:05 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:05.752+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:06.240 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:05 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:05.965+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:06.241 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:06 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:06.044+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:06.241 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:06 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:06.044+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:06.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:06 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:06.025+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:06.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:06 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:06.025+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:06.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:06 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:06.428+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:07.073 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:06 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:06.754+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:07.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:06 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:06.782+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:07.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:06 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:06.782+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:07.252 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:06 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:06.945+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:07.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:07 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:07.001+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:07.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:07 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:07.001+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:07.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:07 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:07.068+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:07.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:07 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:07.068+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:07.573 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:07 smithi129 podman[67049]: 2024-04-21 08:58:07.295485662 +0000 UTC m=+5.619219024 container died 3ca255951bdfa4ca2c51075046aa5bf3ed0cad357e6de5b9106d509b86f7fffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-21T08:58:07.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:07 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:07.545+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:07.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:07 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:07.439+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:07.941 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:07 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:07.779+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:07.941 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:07 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:07.733+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:07.941 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:07 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:07.733+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:07.942 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:07 smithi129 podman[67049]: 2024-04-21 08:58:07.674200769 +0000 UTC m=+5.997934139 container cleanup 3ca255951bdfa4ca2c51075046aa5bf3ed0cad357e6de5b9106d509b86f7fffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:07.942 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:07 smithi129 bash[67049]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4 2024-04-21T08:58:08.297 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:07 smithi129 podman[67078]: 2024-04-21 08:58:07.942165889 +0000 UTC m=+0.644931743 container remove 3ca255951bdfa4ca2c51075046aa5bf3ed0cad357e6de5b9106d509b86f7fffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T08:58:08.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:07 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:07.976+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:08.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:08 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:08.039+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:08.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:08 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:08.039+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:08.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:08 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:08.065+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:08.386 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:08 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:08.065+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:08.574 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:08 smithi129 podman[67171]: 2024-04-21 08:58:08.201451803 +0000 UTC m=+0.018338024 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:08.574 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:08 smithi129 podman[67171]: 2024-04-21 08:58:08.338788505 +0000 UTC m=+0.155674763 container create 54a6959452512254162b0246f0347c4fb71df9a5954bea08331f05a7f7e401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-21T08:58:08.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:08 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:08.557+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:08.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:08 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:08.418+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:08.893 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:08 smithi129 podman[67171]: 2024-04-21 08:58:08.587091996 +0000 UTC m=+0.403978218 container init 54a6959452512254162b0246f0347c4fb71df9a5954bea08331f05a7f7e401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:58:08.894 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:08 smithi129 podman[67171]: 2024-04-21 08:58:08.590242722 +0000 UTC m=+0.407128934 container start 54a6959452512254162b0246f0347c4fb71df9a5954bea08331f05a7f7e401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:08.894 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:08 smithi129 podman[67171]: 2024-04-21 08:58:08.670429482 +0000 UTC m=+0.487315698 container attach 54a6959452512254162b0246f0347c4fb71df9a5954bea08331f05a7f7e401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-21T08:58:08.894 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:08 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:08.763+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:08.894 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:08 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:08.749+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:08.895 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:08 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:08.749+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:09.248 INFO:journalctl@ceph.osd.4.smithi129.stdout:Apr 21 08:58:08 smithi129 podman[67171]: 2024-04-21 08:58:08.893512412 +0000 UTC m=+0.710398627 container died 54a6959452512254162b0246f0347c4fb71df9a5954bea08331f05a7f7e401e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=) 2024-04-21T08:58:09.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:08 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:08.956+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:09.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:09 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:09.044+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:09.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:09 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:09.044+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:09.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:09.040+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:09.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:09.040+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:09.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.4.service' 2024-04-21T08:58:09.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:09.598 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-21T08:58:09.598 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-21T08:58:09.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.5 2024-04-21T08:58:09.824 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:09 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:09.769+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:09.824 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:09 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:09.788+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:09.824 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:09 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:09.788+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:09.885 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:09 smithi136 systemd[1]: Stopping Ceph osd.5 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:09.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:09.572+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:09.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:09.394+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:10.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:09.999+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:10.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:10.072+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:10.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:10 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:10.072+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:10.379 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:09.930+0000 7f3330eac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:10.379 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:09.930+0000 7f3330eac700 -1 osd.5 114 *** Got signal Terminated *** 2024-04-21T08:58:10.379 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:09 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:09.930+0000 7f3330eac700 -1 osd.5 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:10.380 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:10 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:10.069+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:10.380 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:10 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:10.069+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:10.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:10 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:10.379+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:10.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:10 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:10.606+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:11.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:10 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:10.774+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:11.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:10 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:10.774+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:11.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:10 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:10.803+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:11.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:10 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:10.803+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:11.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:11 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:11.005+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:11.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:11 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:11.077+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:11.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:11 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:11.077+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:11.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:11 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:11.111+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:11.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:11 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:11.111+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:11.386 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:11 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:11.111+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.422393+0000 front 2024-04-21T08:57:49.422342+0000 (oldest deadline 2024-04-21T08:58:10.522356+0000) 2024-04-21T08:58:11.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:11 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:11.423+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:11.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:11 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:11.600+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:12.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:11 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:11.803+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:12.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:11 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:11.803+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:12.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:11 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:11.817+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:12.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:11 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:11.817+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:12.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:12 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:12.006+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:12.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:12 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:12.029+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:12.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:12 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:12.029+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:12.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:12 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:12.133+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:12.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:12 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:12.133+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:12.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:12 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:12.133+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.422393+0000 front 2024-04-21T08:57:49.422342+0000 (oldest deadline 2024-04-21T08:58:10.522356+0000) 2024-04-21T08:58:12.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:12 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:12.396+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:12.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:12 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:12.586+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:13.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:12 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:12.796+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:13.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:12 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:12.796+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:13.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:12 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:12.837+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:13.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:12 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:12.837+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:13.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:13.040+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:13.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:13.040+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:13.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:13.002+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:13.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:13.002+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:13.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:13.109+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:13.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:13.109+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:13.386 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:13.109+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.422393+0000 front 2024-04-21T08:57:49.422342+0000 (oldest deadline 2024-04-21T08:58:10.522356+0000) 2024-04-21T08:58:13.885 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:13.387+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:13.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:13.629+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:13.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:13 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:13.629+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:14.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:13 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:13.856+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:14.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:13 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:13.856+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:14.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:13 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:13.840+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:14.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:13 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:13.840+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:14.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:13 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:13.840+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:14.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:14.037+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:14.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:14.037+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:14.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:14.006+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:14.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:14.006+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:14.357 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:14 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:14.102+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.718428+0000 front 2024-04-21T08:57:32.718552+0000 (oldest deadline 2024-04-21T08:57:56.818668+0000) 2024-04-21T08:58:14.357 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:14 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:14.102+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.319647+0000 front 2024-04-21T08:57:37.319589+0000 (oldest deadline 2024-04-21T08:58:02.019736+0000) 2024-04-21T08:58:14.357 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:14 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5[43446]: 2024-04-21T08:58:14.102+0000 7f332d4a0700 -1 osd.5 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.422393+0000 front 2024-04-21T08:57:49.422342+0000 (oldest deadline 2024-04-21T08:58:10.522356+0000) 2024-04-21T08:58:14.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:14 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:14.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:15.135 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:14 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:14.638+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:15.135 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:14 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:14.638+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:15.323 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:14.838+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:15.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:14.838+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:15.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:14.887+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:15.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:14.887+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:15.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:14 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:14.887+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:15.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:15.068+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:15.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:15.068+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:15.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:14.963+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:15.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:14.963+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:15.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:14 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:14.963+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:15.625 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:15 smithi136 podman[68413]: 2024-04-21 08:58:15.514316595 +0000 UTC m=+5.724371392 container died 80023ee2d437c46beadc71a9b9920fca7f6a2bee2268c057d3e02d536d28fceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:15.626 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:15 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:15.624+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:15.626 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:15 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:15.624+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:15.626 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:15 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:15.311+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:15.885 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:15 smithi136 podman[68413]: 2024-04-21 08:58:15.84724145 +0000 UTC m=+6.057296246 container cleanup 80023ee2d437c46beadc71a9b9920fca7f6a2bee2268c057d3e02d536d28fceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:15.885 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:15 smithi136 bash[68413]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5 2024-04-21T08:58:16.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:15.863+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:16.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:15.863+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:16.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:15.893+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:16.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:15.893+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:16.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:15 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:15.893+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:16.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:16.096+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:16.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:16.096+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:16.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:15.961+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:16.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:15.961+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:16.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:15 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:15.961+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:16.346 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:16 smithi136 podman[68430]: 2024-04-21 08:58:16.082087722 +0000 UTC m=+0.568319544 container remove 80023ee2d437c46beadc71a9b9920fca7f6a2bee2268c057d3e02d536d28fceb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-21T08:58:16.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:16 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:16.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:16.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:16 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:16.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:16.636 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:16 smithi136 podman[68521]: 2024-04-21 08:58:16.404488915 +0000 UTC m=+0.017663565 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:16.636 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:16 smithi136 podman[68521]: 2024-04-21 08:58:16.524654199 +0000 UTC m=+0.137828836 container create 9df711c82c699bc88dbee90cfb675a59550951b6dbd181cee2a87af54934f113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-21T08:58:16.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:16 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:16.582+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:16.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:16 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:16.582+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:17.092 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:16 smithi136 podman[68521]: 2024-04-21 08:58:16.794686682 +0000 UTC m=+0.407861319 container init 9df711c82c699bc88dbee90cfb675a59550951b6dbd181cee2a87af54934f113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, ceph=True) 2024-04-21T08:58:17.092 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:16 smithi136 podman[68521]: 2024-04-21 08:58:16.797771931 +0000 UTC m=+0.410946569 container start 9df711c82c699bc88dbee90cfb675a59550951b6dbd181cee2a87af54934f113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-21T08:58:17.092 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:16 smithi136 podman[68521]: 2024-04-21 08:58:16.878078984 +0000 UTC m=+0.491253630 container attach 9df711c82c699bc88dbee90cfb675a59550951b6dbd181cee2a87af54934f113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T08:58:17.298 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:17.108+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:17.299 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:17.108+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:17.299 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:16.967+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:17.299 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:16.967+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:17.299 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:16 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:16.967+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:17.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:16 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:16.862+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:17.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:16 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:16.862+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:17.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:16 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:16.904+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:17.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:16 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:16.904+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:17.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:16 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:16.905+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:17.385 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 21 08:58:17 smithi136 podman[68521]: 2024-04-21 08:58:17.0934857 +0000 UTC m=+0.706660338 container died 9df711c82c699bc88dbee90cfb675a59550951b6dbd181cee2a87af54934f113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-21T08:58:17.385 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:17.338+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:17.386 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:17.338+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:17.715 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:17.541+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:17.715 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:17 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:17.541+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:17.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.5.service' 2024-04-21T08:58:17.772 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:17.772 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-21T08:58:17.773 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-21T08:58:17.773 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.6 2024-04-21T08:58:18.069 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:17.935+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:18.070 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:17.935+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:18.070 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:17 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:17.935+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:18.070 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:17 smithi057 systemd[1]: Stopping Ceph osd.6 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:18.070 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:18.069+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:18.070 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:18.069+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:18.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:17.831+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:18.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:17.831+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:18.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:17.877+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:18.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:17.877+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:18.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:17 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:17.877+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:18.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:18.122+0000 7f3627f9a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:18.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:18.122+0000 7f3627f9a700 -1 osd.6 114 *** Got signal Terminated *** 2024-04-21T08:58:18.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:18.122+0000 7f3627f9a700 -1 osd.6 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:18.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:18 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:18.498+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:18.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:18 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:18.498+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:18.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:18 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:18.300+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:18.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:18 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:18.300+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:19.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:18.812+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:19.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:18.812+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:19.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:18.889+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:19.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:18.889+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:19.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:18 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:18.889+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:19.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:19.064+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:19.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:19.064+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:19.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:18.896+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:19.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:18.896+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:19.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:18.896+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:19.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:19.458+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:19.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:19.458+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:19.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:19.315+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:19.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:19 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:19.315+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:20.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:19.937+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:20.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:19.937+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:20.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:19.937+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:20.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:19.785+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:20.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:19.785+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:20.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:19 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:19.785+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:20.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:20.099+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:20.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:20.099+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:20.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:20.099+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.254385+0000 front 2024-04-21T08:57:56.254351+0000 (oldest deadline 2024-04-21T08:58:19.754494+0000) 2024-04-21T08:58:20.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:19.859+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:20.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:19.859+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:20.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:19.859+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:20.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:19 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:19.859+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:20.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:20.482+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:20.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:20.482+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:20.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:20.364+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:20.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:20 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:20.364+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:21.073 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:20.915+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:21.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:20.915+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:21.074 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:20.915+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:21.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:20.789+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:21.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:20.789+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:21.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:20 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:20.789+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:21.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:21.094+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:21.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:21.094+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:21.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:21.094+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.254385+0000 front 2024-04-21T08:57:56.254351+0000 (oldest deadline 2024-04-21T08:58:19.754494+0000) 2024-04-21T08:58:21.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:20.847+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:21.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:20.847+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:21.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:20.847+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:21.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:20 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:20.847+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:21.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:21.445+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:21.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:21.445+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:21.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:21.445+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:21.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:21.321+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:21.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:21.321+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:21.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:21 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:21.321+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:22.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:21.818+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:22.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:21.818+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:22.074 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:21.818+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:22.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:21.894+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:22.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:21.894+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:22.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:21.894+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:22.075 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:21 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:21.894+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:22.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:22.048+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:22.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:22.048+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:22.340 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:22.048+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.254385+0000 front 2024-04-21T08:57:56.254351+0000 (oldest deadline 2024-04-21T08:58:19.754494+0000) 2024-04-21T08:58:22.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:21.840+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:22.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:21.840+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:22.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:21.840+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:22.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:21 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:21.840+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:22.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:22.491+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:22.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:22.491+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:22.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:22.491+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:22.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:22.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:22.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:22.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:22.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:22 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:22.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:23.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:22.861+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:23.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:22.861+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:23.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:22.861+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:23.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:22.911+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:23.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:22.911+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:23.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:22.911+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:23.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:22.911+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:23.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:23.061+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.851018+0000 front 2024-04-21T08:57:38.851050+0000 (oldest deadline 2024-04-21T08:58:02.951074+0000) 2024-04-21T08:58:23.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:23.061+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.453326+0000 front 2024-04-21T08:57:47.453361+0000 (oldest deadline 2024-04-21T08:58:12.153392+0000) 2024-04-21T08:58:23.339 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6[56435]: 2024-04-21T08:58:23.061+0000 7f3620586700 -1 osd.6 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.254385+0000 front 2024-04-21T08:57:56.254351+0000 (oldest deadline 2024-04-21T08:58:19.754494+0000) 2024-04-21T08:58:23.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:22.848+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:23.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:22.848+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:23.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:22.848+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:23.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:22 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:22.848+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:23.506 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:23.256+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:23.506 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:23.256+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:23.507 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:23.256+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:23.839 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:23 smithi057 podman[108810]: 2024-04-21 08:58:23.559173967 +0000 UTC m=+5.587287406 container died 056ebc46522a9249c790afe6374d00213ac2274b611fb27fe8f75701f90d73c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-21T08:58:23.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:23.506+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:23.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:23.506+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:23.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:23 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:23.506+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:24.224 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:23 smithi057 podman[108810]: 2024-04-21 08:58:23.931767062 +0000 UTC m=+5.959880484 container cleanup 056ebc46522a9249c790afe6374d00213ac2274b611fb27fe8f75701f90d73c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-21T08:58:24.224 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:23 smithi057 bash[108810]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6 2024-04-21T08:58:24.224 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:23.877+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:24.224 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:23.877+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:24.225 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:23.877+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:24.225 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:23 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:23.877+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:24.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:23.908+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:24.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:23.908+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:24.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:23.908+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:24.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:23.956+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:24.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:23.956+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:24.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:23.956+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:24.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:23 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:23.956+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:24.539 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:24.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:24.539 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:24.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:24.539 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:24.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:24.542 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:24 smithi057 podman[108825]: 2024-04-21 08:58:24.224898306 +0000 UTC m=+0.660037883 container remove 056ebc46522a9249c790afe6374d00213ac2274b611fb27fe8f75701f90d73c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T08:58:24.794 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:24 smithi057 podman[108916]: 2024-04-21 08:58:24.44650146 +0000 UTC m=+0.018512976 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:24.795 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:24 smithi057 podman[108916]: 2024-04-21 08:58:24.550964986 +0000 UTC m=+0.122976547 container create 15b170f032fb7fdf46f874bfad1569ead459cb78d1ed2eac847893c4fe4afe0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:58:24.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:24.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:24.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:24.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:24.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:24 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:24.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:25.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:24 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:24.887+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:25.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:24 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:24.887+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:25.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:24 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:24.887+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:25.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:24 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:24.887+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:25.090 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:24 smithi057 podman[108916]: 2024-04-21 08:58:24.795668696 +0000 UTC m=+0.367680212 container init 15b170f032fb7fdf46f874bfad1569ead459cb78d1ed2eac847893c4fe4afe0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-21T08:58:25.091 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:24 smithi057 podman[108916]: 2024-04-21 08:58:24.798905387 +0000 UTC m=+0.370916909 container start 15b170f032fb7fdf46f874bfad1569ead459cb78d1ed2eac847893c4fe4afe0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:58:25.091 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:24 smithi057 podman[108916]: 2024-04-21 08:58:24.879022139 +0000 UTC m=+0.451033673 container attach 15b170f032fb7fdf46f874bfad1569ead459cb78d1ed2eac847893c4fe4afe0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:25.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:24.908+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:25.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:24.908+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:25.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:24.908+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:25.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:24.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:25.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:24.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:25.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:24.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:25.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:24 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:24.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:25.473 INFO:journalctl@ceph.osd.6.smithi057.stdout:Apr 21 08:58:25 smithi057 podman[108916]: 2024-04-21 08:58:25.111776097 +0000 UTC m=+0.683787609 container died 15b170f032fb7fdf46f874bfad1569ead459cb78d1ed2eac847893c4fe4afe0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True) 2024-04-21T08:58:25.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:25 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:25.507+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:25.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:25 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:25.507+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:25.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:25 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:25.507+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:25.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:25 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:25.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:25.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:25 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:25.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:25.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:25 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:25.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:25.811 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.6.service' 2024-04-21T08:58:25.852 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:25.852 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-21T08:58:25.853 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-21T08:58:25.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.7 2024-04-21T08:58:26.089 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:25.929+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:26.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:25.929+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:26.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:25.929+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:26.090 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:25 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:25.929+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:26.159 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:26.012+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:26.160 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:26.012+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:26.160 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:26.012+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:26.160 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:26.012+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:26.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:25 smithi129 systemd[1]: Stopping Ceph osd.7 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:26.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:25 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:25.945+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:26.160 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:25 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:25.945+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:26.161 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:25 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:25.945+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:26.161 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:26.160+0000 7f1108921700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:26.161 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:26.160+0000 7f1108921700 -1 osd.7 114 *** Got signal Terminated *** 2024-04-21T08:58:26.161 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:26.160+0000 7f1108921700 -1 osd.7 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:26.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:26 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:26.488+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:26.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:26 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:26.488+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:26.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:26 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:26.488+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:26.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:26 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:26.369+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:26.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:26 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:26.369+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:26.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:26 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:26.369+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:27.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:26.986+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:27.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:26.986+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:27.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:26 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:26.986+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:27.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:27.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:27.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:27.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:27.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:26 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:26.938+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:27.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:26 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:26.938+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:27.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:26 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:26.938+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:27.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:26 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:26.938+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:27.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:27 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:27.441+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:27.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:27 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:27.441+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:27.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:27 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:27.441+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:27.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:27 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:27.390+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:27.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:27 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:27.390+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:27.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:27 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:27.390+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:28.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:27.954+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:28.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:27.954+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:28.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:27.954+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:28.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:27.954+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:05.207150+0000 front 2024-04-21T08:58:05.207117+0000 (oldest deadline 2024-04-21T08:58:27.507253+0000) 2024-04-21T08:58:28.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.967+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:28.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.967+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:28.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.967+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:28.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.967+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:28.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:27.967+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:28.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:27 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:27.910+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:28.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:27 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:27.910+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:28.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:27 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:27.910+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:28.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:27 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:27.910+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:28.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:28 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:28.404+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:28.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:28 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:28.404+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:28.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:28 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:28.404+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:28.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:28 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:28.404+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:28.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:28 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:28.404+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:28.886 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:28 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:28.404+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:29.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:28.998+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:29.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:28.998+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:29.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:28.998+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:29.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:28.998+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:05.207150+0000 front 2024-04-21T08:58:05.207117+0000 (oldest deadline 2024-04-21T08:58:27.507253+0000) 2024-04-21T08:58:29.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:28.997+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:29.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:28.997+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:29.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:28.997+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:29.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:28.997+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:29.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:28 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:28.997+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:29.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:28.915+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:29.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:28.915+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:29.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:28.915+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:29.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:28 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:28.915+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:29.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:29.440+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:29.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:29.440+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:29.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:29.440+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:29.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:29.440+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:29.887 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:29.419+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:29.887 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:29.419+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:29.887 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:29.419+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:29.887 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:29 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:29.419+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:30.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:29.978+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:30.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:29.978+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:30.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:29.978+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:30.325 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:29.978+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:05.207150+0000 front 2024-04-21T08:58:05.207117+0000 (oldest deadline 2024-04-21T08:58:27.507253+0000) 2024-04-21T08:58:30.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:29.968+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:30.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:29.968+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:30.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:29.968+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:30.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:29.968+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:30.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:29 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:29.968+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:30.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:29 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:29.888+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:30.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:29 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:29.888+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:30.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:29 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:29.888+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:30.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:29 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:29.888+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:30.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:29 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:29.888+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:30.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:30.455+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:30.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:30.455+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:30.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:30.455+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:30.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:30.455+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:30.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:30.370+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:30.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:30.370+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:30.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:30.370+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:30.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:30 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:30.370+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:31.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:30.969+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:37.401885+0000 front 2024-04-21T08:57:37.402067+0000 (oldest deadline 2024-04-21T08:58:02.102001+0000) 2024-04-21T08:58:31.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:30.969+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:48.304541+0000 front 2024-04-21T08:57:48.304585+0000 (oldest deadline 2024-04-21T08:58:10.505108+0000) 2024-04-21T08:58:31.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:30.969+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.606143+0000 front 2024-04-21T08:57:54.606238+0000 (oldest deadline 2024-04-21T08:58:19.306226+0000) 2024-04-21T08:58:31.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7[49219]: 2024-04-21T08:58:30.969+0000 7f1104f15700 -1 osd.7 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:05.207150+0000 front 2024-04-21T08:58:05.207117+0000 (oldest deadline 2024-04-21T08:58:27.507253+0000) 2024-04-21T08:58:31.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:30.966+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:31.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:30.966+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:31.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:30.966+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:31.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:30.966+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:31.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:30 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:30.966+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:31.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:30 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:30.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:31.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:30 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:30.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:31.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:30 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:30.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:31.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:30 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:30.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:31.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:30 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:30.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:31.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:31.478+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:31.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:31.478+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:31.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:31.478+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:31.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:31.478+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:31.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:31.364+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:31.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:31.364+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:31.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:31.364+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:31.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:31 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:31.364+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:31.893 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:31 smithi129 podman[67279]: 2024-04-21 08:58:31.632447505 +0000 UTC m=+5.591896185 container died cc7504d19fd8f7099c68c3a28fde858d89274a4cb1790ff3dd67b58133f848d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:31 smithi129 podman[67279]: 2024-04-21 08:58:31.993503158 +0000 UTC m=+5.952951820 container cleanup cc7504d19fd8f7099c68c3a28fde858d89274a4cb1790ff3dd67b58133f848d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:31 smithi129 bash[67279]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:31 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:31.977+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:31 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:31.977+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:31 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:31.977+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:31 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:31.977+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:32.269 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:31 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:31.977+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:32.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:31 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:31.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:32.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:31 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:31.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:32.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:31 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:31.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:32.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:31 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:31.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:32.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:31 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:31.893+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:32.573 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:32 smithi129 podman[67296]: 2024-04-21 08:58:32.269705849 +0000 UTC m=+0.639842175 container remove cc7504d19fd8f7099c68c3a28fde858d89274a4cb1790ff3dd67b58133f848d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-21T08:58:32.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:32.521+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:32.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:32.521+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:32.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:32.521+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:32.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:32.521+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:32.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:32.315+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:32.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:32.315+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:32.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:32.315+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:32.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:32 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:32.315+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:32.973 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:32 smithi129 podman[67390]: 2024-04-21 08:58:32.5130125 +0000 UTC m=+0.017914688 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:32.973 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:32 smithi129 podman[67390]: 2024-04-21 08:58:32.612198757 +0000 UTC m=+0.117100945 container create 7d71ddba23c2b2a53476070a202165d33091a7ecfd4b2f35ca0110b0106bd5a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T08:58:32.974 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:32 smithi129 podman[67390]: 2024-04-21 08:58:32.848913328 +0000 UTC m=+0.353815516 container init 7d71ddba23c2b2a53476070a202165d33091a7ecfd4b2f35ca0110b0106bd5a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416) 2024-04-21T08:58:32.974 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:32 smithi129 podman[67390]: 2024-04-21 08:58:32.851953376 +0000 UTC m=+0.356855570 container start 7d71ddba23c2b2a53476070a202165d33091a7ecfd4b2f35ca0110b0106bd5a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:58:33.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:32 smithi129 podman[67390]: 2024-04-21 08:58:32.974416137 +0000 UTC m=+0.479318342 container attach 7d71ddba23c2b2a53476070a202165d33091a7ecfd4b2f35ca0110b0106bd5a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:33.324 INFO:journalctl@ceph.osd.7.smithi129.stdout:Apr 21 08:58:33 smithi129 podman[67390]: 2024-04-21 08:58:33.142413952 +0000 UTC m=+0.647316152 container died 7d71ddba23c2b2a53476070a202165d33091a7ecfd4b2f35ca0110b0106bd5a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:58:33.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:33 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:33.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:33.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:33 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:33.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:33.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:33 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:33.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:33.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:33 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:33.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:33.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:33 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:33.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:33.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:32.901+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:33.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:32.901+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:33.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:32.901+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:33.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:32.901+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:33.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:32 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:32.901+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:33.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:33.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:33.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:33.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:33.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:33.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:33.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:33.539+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:33.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:33.362+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:33.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:33.362+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:33.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:33.362+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:33.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:33 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:33.362+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:33.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.7.service' 2024-04-21T08:58:33.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:33.826 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-21T08:58:33.826 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-21T08:58:33.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.8 2024-04-21T08:58:34.134 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:33 smithi136 systemd[1]: Stopping Ceph osd.8 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:34.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:34.004+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:34.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:34.004+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:34.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:34.004+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:34.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:34.004+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:34.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:34 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:34.004+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:34.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:33 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:33.940+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:34.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:33 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:33.940+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:34.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:33 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:33.940+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:34.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:33 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:33.940+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:34.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:33 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:33.940+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:34.385 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.133+0000 7f65d91d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:34.385 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.133+0000 7f65d91d8700 -1 osd.8 114 *** Got signal Terminated *** 2024-04-21T08:58:34.386 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.133+0000 7f65d91d8700 -1 osd.8 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:34.386 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:34.355+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:34.386 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:34.355+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:34.386 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:34.355+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:34.386 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:34.355+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:34.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.573+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:34.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.573+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:34.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.573+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:34.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:34 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:34.573+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:35.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.008+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:35.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.008+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:35.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.008+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:35.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.008+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:35.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.008+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:35.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:34 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:34.979+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:35.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:34 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:34.979+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:35.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:34 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:34.979+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:35.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:34 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:34.979+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:35.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:34 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:34.979+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:35.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:35.580+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:35.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:35.580+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:35.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:35.580+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:35.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:35.580+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:35.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:35.580+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:11.454407+0000 front 2024-04-21T08:58:11.454437+0000 (oldest deadline 2024-04-21T08:58:35.554542+0000) 2024-04-21T08:58:35.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:35.350+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:35.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:35.350+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:35.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:35.350+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:35.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:35 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:35.350+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:36.241 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:35 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:35.966+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:36.241 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:35 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:35.966+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:36.241 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:35 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:35.966+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:36.241 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:35 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:35.966+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:36.242 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:35 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:35.966+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:36.242 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:35 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:35.966+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:36.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:36.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:36.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:36.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:36.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:35 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:35.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:36.635 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:36.548+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:36.548+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:36.548+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:36.548+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:36.548+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:11.454407+0000 front 2024-04-21T08:58:11.454437+0000 (oldest deadline 2024-04-21T08:58:35.554542+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:36.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:36.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:36.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:36.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:36.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:36 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:36.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:37.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:36.978+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:37.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:36.978+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:37.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:36.978+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:37.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:36.978+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:37.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:36.978+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:37.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:36 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:36.941+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:37.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:36 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:36.941+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:37.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:36 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:36.941+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:37.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:36 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:36.941+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:37.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:36 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:36.941+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:37.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:36 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:36.941+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:37.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:37.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:37.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:37.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:37.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:37.279+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:37.512+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:37.512+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:37.512+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:37.512+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:37.636 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:37 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:37.512+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:11.454407+0000 front 2024-04-21T08:58:11.454437+0000 (oldest deadline 2024-04-21T08:58:35.554542+0000) 2024-04-21T08:58:38.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:37 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:37.975+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:38.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:37 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:37.975+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:38.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:37 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:37.975+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:38.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:37 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:37.975+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:38.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:37 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:37.975+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:38.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:37.968+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:38.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:37.968+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:38.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:37.968+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:38.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:37.968+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:38.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:37.968+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:38.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:37 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:37.968+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:38.538 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:38.230+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:38.538 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:38.230+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:38.538 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:38.230+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:38.538 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:38.230+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:38.538 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:38.230+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:38.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:38.537+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:40.946265+0000 front 2024-04-21T08:57:40.946289+0000 (oldest deadline 2024-04-21T08:58:06.846407+0000) 2024-04-21T08:58:38.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:38.537+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:47.947495+0000 front 2024-04-21T08:57:47.947450+0000 (oldest deadline 2024-04-21T08:58:13.148194+0000) 2024-04-21T08:58:38.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:38.537+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:56.550155+0000 front 2024-04-21T08:57:56.550075+0000 (oldest deadline 2024-04-21T08:58:21.249948+0000) 2024-04-21T08:58:38.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:38.537+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:04.552496+0000 front 2024-04-21T08:58:04.552574+0000 (oldest deadline 2024-04-21T08:58:28.652477+0000) 2024-04-21T08:58:38.886 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:38 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8[49990]: 2024-04-21T08:58:38.537+0000 7f65d17c4700 -1 osd.8 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:11.454407+0000 front 2024-04-21T08:58:11.454437+0000 (oldest deadline 2024-04-21T08:58:35.554542+0000) 2024-04-21T08:58:39.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:39 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:39.021+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:39.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:39 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:39.021+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:39.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:39 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:39.021+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:39.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:39 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:39.021+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:39.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:39 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:39.021+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:39.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:39 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:39.021+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:39.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.004+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:39.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.004+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:39.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.004+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:39.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.004+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:39.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.004+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:39.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.004+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:39.595 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:39.276+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:39.595 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:39.276+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:39.595 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:39.276+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:39.595 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:39.276+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:39.596 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:39.276+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:39.876 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:39 smithi136 podman[68636]: 2024-04-21 08:58:39.595067874 +0000 UTC m=+5.572960733 container died b68aead6415217a36556ed916b60004c2012f94c1dab5abde0a7eb209bd55e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True) 2024-04-21T08:58:40.135 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:39 smithi136 podman[68636]: 2024-04-21 08:58:39.967470199 +0000 UTC m=+5.945363066 container cleanup b68aead6415217a36556ed916b60004c2012f94c1dab5abde0a7eb209bd55e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:40.136 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:39 smithi136 bash[68636]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8 2024-04-21T08:58:40.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:40 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:40.027+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:40.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:40 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:40.027+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:40.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:40 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:40.027+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:40.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:40 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:40.027+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:40.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:40 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:40.027+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:40.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:40 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:40.027+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:40.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.992+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:40.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.992+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:40.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.992+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:40.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.992+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:40.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.992+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:40.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:39 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:39.992+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:40.626 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:40 smithi136 podman[68654]: 2024-04-21 08:58:40.277354835 +0000 UTC m=+0.683096274 container remove b68aead6415217a36556ed916b60004c2012f94c1dab5abde0a7eb209bd55e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:40.626 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:40 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:40.264+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:40.626 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:40 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:40.264+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:40.627 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:40 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:40.264+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:40.627 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:40 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:40.264+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:40.627 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:40 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:40.264+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:40.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:40 smithi136 podman[68749]: 2024-04-21 08:58:40.529420893 +0000 UTC m=+0.017658100 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:40.885 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:40 smithi136 podman[68749]: 2024-04-21 08:58:40.682314832 +0000 UTC m=+0.170552076 container create 2303afd4b3a29a0d5596e98d88f7f37b61f142008fb91118bcfd61dab9725d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-21T08:58:41.237 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:40 smithi136 podman[68749]: 2024-04-21 08:58:40.939281712 +0000 UTC m=+0.427518913 container init 2303afd4b3a29a0d5596e98d88f7f37b61f142008fb91118bcfd61dab9725d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:58:41.237 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:40 smithi136 podman[68749]: 2024-04-21 08:58:40.942358816 +0000 UTC m=+0.430596015 container start 2303afd4b3a29a0d5596e98d88f7f37b61f142008fb91118bcfd61dab9725d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-21T08:58:41.237 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:41 smithi136 podman[68749]: 2024-04-21 08:58:41.047636099 +0000 UTC m=+0.535873324 container attach 2303afd4b3a29a0d5596e98d88f7f37b61f142008fb91118bcfd61dab9725d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-21T08:58:41.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:41 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:41.034+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:41.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:41 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:41.034+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:41.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:41 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:41.034+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:41.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:41 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:41.034+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:41.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:41 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:41.034+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:41.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:41 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:41.034+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:41.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:41.023+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:41.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:41.023+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:41.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:41.023+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:41.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:41.023+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:41.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:41.023+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:41.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:41.023+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:41.558 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 21 08:58:41 smithi136 podman[68749]: 2024-04-21 08:58:41.238047701 +0000 UTC m=+0.726284893 container died 2303afd4b3a29a0d5596e98d88f7f37b61f142008fb91118bcfd61dab9725d34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-21T08:58:41.559 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:41.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:41.559 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:41.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:41.559 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:41.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:41.559 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:41.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:41.559 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:41.284+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:41.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.8.service' 2024-04-21T08:58:41.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:41.949 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-21T08:58:41.949 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-21T08:58:41.950 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.9 2024-04-21T08:58:42.252 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:41 smithi057 systemd[1]: Stopping Ceph osd.9 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:42.252 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.037+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:42.252 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.037+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:42.252 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.037+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:42.252 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.037+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:42.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.037+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:42.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.037+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:42.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.251+0000 7f933b5d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:42.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.251+0000 7f933b5d7700 -1 osd.9 114 *** Got signal Terminated *** 2024-04-21T08:58:42.253 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.251+0000 7f933b5d7700 -1 osd.9 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:42.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:42 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:42.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:42.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:42 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:42.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:42.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:42 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:42.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:42.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:42 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:42.006+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:42.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:42 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:42.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:42.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:42 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:42.007+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:42.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:42 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:42.277+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:42.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:42 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:42.277+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:42.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:42 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:42.277+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:42.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:42 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:42.277+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:42.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:42 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:42.277+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:43.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:43 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:43.003+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:43.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:43 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:43.003+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:43.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:43 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:43.003+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:43.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:43 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:43.003+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:43.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:43 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:43.003+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:43.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:43 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:43.003+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:43.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.996+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:43.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.996+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:43.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.996+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:43.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.996+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:43.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.996+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:43.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:42 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:42.996+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:43.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:43 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:43.282+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:43.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:43 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:43.282+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:43.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:43 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:43.282+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:43.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:43 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:43.282+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:43.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:43 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:43.282+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:44.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.031+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:44.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.031+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:44.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.031+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:44.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.031+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:44.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.031+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:44.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.031+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:44.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:44.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:44.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:44.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:44.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:44.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:44.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:44.298+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:44.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:44.298+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:44.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:44.298+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:44.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:44.298+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:44.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:44 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:44.298+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:45.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:45.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:45.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:45.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:45.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:45.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:45.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:44 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:44.995+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:45.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.987+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:45.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.987+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:45.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.987+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:45.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.987+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:45.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.987+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:45.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:44 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:44.987+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:45.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:45 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:45.294+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:45.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:45 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:45.294+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:45.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:45 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:45.294+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:45.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:45 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:45.294+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:45.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:45 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:45.294+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:45.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:45 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:45.294+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:46.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:46.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:46.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:46.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:46.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:46.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:46.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:45 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:45.985+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:46.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:46.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:46.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:46.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:46.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:46.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.020+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:46.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:46 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:46.330+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:46.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:46 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:46.330+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:46.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:46 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:46.330+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:46.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:46 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:46.330+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:46.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:46 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:46.330+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:46.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:46 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:46.330+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:47.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:46 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:46.969+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:47.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.974+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.243240+0000 front 2024-04-21T08:57:32.243329+0000 (oldest deadline 2024-04-21T08:57:53.943401+0000) 2024-04-21T08:58:47.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.974+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:38.044551+0000 front 2024-04-21T08:57:38.044428+0000 (oldest deadline 2024-04-21T08:58:03.344505+0000) 2024-04-21T08:58:47.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.974+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.245958+0000 front 2024-04-21T08:57:49.245877+0000 (oldest deadline 2024-04-21T08:58:14.446719+0000) 2024-04-21T08:58:47.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.974+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:54.447335+0000 front 2024-04-21T08:57:54.447364+0000 (oldest deadline 2024-04-21T08:58:19.747501+0000) 2024-04-21T08:58:47.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.974+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:03.849179+0000 front 2024-04-21T08:58:03.849201+0000 (oldest deadline 2024-04-21T08:58:29.749210+0000) 2024-04-21T08:58:47.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:46 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9[63446]: 2024-04-21T08:58:46.974+0000 7f9333bc3700 -1 osd.9 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.749777+0000 front 2024-04-21T08:58:09.749897+0000 (oldest deadline 2024-04-21T08:58:35.649747+0000) 2024-04-21T08:58:47.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:47.307+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:47.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:47.307+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:47.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:47.307+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:47.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:47.307+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:47.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:47.307+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:47.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:47.307+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:47.839 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:47 smithi057 podman[109033]: 2024-04-21 08:58:47.715559755 +0000 UTC m=+5.566646935 container died 108cc624b3d98fbb40e099c6f682b5cba75f49e923ee6c3d00acdd9b0670290d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:48.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:47 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:47.961+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:48.339 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:48 smithi057 podman[109033]: 2024-04-21 08:58:48.044036471 +0000 UTC m=+5.895123659 container cleanup 108cc624b3d98fbb40e099c6f682b5cba75f49e923ee6c3d00acdd9b0670290d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-21T08:58:48.340 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:48 smithi057 bash[109033]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9 2024-04-21T08:58:48.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:48 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:48.274+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:48.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:48 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:48.274+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:48.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:48 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:48.274+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:48.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:48 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:48.274+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:48.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:48 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:48.274+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:48.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:48 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:48.274+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:48.828 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:48 smithi057 podman[109051]: 2024-04-21 08:58:48.395066141 +0000 UTC m=+0.679493992 container remove 108cc624b3d98fbb40e099c6f682b5cba75f49e923ee6c3d00acdd9b0670290d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:49.089 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:48 smithi057 podman[109143]: 2024-04-21 08:58:48.732510736 +0000 UTC m=+0.018687214 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:49.089 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:48 smithi057 podman[109143]: 2024-04-21 08:58:48.837623996 +0000 UTC m=+0.123800464 container create f0d42f259b19ca4e1e90822df40ce6cf1da8f2dfe577686649c32eb1a0f9a07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:49.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:49 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:49.005+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:49.366 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:49 smithi057 podman[109143]: 2024-04-21 08:58:49.090311761 +0000 UTC m=+0.376488229 container init f0d42f259b19ca4e1e90822df40ce6cf1da8f2dfe577686649c32eb1a0f9a07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-21T08:58:49.366 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:49 smithi057 podman[109143]: 2024-04-21 08:58:49.094372655 +0000 UTC m=+0.380549133 container start f0d42f259b19ca4e1e90822df40ce6cf1da8f2dfe577686649c32eb1a0f9a07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-21T08:58:49.366 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:49 smithi057 podman[109143]: 2024-04-21 08:58:49.165188944 +0000 UTC m=+0.451365431 container attach f0d42f259b19ca4e1e90822df40ce6cf1da8f2dfe577686649c32eb1a0f9a07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T08:58:49.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:49 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:49.262+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:49.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:49 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:49.262+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:49.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:49 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:49.262+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:49.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:49 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:49.262+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:49.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:49 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:49.262+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:49.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:49 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:49.262+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:49.709 INFO:journalctl@ceph.osd.9.smithi057.stdout:Apr 21 08:58:49 smithi057 podman[109143]: 2024-04-21 08:58:49.418059959 +0000 UTC m=+0.704236421 container died f0d42f259b19ca4e1e90822df40ce6cf1da8f2dfe577686649c32eb1a0f9a07e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5) 2024-04-21T08:58:49.986 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.9.service' 2024-04-21T08:58:50.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:50.025 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-21T08:58:50.025 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-21T08:58:50.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.10 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:50.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.053+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:50.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 systemd[1]: Stopping Ceph osd.10 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:50.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:50.293+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:50.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:50.293+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:50.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:50.293+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:50.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:50.293+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:50.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:50.293+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:50.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:50.293+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:50.823 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.355+0000 7fdff1e1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:50.824 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.355+0000 7fdff1e1c700 -1 osd.10 114 *** Got signal Terminated *** 2024-04-21T08:58:50.824 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:50 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:50.355+0000 7fdff1e1c700 -1 osd.10 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:51.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:51.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:51.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:51.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:51.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:51.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:51.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:51 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:51.043+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:51.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:51 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:51.326+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:51.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:51 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:51.326+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:51.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:51 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:51.326+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:51.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:51 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:51.326+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:51.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:51 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:51.326+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:51.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:51 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:51.326+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:52.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.010+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:52.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:52 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:52.325+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:52.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:52 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:52.325+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:52.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:52 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:52.325+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:52.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:52 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:52.325+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:52.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:52 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:52.325+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:52.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:52 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:52.325+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:53.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:53.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:53.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:53.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:53.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:53.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:53.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:52 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:52.998+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:53.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:53.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:53.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:53.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:53.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:53.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:53.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:53 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:53.346+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:58:54.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:54.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:54.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:54.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:54.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:54.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:54.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:54 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:53.999+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:54.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:54.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:54.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:54.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:54.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:54.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:54.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:54 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:54.314+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:58:55.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:32.644557+0000 front 2024-04-21T08:57:32.644571+0000 (oldest deadline 2024-04-21T08:57:56.144473+0000) 2024-04-21T08:58:55.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6804 osd.1 since back 2024-04-21T08:57:36.144821+0000 front 2024-04-21T08:57:36.144803+0000 (oldest deadline 2024-04-21T08:58:02.045134+0000) 2024-04-21T08:58:55.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:44.946285+0000 front 2024-04-21T08:57:44.946297+0000 (oldest deadline 2024-04-21T08:58:13.746833+0000) 2024-04-21T08:58:55.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:53.747555+0000 front 2024-04-21T08:57:53.747417+0000 (oldest deadline 2024-04-21T08:58:21.348021+0000) 2024-04-21T08:58:55.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:06.449548+0000 front 2024-04-21T08:58:06.449574+0000 (oldest deadline 2024-04-21T08:58:27.549746+0000) 2024-04-21T08:58:55.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:13.251868+0000 front 2024-04-21T08:58:13.251749+0000 (oldest deadline 2024-04-21T08:58:38.551632+0000) 2024-04-21T08:58:55.325 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10[55839]: 2024-04-21T08:58:55.019+0000 7fdfee410700 -1 osd.10 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:21.353192+0000 front 2024-04-21T08:58:21.353280+0000 (oldest deadline 2024-04-21T08:58:44.852942+0000) 2024-04-21T08:58:55.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:55.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:55.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:55.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:55.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:55.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:55.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:55 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:55.357+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:58:56.048 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:55 smithi129 podman[67507]: 2024-04-21 08:58:55.769285698 +0000 UTC m=+5.545237356 container died 826f930b8a733196b79d90c0c016d7bd1768d7cccd9b25cac4295d2d945d018d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-21T08:58:56.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 podman[67507]: 2024-04-21 08:58:56.148250477 +0000 UTC m=+5.924202132 container cleanup 826f930b8a733196b79d90c0c016d7bd1768d7cccd9b25cac4295d2d945d018d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True) 2024-04-21T08:58:56.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 bash[67507]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10 2024-04-21T08:58:56.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:56.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:56.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:56.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:56.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:56.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:56.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:56 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:56.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:58:56.746 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 podman[67524]: 2024-04-21 08:58:56.441043544 +0000 UTC m=+0.668077699 container remove 826f930b8a733196b79d90c0c016d7bd1768d7cccd9b25cac4295d2d945d018d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-21T08:58:56.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 podman[67614]: 2024-04-21 08:58:56.650414468 +0000 UTC m=+0.017815328 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:58:56.864 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 podman[67614]: 2024-04-21 08:58:56.749984844 +0000 UTC m=+0.117385749 container create 0213fbb8d22875a871115695785e7a8e09e6134f61823091927d6c899186466d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-04-21T08:58:57.059 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 podman[67614]: 2024-04-21 08:58:56.986123539 +0000 UTC m=+0.353524397 container init 0213fbb8d22875a871115695785e7a8e09e6134f61823091927d6c899186466d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:58:57.060 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:56 smithi129 podman[67614]: 2024-04-21 08:58:56.989473033 +0000 UTC m=+0.356873882 container start 0213fbb8d22875a871115695785e7a8e09e6134f61823091927d6c899186466d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-21T08:58:57.060 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:57 smithi129 podman[67614]: 2024-04-21 08:58:57.060670442 +0000 UTC m=+0.428071307 container attach 0213fbb8d22875a871115695785e7a8e09e6134f61823091927d6c899186466d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:57.324 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:57 smithi129 podman[67614]: 2024-04-21 08:58:57.280596733 +0000 UTC m=+0.647997579 container died 0213fbb8d22875a871115695785e7a8e09e6134f61823091927d6c899186466d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:58:57.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:57.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:57.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:57.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:57.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:57.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:57.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:57.313+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:58:57.813 INFO:journalctl@ceph.osd.10.smithi129.stdout:Apr 21 08:58:57 smithi129 podman[67640]: 2024-04-21 08:58:57.797554957 +0000 UTC m=+0.507825678 container remove 0213fbb8d22875a871115695785e7a8e09e6134f61823091927d6c899186466d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-21T08:58:57.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.10.service' 2024-04-21T08:58:57.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:58:57.854 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-21T08:58:57.854 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-21T08:58:57.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.11 2024-04-21T08:58:58.135 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:57 smithi136 systemd[1]: Stopping Ceph osd.11 for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:58:58.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.203+0000 7f83ed0ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.203+0000 7f83ed0ce700 -1 osd.11 114 *** Got signal Terminated *** 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.203+0000 7f83ed0ce700 -1 osd.11 114 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:58.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:58.637 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:58 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:58.306+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:58:59.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:58:59.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:58:59.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:58:59.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:58:59.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:58:59.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:58:59.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:58:59 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:58:59.344+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:59:00.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:59:00.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:59:00.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:59:00.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:59:00.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:59:00.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:59:00.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:00 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:00.333+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:59:01.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:59:01.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:59:01.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:59:01.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:59:01.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:59:01.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:59:01.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:01 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:01.371+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:59:02.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6806 osd.0 since back 2024-04-21T08:57:29.676199+0000 front 2024-04-21T08:57:29.676174+0000 (oldest deadline 2024-04-21T08:57:54.376298+0000) 2024-04-21T08:59:02.635 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-21T08:57:49.579228+0000 front 2024-04-21T08:57:49.579189+0000 (oldest deadline 2024-04-21T08:58:15.479412+0000) 2024-04-21T08:59:02.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6814 osd.3 since back 2024-04-21T08:57:57.681387+0000 front 2024-04-21T08:57:57.681457+0000 (oldest deadline 2024-04-21T08:58:21.181343+0000) 2024-04-21T08:59:02.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6812 osd.4 since back 2024-04-21T08:58:01.682539+0000 front 2024-04-21T08:58:01.682559+0000 (oldest deadline 2024-04-21T08:58:29.283085+0000) 2024-04-21T08:59:02.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-21T08:58:09.283506+0000 front 2024-04-21T08:58:09.283563+0000 (oldest deadline 2024-04-21T08:58:37.484407+0000) 2024-04-21T08:59:02.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.57:6822 osd.6 since back 2024-04-21T08:58:20.385687+0000 front 2024-04-21T08:58:20.385610+0000 (oldest deadline 2024-04-21T08:58:45.085506+0000) 2024-04-21T08:59:02.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.129:6820 osd.7 since back 2024-04-21T08:58:30.886907+0000 front 2024-04-21T08:58:30.886861+0000 (oldest deadline 2024-04-21T08:58:53.186856+0000) 2024-04-21T08:59:02.636 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:02 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11[56749]: 2024-04-21T08:59:02.368+0000 7f83e56ba700 -1 osd.11 114 heartbeat_check: no reply from 172.21.15.136:6820 osd.8 since back 2024-04-21T08:58:37.887985+0000 front 2024-04-21T08:58:37.887940+0000 (oldest deadline 2024-04-21T08:59:01.387979+0000) 2024-04-21T08:59:04.086 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:03 smithi136 podman[68854]: 2024-04-21 08:59:03.796331183 +0000 UTC m=+5.753046109 container died 0539d2f76ea9e8e4ff56256f03e279cd876ea222938577ac80cf1fd987fc5851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-21T08:59:04.385 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 podman[68854]: 2024-04-21 08:59:04.186570374 +0000 UTC m=+6.143285292 container cleanup 0539d2f76ea9e8e4ff56256f03e279cd876ea222938577ac80cf1fd987fc5851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-21T08:59:04.385 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 bash[68854]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11 2024-04-21T08:59:04.748 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 podman[68873]: 2024-04-21 08:59:04.425021695 +0000 UTC m=+0.632447847 container remove 0539d2f76ea9e8e4ff56256f03e279cd876ea222938577ac80cf1fd987fc5851 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T08:59:05.057 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 podman[68966]: 2024-04-21 08:59:04.651697977 +0000 UTC m=+0.017609556 image pull b45a994450e2a16d2cd2438b86c3e132853c7dd0b34f48fede9705dbcfa91192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda 2024-04-21T08:59:05.058 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 podman[68966]: 2024-04-21 08:59:04.763991411 +0000 UTC m=+0.129902975 container create fb1496b84aa7648d1be0bab92d08ad0551d0c2ce1136eeee73a0cebcfee0f71a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-21T08:59:05.058 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 podman[68966]: 2024-04-21 08:59:04.97534213 +0000 UTC m=+0.341253703 container init fb1496b84aa7648d1be0bab92d08ad0551d0c2ce1136eeee73a0cebcfee0f71a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:59:05.058 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:04 smithi136 podman[68966]: 2024-04-21 08:59:04.978621643 +0000 UTC m=+0.344533215 container start fb1496b84aa7648d1be0bab92d08ad0551d0c2ce1136eeee73a0cebcfee0f71a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS) 2024-04-21T08:59:05.058 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:05 smithi136 podman[68966]: 2024-04-21 08:59:05.058256194 +0000 UTC m=+0.424167769 container attach fb1496b84aa7648d1be0bab92d08ad0551d0c2ce1136eeee73a0cebcfee0f71a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:05.385 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:05 smithi136 podman[68966]: 2024-04-21 08:59:05.270053943 +0000 UTC m=+0.635965499 container died fb1496b84aa7648d1be0bab92d08ad0551d0c2ce1136eeee73a0cebcfee0f71a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2) 2024-04-21T08:59:05.804 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 21 08:59:05 smithi136 podman[68992]: 2024-04-21 08:59:05.786992271 +0000 UTC m=+0.508740734 container remove fb1496b84aa7648d1be0bab92d08ad0551d0c2ce1136eeee73a0cebcfee0f71a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:05.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@osd.11.service' 2024-04-21T08:59:05.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:05.847 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-21T08:59:05.847 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-21T08:59:05.847 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.a 2024-04-21T08:59:06.146 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:59:05 smithi057 systemd[1]: Stopping Ceph mds.a for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:06.589 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:59:06 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a[70423]: 2024-04-21T08:59:06.146+0000 7fae439a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:06.589 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:59:06 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a[70423]: 2024-04-21T08:59:06.146+0000 7fae439a9700 -1 mds.a *** got signal Terminated *** 2024-04-21T08:59:08.720 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:59:08 smithi057 podman[109262]: 2024-04-21 08:59:08.455026486 +0000 UTC m=+2.410582634 container died d9b3e4ca368b65bee0d2063f20f1739f79321bc3ee01d355e5a485138a835d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-21T08:59:09.063 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:59:08 smithi057 podman[109262]: 2024-04-21 08:59:08.812521021 +0000 UTC m=+2.768077168 container cleanup d9b3e4ca368b65bee0d2063f20f1739f79321bc3ee01d355e5a485138a835d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-21T08:59:09.063 INFO:journalctl@ceph.mds.a.smithi057.stdout:Apr 21 08:59:08 smithi057 bash[109262]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-a 2024-04-21T08:59:09.175 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.a.service' 2024-04-21T08:59:09.209 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:09.209 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-21T08:59:09.210 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-21T08:59:09.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.d 2024-04-21T08:59:09.339 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:59:09 smithi057 systemd[1]: Stopping Ceph mds.d for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:09.839 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:59:09 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d[69467]: 2024-04-21T08:59:09.558+0000 7f698c7b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:09.839 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:59:09 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d[69467]: 2024-04-21T08:59:09.558+0000 7f698c7b6700 -1 mds.d *** got signal Terminated *** 2024-04-21T08:59:12.550 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:59:12 smithi057 podman[109395]: 2024-04-21 08:59:12.269001589 +0000 UTC m=+2.835397499 container died daf41631cee289a1e40e2048697b2d14bd4d2836f184b6d4247fc888d6016fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-21T08:59:12.839 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:59:12 smithi057 podman[109395]: 2024-04-21 08:59:12.62500246 +0000 UTC m=+3.191398425 container cleanup daf41631cee289a1e40e2048697b2d14bd4d2836f184b6d4247fc888d6016fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:59:12.839 INFO:journalctl@ceph.mds.d.smithi057.stdout:Apr 21 08:59:12 smithi057 bash[109395]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-d 2024-04-21T08:59:13.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.d.service' 2024-04-21T08:59:13.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:13.093 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-21T08:59:13.093 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-21T08:59:13.093 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.g 2024-04-21T08:59:13.420 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:13 smithi057 systemd[1]: Stopping Ceph mds.g for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:13.839 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g[70074]: 2024-04-21T08:59:13.420+0000 7fc7e8409700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:13.839 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:13 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g[70074]: 2024-04-21T08:59:13.420+0000 7fc7e8409700 -1 mds.g *** got signal Terminated *** 2024-04-21T08:59:17.795 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:17 smithi057 podman[109518]: 2024-04-21 08:59:17.425561425 +0000 UTC m=+4.109269686 container died d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-21T08:59:18.089 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:17 smithi057 podman[109518]: 2024-04-21 08:59:17.87062534 +0000 UTC m=+4.554333601 container cleanup d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:18.089 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:17 smithi057 bash[109518]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g 2024-04-21T08:59:18.349 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.g.service' 2024-04-21T08:59:18.370 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:18 smithi057 podman[109532]: 2024-04-21 08:59:18.106566723 +0000 UTC m=+0.674075748 container remove d8db5d41b5b7acbcd34abfe33dc9297407dd9a12d5b30caeedb5ec3a0a46add3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, GIT_CLEAN=True, ceph=True) 2024-04-21T08:59:18.370 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:18 smithi057 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.g.service: Deactivated successfully. 2024-04-21T08:59:18.370 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:18 smithi057 systemd[1]: Stopped Ceph mds.g for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:59:18.370 INFO:journalctl@ceph.mds.g.smithi057.stdout:Apr 21 08:59:18 smithi057 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.g.service: Consumed 1.134s CPU time. 2024-04-21T08:59:18.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:18.382 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-21T08:59:18.382 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-21T08:59:18.382 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.j 2024-04-21T08:59:18.707 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:59:18 smithi057 systemd[1]: Stopping Ceph mds.j for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:19.089 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:59:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-j[69121]: 2024-04-21T08:59:18.706+0000 7f707bb7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:19.089 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:59:18 smithi057 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-j[69121]: 2024-04-21T08:59:18.706+0000 7f707bb7b700 -1 mds.j *** got signal Terminated *** 2024-04-21T08:59:21.549 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:59:21 smithi057 podman[109639]: 2024-04-21 08:59:21.286309227 +0000 UTC m=+2.682818931 container died 4519deced7b15e449f23594201949968b73393593cd2f8d1ac79d377c7f02fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-21T08:59:21.839 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:59:21 smithi057 podman[109639]: 2024-04-21 08:59:21.649496326 +0000 UTC m=+3.046006041 container cleanup 4519deced7b15e449f23594201949968b73393593cd2f8d1ac79d377c7f02fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:21.839 INFO:journalctl@ceph.mds.j.smithi057.stdout:Apr 21 08:59:21 smithi057 bash[109639]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-j 2024-04-21T08:59:22.084 DEBUG:teuthology.orchestra.run.smithi057:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.j.service' 2024-04-21T08:59:22.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:22.115 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-21T08:59:22.115 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-21T08:59:22.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.b 2024-04-21T08:59:22.399 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:59:22 smithi129 systemd[1]: Stopping Ceph mds.b for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:22.658 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:59:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: 2024-04-21T08:59:22.415+0000 7f8fbb38c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:22.658 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:59:22 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b[61093]: 2024-04-21T08:59:22.415+0000 7f8fbb38c700 -1 mds.b *** got signal Terminated *** 2024-04-21T08:59:26.433 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:59:26 smithi129 podman[67720]: 2024-04-21 08:59:26.164562956 +0000 UTC m=+3.860428704 container died 302607c3a1b7baed623e7dc6035112b31665926fa2ae5f72a1191d8dd74ba9f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:26.816 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:59:26 smithi129 podman[67720]: 2024-04-21 08:59:26.532311242 +0000 UTC m=+4.228176991 container cleanup 302607c3a1b7baed623e7dc6035112b31665926fa2ae5f72a1191d8dd74ba9f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-21T08:59:26.816 INFO:journalctl@ceph.mds.b.smithi129.stdout:Apr 21 08:59:26 smithi129 bash[67720]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-b 2024-04-21T08:59:26.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.b.service' 2024-04-21T08:59:27.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:27.012 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-21T08:59:27.012 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-21T08:59:27.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.e 2024-04-21T08:59:27.324 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:59:27 smithi129 systemd[1]: Stopping Ceph mds.e for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:27.823 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:59:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e[60475]: 2024-04-21T08:59:27.343+0000 7f353ec58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:27.824 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:59:27 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e[60475]: 2024-04-21T08:59:27.343+0000 7f353ec58700 -1 mds.e *** got signal Terminated *** 2024-04-21T08:59:31.520 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:59:31 smithi129 podman[67855]: 2024-04-21 08:59:31.111993007 +0000 UTC m=+3.880157272 container died 0317caf43338e5d935327b295f4223bc7fe4f5b06aa4519c56a058e3cf1cc127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-21T08:59:31.823 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:59:31 smithi129 podman[67855]: 2024-04-21 08:59:31.594154007 +0000 UTC m=+4.362318270 container cleanup 0317caf43338e5d935327b295f4223bc7fe4f5b06aa4519c56a058e3cf1cc127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-21T08:59:31.824 INFO:journalctl@ceph.mds.e.smithi129.stdout:Apr 21 08:59:31 smithi129 bash[67855]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-e 2024-04-21T08:59:32.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.e.service' 2024-04-21T08:59:32.056 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:32.056 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-21T08:59:32.057 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-21T08:59:32.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.h 2024-04-21T08:59:32.323 INFO:journalctl@ceph.mds.h.smithi129.stdout:Apr 21 08:59:32 smithi129 systemd[1]: Stopping Ceph mds.h for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:32.824 INFO:journalctl@ceph.mds.h.smithi129.stdout:Apr 21 08:59:32 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-h[60786]: 2024-04-21T08:59:32.380+0000 7f523e6c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:32.824 INFO:journalctl@ceph.mds.h.smithi129.stdout:Apr 21 08:59:32 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-h[60786]: 2024-04-21T08:59:32.380+0000 7f523e6c5700 -1 mds.h *** got signal Terminated *** 2024-04-21T08:59:35.973 INFO:journalctl@ceph.mds.h.smithi129.stdout:Apr 21 08:59:35 smithi129 podman[67976]: 2024-04-21 08:59:35.698061902 +0000 UTC m=+3.414220475 container died aa35b7b8ebba5881314afd49b0497d2d645781501918f9f9fbb85479f67d3f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-21T08:59:36.315 INFO:journalctl@ceph.mds.h.smithi129.stdout:Apr 21 08:59:36 smithi129 podman[67976]: 2024-04-21 08:59:36.056717313 +0000 UTC m=+3.772875887 container cleanup aa35b7b8ebba5881314afd49b0497d2d645781501918f9f9fbb85479f67d3f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-21T08:59:36.315 INFO:journalctl@ceph.mds.h.smithi129.stdout:Apr 21 08:59:36 smithi129 bash[67976]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-h 2024-04-21T08:59:36.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.h.service' 2024-04-21T08:59:36.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:36.463 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-21T08:59:36.463 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-21T08:59:36.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.k 2024-04-21T08:59:36.573 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:59:36 smithi129 systemd[1]: Stopping Ceph mds.k for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:37.073 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:59:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k[60155]: 2024-04-21T08:59:36.767+0000 7fd175be6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:37.074 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:59:36 smithi129 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k[60155]: 2024-04-21T08:59:36.767+0000 7fd175be6700 -1 mds.k *** got signal Terminated *** 2024-04-21T08:59:38.573 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:59:38 smithi129 podman[68095]: 2024-04-21 08:59:38.314989737 +0000 UTC m=+1.640001996 container died 2f532aef29f7e130a1cb000eba3a657b71a5d3a2b5cf16f2d5fa7364c7062118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:38.879 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:59:38 smithi129 podman[68095]: 2024-04-21 08:59:38.694181754 +0000 UTC m=+2.019194014 container cleanup 2f532aef29f7e130a1cb000eba3a657b71a5d3a2b5cf16f2d5fa7364c7062118 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiescer-a630765) 2024-04-21T08:59:38.879 INFO:journalctl@ceph.mds.k.smithi129.stdout:Apr 21 08:59:38 smithi129 bash[68095]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-k 2024-04-21T08:59:39.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.k.service' 2024-04-21T08:59:39.134 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:39.134 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-21T08:59:39.134 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-21T08:59:39.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.c 2024-04-21T08:59:39.452 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:59:39 smithi136 systemd[1]: Stopping Ceph mds.c for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:39.786 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:59:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c[61751]: 2024-04-21T08:59:39.451+0000 7fbfdd5bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:39.786 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:59:39 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c[61751]: 2024-04-21T08:59:39.451+0000 7fbfdd5bb700 -1 mds.c *** got signal Terminated *** 2024-04-21T08:59:40.885 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:59:40 smithi136 podman[69084]: 2024-04-21 08:59:40.651393476 +0000 UTC m=+1.323459940 container died ea94fe2f9854badfa6beace68724735db789dd260dadb89e969026d7e64c2d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-a630765, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-21T08:59:41.269 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:59:41 smithi136 podman[69084]: 2024-04-21 08:59:41.018744674 +0000 UTC m=+1.690811146 container cleanup ea94fe2f9854badfa6beace68724735db789dd260dadb89e969026d7e64c2d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c, RELEASE=wip-lusov-quiescer-a630765, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T08:59:41.269 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 21 08:59:41 smithi136 bash[69084]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-c 2024-04-21T08:59:41.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.c.service' 2024-04-21T08:59:41.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:41.467 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-21T08:59:41.467 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-21T08:59:41.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.f 2024-04-21T08:59:41.781 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:59:41 smithi136 systemd[1]: Stopping Ceph mds.f for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:42.135 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:59:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f[61443]: 2024-04-21T08:59:41.780+0000 7fa0fdc1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:42.135 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:59:41 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f[61443]: 2024-04-21T08:59:41.780+0000 7fa0fdc1a700 -1 mds.f *** got signal Terminated *** 2024-04-21T08:59:46.063 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:59:45 smithi136 podman[69216]: 2024-04-21 08:59:45.796531916 +0000 UTC m=+4.110964752 container died 445152d84157217d263c824dc90638efd7e4f251e452af109d952007b47a277b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-21T08:59:46.385 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:59:46 smithi136 podman[69216]: 2024-04-21 08:59:46.163680307 +0000 UTC m=+4.478113142 container cleanup 445152d84157217d263c824dc90638efd7e4f251e452af109d952007b47a277b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-21T08:59:46.385 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 21 08:59:46 smithi136 bash[69216]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-f 2024-04-21T08:59:46.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.f.service' 2024-04-21T08:59:46.683 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:46.683 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-21T08:59:46.684 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-21T08:59:46.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.i 2024-04-21T08:59:47.000 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:46 smithi136 systemd[1]: Stopping Ceph mds.i for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:47.385 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i[62065]: 2024-04-21T08:59:47.000+0000 7f7f3b537700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:47.385 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:47 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i[62065]: 2024-04-21T08:59:47.000+0000 7f7f3b537700 -1 mds.i *** got signal Terminated *** 2024-04-21T08:59:49.859 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:49 smithi136 podman[69336]: 2024-04-21 08:59:49.562846537 +0000 UTC m=+2.665313290 container died d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-a630765, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-21T08:59:50.135 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:49 smithi136 podman[69336]: 2024-04-21 08:59:49.959352768 +0000 UTC m=+3.061819518 container cleanup d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-21T08:59:50.135 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:49 smithi136 bash[69336]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i 2024-04-21T08:59:50.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.i.service' 2024-04-21T08:59:50.408 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:50 smithi136 podman[69350]: 2024-04-21 08:59:50.143767361 +0000 UTC m=+0.577475804 container remove d8780213deb4d10534588456c0e970da2f720634681bfe2eeed3d44fa309a0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-a630765, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-21T08:59:50.409 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:50 smithi136 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.i.service: Deactivated successfully. 2024-04-21T08:59:50.409 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:50 smithi136 systemd[1]: Stopped Ceph mds.i for df0fb3ae-ffb9-11ee-bc93-c7b262605968. 2024-04-21T08:59:50.409 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 21 08:59:50 smithi136 systemd[1]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.i.service: Consumed 42.736s CPU time. 2024-04-21T08:59:50.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:50.420 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-21T08:59:50.420 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-21T08:59:50.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.l 2024-04-21T08:59:50.745 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:59:50 smithi136 systemd[1]: Stopping Ceph mds.l for df0fb3ae-ffb9-11ee-bc93-c7b262605968... 2024-04-21T08:59:51.135 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:59:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l[61131]: 2024-04-21T08:59:50.745+0000 7f35aaf87700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-21T08:59:51.135 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:59:50 smithi136 ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l[61131]: 2024-04-21T08:59:50.745+0000 7f35aaf87700 -1 mds.l *** got signal Terminated *** 2024-04-21T08:59:55.038 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:59:54 smithi136 podman[69458]: 2024-04-21 08:59:54.75295144 +0000 UTC m=+4.113563197 container died e954924f42a4b4136ea08c2eb0bf6134f07966b1ad3ed293301e277e83de104d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-21T08:59:55.385 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:59:55 smithi136 podman[69458]: 2024-04-21 08:59:55.138369693 +0000 UTC m=+4.498981401 container cleanup e954924f42a4b4136ea08c2eb0bf6134f07966b1ad3ed293301e277e83de104d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:36ea816f637ab0eb3aba78ac1a5e79acaf4ee67536fe53155c0137fedf936fda, name=ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-a630765, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-21T08:59:55.385 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 21 08:59:55 smithi136 bash[69458]: ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968-mds-l 2024-04-21T08:59:55.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-df0fb3ae-ffb9-11ee-bc93-c7b262605968@mds.l.service' 2024-04-21T08:59:55.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-21T08:59:55.596 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-21T08:59:55.596 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T08:59:55.795 INFO:teuthology.orchestra.run.smithi057.stdout:Deleting cluster with fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T08:59:58.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T08:59:58.561 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:00:01.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --force --keep-logs 2024-04-21T09:00:01.335 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:00:03.628 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:00:03.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:00:03.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-21T09:00:03.718 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-21T09:00:03.720 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi057/crash 2024-04-21T09:00:03.721 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:00:03.749 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:00:03.750 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:00:03.751 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi129/crash 2024-04-21T09:00:03.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:00:03.790 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:00:03.790 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:00:03.792 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi136/crash 2024-04-21T09:00:03.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash -- . 2024-04-21T09:00:03.830 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-21T09:00:03.831 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-04-21T09:00:03.831 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-21T09:00:03.832 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-21T09:00:03.867 INFO:tasks.cephadm:Compressing logs... 2024-04-21T09:00:03.867 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:00:03.910 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:00:03.913 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-21T09:00:03.935 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:00:03.935 INFO:teuthology.orchestra.run.smithi057.stderr:: No such file or directory 2024-04-21T09:00:03.935 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:00:03.936 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log 2024-04-21T09:00:03.936 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log 2024-04-21T09:00:03.936 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:00:03.943 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log: 90.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:00:03.944 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-21T09:00:03.944 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-21T09:00:03.946 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-21T09:00:03.946 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-21T09:00:03.946 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log 2024-04-21T09:00:03.947 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log: 86.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log: 90.3% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: 94.6% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr: 83.7% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log 2024-04-21T09:00:03.948 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:00:03.949 INFO:teuthology.orchestra.run.smithi136.stderr: 94.4% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-21T09:00:03.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log: 92.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-21T09:00:03.950 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log 2024-04-21T09:00:03.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:00:03.951 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log 2024-04-21T09:00:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:00:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log: 84.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:00:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log 2024-04-21T09:00:03.954 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log 2024-04-21T09:00:03.955 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log: 94.5% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-21T09:00:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-21T09:00:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log 2024-04-21T09:00:03.956 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log 2024-04-21T09:00:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log: -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log 2024-04-21T09:00:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log 2024-04-21T09:00:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log 2024-04-21T09:00:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log 2024-04-21T09:00:03.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log 2024-04-21T09:00:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log 2024-04-21T09:00:03.959 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log 2024-04-21T09:00:03.960 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:00:03.962 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log: 94.4% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-21T09:00:03.963 INFO:teuthology.orchestra.run.smithi057.stderr: 94.5% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-21T09:00:03.963 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:00:03.963 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-21T09:00:03.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log 2024-04-21T09:00:03.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log 2024-04-21T09:00:03.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log 2024-04-21T09:00:03.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log 2024-04-21T09:00:03.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log 2024-04-21T09:00:03.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-21T09:00:03.965 INFO:teuthology.orchestra.run.smithi136.stderr: 91.0% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:00:03.968 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:00:03.968 INFO:teuthology.orchestra.run.smithi057.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:00:03.970 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-21T09:00:04.126 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-21T09:00:04.127 INFO:teuthology.orchestra.run.smithi129.stderr: 94.5% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-21T09:00:04.217 INFO:teuthology.orchestra.run.smithi129.stderr: 94.5% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-21T09:00:04.222 INFO:teuthology.orchestra.run.smithi129.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-21T09:00:04.226 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-volume.log.gz 2024-04-21T09:00:04.269 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.log.gz 2024-04-21T09:00:04.274 INFO:teuthology.orchestra.run.smithi129.stderr: 90.3% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.audit.log.gz 2024-04-21T09:00:04.319 INFO:teuthology.orchestra.run.smithi057.stderr: 90.3% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-21T09:00:04.359 INFO:teuthology.orchestra.run.smithi136.stderr: 91.8% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-21T09:00:04.529 INFO:teuthology.orchestra.run.smithi129.stderr: 92.0% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-21T09:00:04.959 INFO:teuthology.orchestra.run.smithi129.stderr: 83.7% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-21T09:00:05.260 INFO:teuthology.orchestra.run.smithi057.stderr: 90.3% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-21T09:00:09.252 INFO:teuthology.orchestra.run.smithi129.stderr: 91.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-21T09:00:15.437 INFO:teuthology.orchestra.run.smithi136.stderr: 95.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-21T09:00:37.356 INFO:teuthology.orchestra.run.smithi057.stderr: 91.8% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-21T09:00:37.711 INFO:teuthology.orchestra.run.smithi136.stderr: 92.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-21T09:00:39.339 INFO:teuthology.orchestra.run.smithi136.stderr: 92.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-21T09:00:42.244 INFO:teuthology.orchestra.run.smithi057.stderr: 91.7% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-21T09:00:45.556 INFO:teuthology.orchestra.run.smithi057.stderr: 91.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-21T09:00:47.458 INFO:teuthology.orchestra.run.smithi136.stderr: 92.1% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-21T09:00:48.676 INFO:teuthology.orchestra.run.smithi057.stderr: 91.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-21T09:00:48.678 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-04-21T09:00:48.678 INFO:teuthology.orchestra.run.smithi057.stderr:real 0m44.754s 2024-04-21T09:00:48.678 INFO:teuthology.orchestra.run.smithi057.stderr:user 2m35.081s 2024-04-21T09:00:48.678 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m4.480s 2024-04-21T09:00:54.306 INFO:teuthology.orchestra.run.smithi136.stderr: 91.8% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-21T09:00:54.308 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-21T09:00:54.308 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m50.381s 2024-04-21T09:00:54.308 INFO:teuthology.orchestra.run.smithi136.stderr:user 2m49.020s 2024-04-21T09:00:54.308 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m5.427s 2024-04-21T09:01:10.421 INFO:teuthology.orchestra.run.smithi129.stderr: 91.8% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-21T09:02:22.624 INFO:teuthology.orchestra.run.smithi129.stderr: 91.7% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-21T09:03:34.193 INFO:teuthology.orchestra.run.smithi129.stderr: 91.9% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-21T09:04:30.061 INFO:teuthology.orchestra.run.smithi129.stderr: 92.3% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-21T09:06:44.966 INFO:teuthology.orchestra.run.smithi129.stderr: 93.2% -- replaced with /var/log/ceph/df0fb3ae-ffb9-11ee-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-21T09:06:44.968 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-04-21T09:06:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:real 6m41.040s 2024-04-21T09:06:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:user 5m11.158s 2024-04-21T09:06:44.968 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m12.137s 2024-04-21T09:06:44.969 INFO:tasks.cephadm:Archiving logs... 2024-04-21T09:06:44.969 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi057/log 2024-04-21T09:06:44.971 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:07:10.983 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi129/log 2024-04-21T09:07:10.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:08:01.357 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi136/log 2024-04-21T09:08:01.358 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-21T09:08:27.123 INFO:tasks.cephadm:Removing cluster... 2024-04-21T09:08:27.124 DEBUG:teuthology.orchestra.run.smithi057:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T09:08:27.324 INFO:teuthology.orchestra.run.smithi057.stdout:Deleting cluster with fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:08:27.834 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T09:08:28.027 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:08:28.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid df0fb3ae-ffb9-11ee-bc93-c7b262605968 --force 2024-04-21T09:08:28.849 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: df0fb3ae-ffb9-11ee-bc93-c7b262605968 2024-04-21T09:08:29.358 INFO:tasks.cephadm:Removing cephadm ... 2024-04-21T09:08:29.359 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:08:29.379 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:08:29.396 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-21T09:08:29.412 INFO:tasks.cephadm:Teardown complete 2024-04-21T09:08:29.413 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-21T09:08:29.424 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-21T09:08:29.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:08:29.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:08:29.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-21T09:08:29.551 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-21T09:08:29.561 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-21T09:08:29.562 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:08:29.564 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:08:29.567 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-21T09:08:29.579 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:08:29.580 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:08:29.581 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-04-21T09:08:29.584 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:08:29.584 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-04-21T09:08:29.584 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 -137us[ -103us] +/- 89ms 2024-04-21T09:08:29.585 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -403us[ -370us] +/- 54ms 2024-04-21T09:08:29.585 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +645us[ +645us] +/- 63ms 2024-04-21T09:08:29.585 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:08:29.586 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:08:29.587 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-04-21T09:08:29.587 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 -126us[ -102us] +/- 89ms 2024-04-21T09:08:29.587 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -410us[ -386us] +/- 54ms 2024-04-21T09:08:29.587 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +646us[ +646us] +/- 63ms 2024-04-21T09:08:29.587 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:08:29.598 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-21T09:08:29.598 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-04-21T09:08:29.598 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 -161us[ -161us] +/- 89ms 2024-04-21T09:08:29.598 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -431us[ -427us] +/- 54ms 2024-04-21T09:08:29.599 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +613us[ +613us] +/- 63ms 2024-04-21T09:08:29.599 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-21T09:08:29.600 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-21T09:08:29.611 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-21T09:08:29.611 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-21T09:08:29.621 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:08:29.711 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:08:29.789 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-21T09:08:29.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:08:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:08:29.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:08:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:08:29.912 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-21T09:08:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:08:29.938 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-21T09:08:29.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-21T09:08:29.960 INFO:teuthology.task.internal:Duration was 2455.282146 seconds 2024-04-21T09:08:29.961 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-21T09:08:29.972 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-21T09:08:29.972 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:08:29.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:08:29.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-21T09:08:30.013 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:08:30.013 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:08:30.029 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-21T09:08:30.383 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-21T09:08:30.383 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2024-04-21T09:08:30.383 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:08:30.446 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-04-21T09:08:30.446 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:08:30.478 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-04-21T09:08:30.479 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-21T09:08:30.507 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-21T09:08:30.507 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:08:30.510 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:08:30.521 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-21T09:08:30.573 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-21T09:08:30.573 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:08:30.578 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:08:30.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-21T09:08:30.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-21T09:08:30.989 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-21T09:08:30.989 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:08:31.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:08:31.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-21T09:08:31.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-21T09:08:31.075 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:08:31.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:08:31.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:08:31.098 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2024-04-21T09:08:31.107 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-04-21T09:08:31.126 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-04-21T09:08:31.150 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:08:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:08:31.165 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:08:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:08:31.184 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-21T09:08:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-21T09:08:31.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-21T09:08:31.217 INFO:teuthology.task.internal:Transferring archived files... 2024-04-21T09:08:31.218 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi057 2024-04-21T09:08:31.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:08:31.260 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi129 2024-04-21T09:08:31.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:08:31.297 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520/remote/smithi136 2024-04-21T09:08:31.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-21T09:08:31.332 INFO:teuthology.task.internal:Removing archive directory... 2024-04-21T09:08:31.332 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:08:31.335 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:08:31.337 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-21T09:08:31.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-21T09:08:31.394 INFO:teuthology.task.internal:Not uploading archives. 2024-04-21T09:08:31.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-21T09:08:31.405 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-21T09:08:31.405 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:08:31.408 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:08:31.410 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-21T09:08:31.420 INFO:teuthology.orchestra.run.smithi057.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:08 /home/ubuntu/cephtest 2024-04-21T09:08:31.422 INFO:teuthology.orchestra.run.smithi129.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:08 /home/ubuntu/cephtest 2024-04-21T09:08:31.435 INFO:teuthology.orchestra.run.smithi136.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 21 09:08 /home/ubuntu/cephtest 2024-04-21T09:08:31.437 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-21T09:08:31.448 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-21T09:08:31.471 INFO:teuthology.nuke:Checking targets against current locks 2024-04-21T09:08:31.498 DEBUG:teuthology.nuke:shortname: smithi057 2024-04-21T09:08:31.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:08:31.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:57.320004', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDjX0ijBTgLbBPUyjZ3nVXCfPy93da4luDPppeYTkoeK3VUkNvP74ytGMnARsTwcQXdfHAXq2vUJFV1poy7h3k='} 2024-04-21T09:08:31.526 DEBUG:teuthology.nuke:shortname: smithi129 2024-04-21T09:08:31.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:08:31.550 DEBUG:teuthology.nuke:shortname: smithi136 2024-04-21T09:08:31.550 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-21T09:08:31.554 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:57.318679', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH4TrixGAKYMBUBaznn3bph4e44sEH3clkY2q4tMkTVDky9gv6scAOb9g0W4aWl+MTC92IGXxocVOBbocn8CsOk='} 2024-04-21T09:08:31.583 INFO:teuthology.orchestra.console.smithi057:Power off 2024-04-21T09:08:31.584 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:08:31.593 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-21_08:12:55-fs-wip-lusov-quiescer-distro-default-smithi/7666520', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-21 08:21:57.317285', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHQbN6MN5RiTZ5v4DsWWIfrctTqM3x8cZ2VWQSYOaxwxFN5lfvQalZiR50T+HhQMwG4KqaINUkDqHYE40tslcms='} 2024-04-21T09:08:31.608 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-04-21T09:08:31.608 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:31.626 INFO:teuthology.orchestra.console.smithi129:Power off 2024-04-21T09:08:31.627 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:08:31.637 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-04-21T09:08:31.650 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-04-21T09:08:31.650 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:31.663 INFO:teuthology.orchestra.console.smithi136:Power off 2024-04-21T09:08:31.663 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-21T09:08:31.673 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-04-21T09:08:31.685 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-04-21T09:08:31.686 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:31.710 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-21T09:08:35.638 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:35.674 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:35.711 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:35.749 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-04-21T09:08:35.783 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-04-21T09:08:35.820 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-21T09:08:39.754 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:39.783 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:39.822 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:39.864 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-04-21T09:08:39.892 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-04-21T09:08:39.930 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-04-21T09:08:39.993 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-04-21T09:08:40.031 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-04-21T09:08:43.865 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-21T09:08:43.974 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-04-21T09:08:44.075 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-04-21T09:08:44.181 INFO:teuthology.run:Summary data: description: fs/workload/{0-ubuntu_22.04 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} duration: 2455.282145500183 failure_reason: 'Command failed (workunit test kernel_untar_build.sh) on smithi057 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f6bc5273643aaf5660909e85400647a8951c10b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d7a0e8de39354bb580519907c654895b status: fail success: false 2024-04-21T09:08:44.181 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-21T09:08:44.264 INFO:teuthology.run:FAIL